检索结果-内蒙古大学图书馆

On the Robustness of Generative Retrieval Models: An Out-of-Distribution Perspective

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liu, Yu-An Zhang, Ruqing Guo, Jiafeng Chen, Wei Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

Recently, we have witnessed generative retrieval increasingly gaining attention in the information retrieval (IR) field, which retrieves documents by directly generating their identifiers. So far, much effort has been devoted to developing effective generative retrieval models. There has been less attention paid to the robustness perspective. When a new retrieval paradigm enters into the real-world application, it is also critical to measure the out-of-distribution (OOD) generalization, i.e., how would generative retrieval models generalize to new distributions. To answer this question, firstly, we define OOD robustness from three perspectives in retrieval problems: 1) The query variations;2) The unforeseen query types;and 3) The unforeseen tasks. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of several representative generative retrieval models against dense retrieval models. The empirical results indicate that the OOD robustness of generative retrieval models requires enhancement. We hope studying the OOD robustness of generative retrieval models would be advantageous to the IR community. Copyright © 2023, The Authors. All rights reserved.

关键词： Query processing

Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Ma, Xinyu Guo, Jiafeng Zhang, Ruqing Fan, Yixing Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

Pre-training and fine-tuning have achieved significant advances in the information retrieval (IR). A typical approach is to fine-tune all the parameters of large-scale pre-trained models (PTMs) on downstream tasks. As the model size and the number of tasks increase greatly, such approach becomes less feasible and prohibitively expensive. Recently, a variety of parameter-efficient tuning methods have been proposed in natural language processing (NLP) that only fine-tune a small number of parameters while still attaining strong performance. Yet there has been little effort to explore parameter-efficient tuning for IR. In this work, we first conduct a comprehensive study of existing parameter-efficient tuning methods at both the retrieval and re-ranking stages. Unlike the promising results in NLP, we find that these methods cannot achieve comparable performance to full fine-tuning at both stages when updating less than 1% of the original model parameters. More importantly, we find that the existing methods are just parameter-efficient, but not learning-efficient as they suffer from unstable training and slow convergence. To analyze the underlying reason, we conduct a theoretical analysis and show that the separation of the inserted trainable modules makes the optimization difficult. To alleviate this issue, we propose to inject additional modules alongside the pre-trained models (PTMs) to make the original scattered modules connected. In this way, all the trainable modules can form a pathway to smooth the loss surface and thus help stabilize the training process. Experiments at both retrieval and re-ranking stages show that our method outperforms existing parameter-efficient methods significantly, and achieves comparable or even better performance over full fine-tuning. Copyright © 2022, The Authors. All rights reserved.

关键词： Information retrieval

CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cai, Yinqiong Fan, Yixing Bi, Keping Guo, Jiafeng Chen, Wei Zhang, Ruqing Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

The first-stage retrieval aims to retrieve a subset of candidate documents from a huge collection both effectively and efficiently. Since various matching patterns can exist between queries and relevant documents, previous work tries to combine multiple retrieval models to find as many relevant results as possible. The constructed ensembles, whether learned independently or jointly, do not care which component model is more suitable to an instance during training. Thus, they cannot fully exploit the capabilities of different types of retrieval models in identifying diverse relevance patterns. Motivated by this observation, in this paper, we propose a Mixture-of-Experts (MoE) model consisting of representative matching experts and a novel competitive learning mechanism to let the experts develop and enhance their expertise during training. Specifically, our MoE model shares the bottom layers to learn common semantic representations and uses differently structured upper layers to represent various types of retrieval experts. Our competitive learning mechanism has two stages: (1) a standardized learning stage to train the experts equally to develop their capabilities to conduct relevance matching;(2) a specialized learning stage where the experts compete with each other on every training instance and get rewards and updates according to their performance to enhance their expertise on certain types of samples. Experimental results on three retrieval benchmark datasets show that our method significantly outperforms the state-of-the-art baselines. Copyright © 2023, The Authors. All rights reserved.

关键词： Semantics

Ensemble Ranking Model with Multiple Pretraining Strategies for Web Search

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Sun, Xiaojie Yu, Lulu Wang, Yiting Bi, Keping Guo, Jiafeng CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

An effective ranking model usually requires a large amount of training data to learn the relevance between documents and queries. User clicks are often used as training data since they can indicate relevance and are cheap to collect, but they contain substantial bias and noise. There has been some work on mitigating various types of bias in simulated user clicks to train effective learning-to-rank models based on multiple features. However, how to effectively use such methods on large-scale pre-trained models with real-world click data is unknown. To alleviate the data bias in the real world, we incorporate heuristic-based features, refine the ranking objective, add random negatives, and calibrate the propensity calculation in the pre-training stage. Then we fine-tune several pre-trained models and train an ensemble model to aggregate all the predictions from various pre-trained models with human-annotation data in the fine-tuning stage. Our approaches won 3rd place in the "Pretraining for Web Search" task in WSDM Cup 2023 and are 22.6% better than the 4th-ranked team. Copyright © 2023, The Authors. All rights reserved.

关键词： Websites

When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ni, Shiyu Bi, Keping Guo, Jiafeng Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS China University of Chinese Academy of Sciences China

Large Language Models (LLMs) have been found to have difficulty knowing they do not possess certain knowledge and tend to provide specious answers in such cases. Retrieval Augmentation (RA) has been extensively studied to mitigate LLMs’ hallucinations. However, due to the extra overhead and unassured quality of retrieval, it may not be optimal to conduct RA all the time. A straightforward idea is to only conduct retrieval when LLMs are uncertain about a question. This motivates us to enhance the LLMs’ ability to perceive their knowledge boundaries to help RA. In this paper, we first quantitatively measure LLMs’ such ability and confirm their overconfidence. Then, we study how LLMs’ certainty about a question correlates with their dependence on external retrieved information. We propose several methods to enhance LLMs’ perception of knowledge boundaries and show that they are effective in reducing overconfidence. Additionally, equipped with these methods, LLMs can achieve comparable or even better performance of RA with much fewer retrieval calls. The code can be found at https://***/ShiyuNee/When-to-Retrieve. Copyright © 2024, The Authors. All rights reserved.

关键词： Machine learning

An end-to-end progressive multi-task learning framework for medical named entity recognition and normalization 59

学校读者我要写书评

暂无评论

An end-to-end progressive multi-task learning framework for ...

Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP 2021

作者： Zhou, Baohang Cai, Xiangrui Zhang, Ying Yuan, Xiaojie College of Computer Science Nankai University Tianjin300350 China College of Cyber Science Nankai University Tianjin300350 China Tianjin Key Laboratory of Network and Data Security Technology Tianjin300350 China

ISBN: (纸本)9781954085527

Medical named entity recognition (NER) and normalization (NEN) are fundamental for constructing knowledge graphs and building QA systems. Existing implementations for medical NER and NEN are suffered from the error propagation between the two tasks. The mispredicted mentions from NER will directly influence the results of NEN. Therefore, the NER module is the bottleneck of the whole system. Besides, the learnable features for both tasks are beneficial to improving the model performance. To avoid the disadvantages of existing models and exploit the generalized representation across the two tasks, we design an end-to-end progressive multi-task learning model for jointly modeling medical NER and NEN in an effective way. There are three level tasks with progressive difficulty in the framework. The progressive tasks can reduce the error propagation with the incremental task settings which implies the lower level tasks gain the supervised signals other than errors from the higher level tasks to improve their performances. Besides, the context features are exploited to enrich the semantic information of entity mentions extracted by NER. The performance of NEN profits from the enhanced entity mention features. The standard entities from knowledge bases are introduced into the NER module for extracting corresponding entity mentions correctly. The empirical results on two publicly available medical literature datasets demonstrate the superiority of our method over nine typical methods. © 2021 Association for Computational Linguistics

关键词： Knowledge graph

Regularization Mixup Adversarial Training: A Defense Strategy for Membership Privacy with Model Availability Assurance 2

学校读者我要写书评

暂无评论

Regularization Mixup Adversarial Training: A Defense Strateg...

2nd International Conference on Big data and Privacy Computing, BDPC 2024

作者： Ding, Zehua Tian, Youliang Wang, Guorong Xiong, Jinbo College of Computer Science and Technology Guizhou University State Key Laboratory of Public Big Data Guiyang China College of Computer and Cyber Security Fujian Normal University Fujian Provincial Key Laboratory of Network Security and Cryptology Fuzhou China

ISBN: (纸本)9798350307597

Neural network models face two highly destructive threats in real-world applications: membership inference attacks (MIAs) and adversarial attacks (AAs). One compromises the model's confidentiality, leading to membership privacy breaches, while the other disrupts the model's availability, rendering it dysfunctional. Recent work has shown that adversarial examples can pose dual threats to neural network models, both MIAs and AAs, so that these two different types of attacks have a certain degree of intersection. However, existing defense methods typically focus on defending against one of the two attacks and cannot simultaneously address both. To address the above issues, we propose a defense method called regularization mixup adversarial training (RMAT), aiming to protect model membership privacy while ensuring model availability. The core idea of RMAT is to use a continuous augmentation strategy during training to generate mixup adversarial examples, weakening attackers' understanding of membership data, and updating the model with a new membership-weighted loss strategy to improve its generalization ability for handling unknown samples. This approach can further protect the model's membership privacy and ensure model availability. Experimental results demonstrate that compared to single defense strategies, RMAT can simultaneously protect the model's membership privacy and availability. © 2024 IEEE.

关键词： Adversarial machine learning

Secure real-time image protection scheme with near-duplicate detection in cloud computing

学校读者我要写书评

暂无评论

Secure real-time image protection scheme with near-duplicate...

作者： Liu, Dengzhi Shen, Jian Wang, Anxi Wang, Chen Jiangsu Engineering Center of Network Monitoring Nanjing University of Information Science and Technology Nanjing210044 China State Key Laboratory of Cryptology Beijing100878 China Guizhou Provincial Key Laboratory of Public Big Data Guiyang550025 China

With advancements in technologies of the Internet and multi-media, various images need to be generated and transmitted anytime. Restricted by local constrained storage space, users can store their images with the assist of the cloud. However, the cloud is a remote semi-trusted party that may extract stored images for adversaries due to monetary reasons. In this paper, a secure real-time image protection scheme is proposed, which can be used to enhance the security of the stored images in cloud computing. Moreover, the convergent encryption is used to construct our scheme, which can provide functionalities of image deduplication checking and near-duplicate detection for the image owner. To improve the efficiency of the near-duplicate detection, deep learning is exploited in our scheme to extract images. Security analysis indicates that the proposed scheme can meet the security requirements of correctness and security. Performance analysis shows that the proposed scheme can be performed with low computational cost. © 2019, Springer-Verlag GmbH Germany, part of Springer Nature.

关键词： Cloud computing

Reproducibility Analysis and Enhancements for Multi-Aspect Dense Retriever with Aspect Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Bi, Keping Sun, Xiaojie Guo, Jiafeng Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS China University of Chinese Academy of Sciences China

Multi-aspect dense retrieval aims to incorporate aspect information (e.g., brand and category) into dual encoders to facilitate relevance matching. As an early and representative multi-aspect dense retriever, MADRAL learns several extra aspect embeddings and fuses the explicit aspects with an implicit aspect "OTHER" for final representation. MADRAL was evaluated on proprietary data and its code was not released, making it challenging to validate its effectiveness on other datasets. We failed to reproduce its effectiveness on the public MA-Amazon data, motivating us to probe the reasons and re-examine its components. We propose several component alternatives for comparisons, including replacing "OTHER" with "CLS" and representing aspects with the first several content tokens. Through extensive experiments, we confirm that learning "OTHER" from scratch in aspect fusion is harmful. In contrast, our proposed variants can greatly enhance the retrieval performance. Our research not only sheds light on the limitations of MADRAL but also provides valuable insights for future studies on more powerful multi-aspect dense retrieval models. Code will be released at: https://***/sunxiaojie99/Reproducibility-for-MADRAL. © 2024, CC BY.

关键词：