检索结果-内蒙古大学图书馆

Multi-granular Adversarial Attacks against Black-box Neural Ranking Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Liu, Yu-An Zhang, Ruqing Guo, Jiafeng de Rijke, Maarten Fan, Yixing Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China University of Amsterdam Amsterdam Netherlands

Adversarial ranking attacks have gained increasing attention due to their success in probing vulnerabilities, and, hence, enhancing the robustness, of neural ranking models. Conventional attack methods employ perturbations at a single granularity, e.g., word or sentence level, to target documents. However, limiting perturbations to a single level of granularity may reduce the flexibility of adversarial examples, thereby diminishing the potential threat of the attack. Therefore, we focus on generating high-quality adversarial examples by incorporating multi-granular perturbations. Achieving this objective involves tackling a combinatorial explosion problem, which requires identifying an optimal combination of perturbations across all possible levels of granularity, positions, and textual pieces. To address this challenge, we transform the multi-granular adversarial attack into a sequential decision-making process, where perturbations in the next attack step build on the perturbed document in the current attack step. Since the attack process can only access the final state without direct intermediate signals, we use reinforcement learning to perform multi-granular attacks. During the reinforcement learning process, two agents work cooperatively to identify multi-granular vulnerabilities as attack targets and organize perturbation candidates into a final perturbation sequence. Experimental results show that our attack method surpasses prevailing baselines in both attack effectiveness and imperceptibility. Copyright © 2024, The Authors. All rights reserved.

关键词： Reinforcement learning

Inducing Causal Structure for Abstractive Text Summarization

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Chen, Lu Zhang, Ruqing Huang, Wei Chen, Wei Guo, Jiafeng Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

The mainstream of data-driven abstractive summarization models tends to explore the correlations rather than the causal relationships. Among such correlations, there can be spurious ones which suffer from the language prior learned from the training corpus and therefore undermine the overall effectiveness of the learned model. To tackle this issue, we introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We assume several latent causal factors and non-causal factors, representing the content and style of the document and summary. Theoretically, we prove that the latent factors in our SCM can be identified by fitting the observed training data under certain conditions. On the basis of this, we propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors, guiding us to pursue causal information for summary generation. The key idea is to reformulate the Variational Auto-encoder (VAE) to fit the joint distribution of the document and summary variables from the training corpus. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach. © 2023, CC BY-NC-ND.

关键词： Text processing

L2R: Lifelong Learning for First-stage Retrieval with Backward-Compatible Representations

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cai, Yinqiong Bi, Keping Fan, Yixing Guo, Jiafeng Chen, Wei Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

First-stage retrieval is a critical task that aims to retrieve relevant document candidates from a large-scale collection. While existing retrieval models have achieved impressive performance, they are mostly studied on static data sets, ignoring that in the real-world, the data on the Web is continuously growing with potential distribution drift. Consequently, retrievers trained on static old data may not suit new-coming data well and inevitably produce sub-optimal results. In this work, we study lifelong learning for first-stage retrieval, especially focusing on the setting where the emerging documents are unlabeled since relevance annotation is expensive and may not keep up with data emergence. Under this setting, we aim to develop model updating with two goals: (1) to effectively adapt to the evolving distribution with the unlabeled new-coming data, and (2) to avoid re-inferring all embeddings of old documents to efficiently update the index each time the model is updated. We first formalize the task and then propose a novel Lifelong Learning method for the first-stage Retrieval, namely L2R. L2R adopts the typical memory mechanism for lifelong learning, and incorporates two crucial components: (1) selecting diverse support negatives for model training and memory updating for effective model adaptation, and (2) a ranking alignment objective to ensure the backward-compatibility of representations to save the cost of index rebuilding without hurting the model performance. For evaluation, we construct two new benchmarks from LoTTE and Multi-CPR © 2023, CC BY-NC-ND.

关键词： Learning systems

A Comparative Study of Training Objectives for Clarification Facet Generation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ni, Shiyu Bi, Keping Guo, Jiafeng Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

Due to the ambiguity and vagueness of a user query, it is essential to identify the query facets for the clarification of user intents. Existing work on query facet generation has achieved compelling performance by sequentially predicting the next facet given previously generated facets based on pre-trained language generation models such as BART. Given a query, there are mainly two types of training objectives to guide the facet generation models. One is to generate the default sequence of ground-truth facets, and the other is to enumerate all the permutations of ground-truth facets and use the sequence that has the minimum loss for model updates. The second is permutation-invariant while the first is not. In this paper, we aim to conduct a systematic comparative study of various types of training objectives, with different properties of not only whether it is permutation-invariant but also whether it conducts sequential prediction and whether it can control the count of output facets. To this end, we propose another three training objectives of different aforementioned properties. For comprehensive comparisons, besides the commonly used evaluation that measures the matching with ground-truth facets, we also introduce two diversity metrics to measure the diversity of the generated facets. Based on an open-domain query facet dataset, i.e., MIMICS, we conduct extensive analyses and show the pros and cons of each method, which could shed light on model training for clarification facet generation. The code can be found at https://***/ShiyuNee/Facet-Generation. © 2023, CC BY.

关键词： Clarifiers

Are Large Language Models Good at Utility Judgments?

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Hengran Zhang, Ruqing Guo, Jiafeng de Rijke, Maarten Fan, Yixing Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China University of Amsterdam Amsterdam Netherlands

Retrieval-augmented generation (RAG) is considered to be a promising approach to alleviate the hallucination issue of large language models (LLMs), and it has received widespread attention from researchers recently. Due to the limitation in the semantic understanding of retrieval models, the success of RAG heavily lies on the ability of LLMs to identify passages with utility. Recent efforts have explored the ability of LLMs to assess the relevance of passages in retrieval, but there has been limited work on evaluating the utility of passages in supporting question answering. In this work, we conduct a comprehensive study about the capabilities of large language models (LLMs) in utility evaluation for open-domain question answering (QA). Specifically, we introduce a benchmarking procedure and collection of candidate passages with different characteristics, facilitating a series of experiments with five representative LLMs. Our experiments reveal that: (i) well-instructed LLMs can distinguish between relevance and utility, and that LLMs are highly receptive to newly generated counterfactual passages. Moreover, (ii) we scrutinize key factors that affect utility judgments in the instruction design. And finally, (iii) to verify the efficacy of utility judgments in practical retrieval augmentation applications, we delve into LLMs’ QA capabilities using the evidence judged with utility and direct dense retrieval results. (iv) We propose a k-sampling, listwise approach to reduce the dependency of LLMs on the sequence of input passages, thereby facilitating subsequent answer generation. We believe that the way we formalize and study the problem along with our findings contributes to a critical assessment of retrieval-augmented LLMs. Our code and benchmark can be found at https://***/ict-bigdatalab/utility_judgments. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantics

Ensemble Ranking Model with Multiple Pretraining Strategies for Web Search

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Sun, Xiaojie Yu, Lulu Wang, Yiting Bi, Keping Guo, Jiafeng CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

An effective ranking model usually requires a large amount of training data to learn the relevance between documents and queries. User clicks are often used as training data since they can indicate relevance and are cheap to collect, but they contain substantial bias and noise. There has been some work on mitigating various types of bias in simulated user clicks to train effective learning-to-rank models based on multiple features. However, how to effectively use such methods on large-scale pre-trained models with real-world click data is unknown. To alleviate the data bias in the real world, we incorporate heuristic-based features, refine the ranking objective, add random negatives, and calibrate the propensity calculation in the pre-training stage. Then we fine-tune several pre-trained models and train an ensemble model to aggregate all the predictions from various pre-trained models with human-annotation data in the fine-tuning stage. Our approaches won 3rd place in the "Pretraining for Web Search" task in WSDM Cup 2023 and are 22.6% better than the 4th-ranked team. Copyright © 2023, The Authors. All rights reserved.

关键词： Websites

On the Robustness of Generative Retrieval Models: An Out-of-Distribution Perspective

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liu, Yu-An Zhang, Ruqing Guo, Jiafeng Chen, Wei Cheng, Xueqi CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

Recently, we have witnessed generative retrieval increasingly gaining attention in the information retrieval (IR) field, which retrieves documents by directly generating their identifiers. So far, much effort has been devoted to developing effective generative retrieval models. There has been less attention paid to the robustness perspective. When a new retrieval paradigm enters into the real-world application, it is also critical to measure the out-of-distribution (OOD) generalization, i.e., how would generative retrieval models generalize to new distributions. To answer this question, firstly, we define OOD robustness from three perspectives in retrieval problems: 1) The query variations;2) The unforeseen query types;and 3) The unforeseen tasks. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of several representative generative retrieval models against dense retrieval models. The empirical results indicate that the OOD robustness of generative retrieval models requires enhancement. We hope studying the OOD robustness of generative retrieval models would be advantageous to the IR community. Copyright © 2023, The Authors. All rights reserved.

关键词： Query processing

Feature-Enhanced network with Hybrid Debiasing Strategies for Unbiased Learning to Rank

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yu, Lulu Wang, Yiting Sun, Xiaojie Bi, Keping Guo, Jiafeng CAS Key Lab of Network Data Science and Technology ICT CAS University of Chinese Academy of Sciences Beijing China

Unbiased learning to rank (ULTR) aims to mitigate various biases existing in user clicks, such as position bias, trust bias, presentation bias, and learn an effective ranker. In this paper, we introduce our winning approach for the "Unbiased Learning to Rank" task in WSDM Cup 2023. We find that the provided data is severely biased so neural models trained directly with the top 10 results with click information are unsatisfactory. So we extract multiple heuristic-based features for multi-fields of the results, adjust the click labels, add true negatives, and re-weight the samples during model training. Since the propensities learned by existing ULTR methods are not decreasing w.r.t. positions, we also calibrate the propensities according to the click ratios and ensemble the models trained in two different ways. Our method won the 3rd prize with a DCG@10 score of 9.80, which is 1.1% worse than the 2nd and 25.3% higher than the 4th. Copyright © 2023, The Authors. All rights reserved.

关键词： Machine learning

Heavy Rainfall Prediction Model Using Sample Entropy Derived from GNSS-PWV and PSO-SVM 14th

学校读者我要写书评

暂无评论

Heavy Rainfall Prediction Model Using Sample Entropy Derived...

14th China Satellite Navigation Conference, CSNC 2024

作者： Wu, Fanming Li, Dengao Zhao, Jinhua Feng, Ran Shi, Danyang Zhang, Xinfang Zhao, Jumin College of Data Science Taiyuan University of Technology Taiyuan030024 China College of Information and Computer Taiyuan University of Technology Taiyuan030024 China Key Laboratory of Big Data Fusion Analysis and Application of Shanxi Province Taiyuan030024 China Intelligent Perception Engineering Technology Center of Shanxi Taiyuan030024 China Shanxi Province Engineering Technology Research Center of Spatial Information Network Taiyuan030024 China

ISBN: (纸本)9789819969272

There is a growing interest to use Global Navigation Satellite System (GNSS) inversed PWV for heavy rainfall prediction. When heavy rainfall occurs, it requires the atmosphere to contain sufficient water vapour and undergo strong upward motion. However, existing models using GNSS-PWV for heavy rainfall prediction have not accounted for the effect of the complex motion of water vapour. In this paper, an hourly heavy rainfall prediction model using sample entropy derived from GNSS-PWV and PSO-SVM is proposed. The sample entropy of GNSS-PWV is used to measure the complexity of the water vapour movement process before heavy rainfall occurs. Meanwhile, combining GNSS-PWV time domain features and co-located meteorological data, the occurrence of hourly heavy rainfall events is predicted by support vector machine optimized with particle swarm. To verify the validity and feasibility of the proposed algorithm, Hong Kong's HKSC station in Sham Shui Po is used for the model to train and test. The result shows that probability of detection, false alarm rate and critical success index of the proposed algorithm in this paper have significantly improved over other heavy rainfall prediction models. © 2024, Aerospace Information Research Institute.

关键词： Support vector machines