检索结果-内蒙古大学图书馆

Coded Residual Transform for Generalizable Deep Metric Learning

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Kan, Shichao Liang, Yixiong Li, Min Cen, Yigang Wang, Jianxin He, Zhihai School of Computer Science and Engineering Central South University Hunan Changsha410083 China Institute of Information Science School of Computer and Information Technology Beijing Jiaotong University Beijing100044 China Beijing Key Laboratory of Advanced Information Science and Network Technology Beijing100044 China Department of Electrical and Electronic Engineering Southern University of Science and Technology Shenzhen China Pengcheng Lab Shenzhen518066 China

A fundamental challenge in deep metric learning is the generalization capability of the feature embedding network model since the embedding network learned on training classes need to be evaluated on new test classes. To address this challenge, in this paper, we introduce a new method called coded residual transform (CRT) for deep metric learning to significantly improve its generalization capability. Specifically, we learn a set of diversified prototype features, project the feature map onto each prototype, and then encode its features using their projection residuals weighted by their correlation coefficients with each prototype. The proposed CRT method has the following two unique characteristics. First, it represents and encodes the feature map from a set of complimentary perspectives based on projections onto diversified prototypes. Second, unlike existing transformer-based feature representation approaches which encode the original values of features based on global correlation analysis, the proposed coded residual transform encodes the relative differences between the original features and their projected prototypes. Embedding space density and spectral decay analysis show that this multi-perspective projection onto diversified prototypes and coded residual representation are able to achieve significantly improved generalization capability in metric learning. Finally, to further enhance the generalization performance, we propose to enforce the consistency on their feature similarity matrices between coded residual transforms with different sizes of projection prototypes and embedding dimensions. Our extensive experimental results and ablation studies demonstrate that the proposed CRT method outperform the state-of-the-art deep metric learning methods by large margins and improving upon the current best method by up to 4.28% on the CUB dataset. Copyright © 2022, The Authors. All rights reserved.

关键词： Deep learning

Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Liu, Ji Jia, Juncheng Zhang, Hong Yun, Yuhui Wang, Leye Zhou, Yang Dai, Huaiyu Dou, Dejing Hithink RoyalFlush Information Network Co. Ltd. Hangzhou China School of Computer Science and Technology Soochow University Collaborative Innovation Center of Novel Software Technology and Industrialization Suzhou China School of Computer Science and Technology Soochow University Suzhou China Baidu Research Beijing China Key Lab of High Confidence Software Technologies Ministry of Education Software Institute Peking University Beijing China Department of Computer Science and Software Engineering Auburn University Auburn United States Department of Electrical and Computer Engineering North Carolina State University NC United States BEDI Cloud and School of Computer Science Fudan University Beijing China

Despite achieving remarkable performance, Federated Learning (FL) encounters two important problems, i.e., low training efficiency and limited computational resources. In this paper, we propose a new FL framework, i.e., FedDUMAP, with three original contributions, to leverage the shared insensitive data on the server in addition to the distributed data in edge devices so as to efficiently train a global model. First, we propose a simple dynamic server update algorithm, which takes advantage of the shared insensitive data on the server while dynamically adjusting the update steps on the server in order to speed up the convergence and improve the accuracy. Second, we propose an adaptive optimization method with the dynamic server update algorithm to exploit the global momentum on the server and each local device for superior accuracy. Third, we develop a layer-adaptive model pruning method to carry out specific pruning operations, which is adapted to the diverse features of each layer so as to attain an excellent trade-off between effectiveness and efficiency. Our proposed FL model, FedDUMAP, combines the three original techniques and has a significantly better performance compared with baseline approaches in terms of efficiency (up to 16.9 times faster), accuracy (up to 20.4% higher), and computational cost (up to 62.6% smaller). © 2024, CC BY.

关键词： Federated learning

Learning to truncate ranked lists for information retrieval

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Wu, Chen Zhang, Ruqing Guo, Jiafeng Fan, Yixing Lan, Yanyan Cheng, Xueqi CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences Beijing China University of Chinese Academy of Sciences Beijing China

Ranked list truncation is of critical importance in a variety of professional information retrieval applications such as patent search or legal search. The goal is to dynamically determine the number of returned documents according to some user-defined objectives, in order to reach a balance between the overall utility of the results and user efforts. Existing methods formulate this task as a sequential decision problem and take some pre-defined loss as a proxy objective, which suffers from the limitation of local decision and non-direct optimization. In this work, we propose a global decision based truncation model named AttnCut, which directly optimizes user-defined objectives for the ranked list truncation. Specifically, we take the successful transformer architecture to capture the global dependency within the ranked list for truncation decision, and employ the reward augmented maximum likelihood (RAML) for direct optimization. We consider two types of user-defined objectives which are of practical usage. One is the widely adopted metric such as F1 which acts as a balanced objective, and the other is the best F1 under some minimal recall constraint which represents a typical objective in professional search. Empirical results over the Robust04 and MQ2007 datasets demonstrate the effectiveness of our approach as compared with the state-of-the-art baselines. Copyright © 2021, The Authors. All rights reserved.

关键词： Information retrieval

Logiformer: A Two-Branch Graph Transformer network for Interpretable Logical Reasoning

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Xu, Fangzhi Liu, Jun Lin, Qika Pan, Yudai Zhang, Lingling School of Computer Science and Technology Xi'an Jiaotong University Xi'an China Shaanxi Province Key Laboratory of Satellite and Terrestrial Network Tech. R&d National Engineering Lab for Big Data Analytics Xi'an China

Machine reading comprehension has aroused wide concerns, since it explores the potential of model for text understanding. To further equip the machine with the reasoning capability, the challenging task of logical reasoning is proposed. Previous works on logical reasoning have proposed some strategies to extract the logical units from different aspects. However, there still remains a challenge to model the long distance dependency among the logical units. Also, it is demanding to uncover the logical structures of the text and further fuse the discrete logic to the continuous text embedding. To tackle the above issues, we propose an end-to-end model Logiformer which utilizes a two-branch graph transformer network for logical reasoning of text. Firstly, we introduce different extraction strategies to split the text into two sets of logical units, and construct the logical graph and the syntax graph respectively. The logical graph models the causal relations for the logical branch while the syntax graph captures the co-occurrence relations for the syntax branch. Secondly, to model the long distance dependency, the node sequence from each graph is fed into the fully connected graph transformer structures. The two adjacent matrices are viewed as the attention biases for the graph transformer layers, which map the discrete logical structures to the continuous text embedding space. Thirdly, a dynamic gate mechanism and a question-aware self-attention module are introduced before the answer prediction to update the features. The reasoning process provides the interpretability by employing the logical units, which are consistent with human cognition. The experimental results show the superiority of our model, which outperforms the state-of-the-art single model on two logical reasoning benchmarks. © 2022, CC BY.

关键词： Embeddings

A linguistic study on relevance modeling in information retrieval

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Fan, Yixing Guo, Jiafeng Ma, Xinyu Zhang, Ruqing Lan, Yanyan Cheng, Xueqi University of Chinese Academy of Sciences Beijing China CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences Beijing China

Relevance plays a central role in information retrieval (IR), which has received extensive studies starting from the 20th century. The definition and the modeling of relevance has always been critical challenges in both information science and computer science research areas. Along with the debate and exploration on relevance, IR has already become a core task in many real-world applications, such as Web search engines, question answering systems, conversational bots, and so on. While relevance acts as a unified concept in all these retrieval tasks, the inherent definitions are quite different due to the heterogeneity of these tasks. This raises a question to us: Do these different forms of relevance really lead to different modeling focuses? To answer this question, in this work, we conduct an empirical study on relevance modeling in three representative IR tasks, i.e., document retrieval, answer retrieval, and response retrieval. Specifically, we attempt to study the following two questions: 1) Does relevance modeling in these tasks really show differences in terms of natural language understanding (NLU)? We employ 16 linguistic tasks to probe a unified retrieval model over these three retrieval tasks to answer this question. 2) If there do exist differences, how can we leverage the findings to enhance the relevance modeling? We proposed three intervention methods to investigate how to leverage different modeling focuses of relevance to improve these IR tasks. We believe the way we study the problem as well as our findings would be beneficial to the IR community. © 2021, CC BY.

关键词： Information retrieval

FedMatch: Federated learning over heterogeneous question answering data

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Chen, Jiangui Zhang, Ruqing Guo, Jiafeng Fan, Yixing Cheng, Xueqi CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences Beijing China University of Chinese Academy of Sciences Beijing China

Question Answering (QA), a popular and promising technique for intelligent information access, faces a dilemma about data as most other AI techniques. On one hand, modern QA methods rely on deep learning models which are typically data-hungry. Therefore, it is expected to collect and fuse all the available QA datasets together in a common site for developing a powerful QA model. On the other hand, real-world QA datasets are typically distributed in the form of isolated islands belonging to different parties. Due to the increasing awareness of privacy security, it is almost impossible to integrate the data scattered around, or the cost is prohibited. A possible solution to this dilemma is a new approach known as federated learning, which is a privacy-preserving machine learning technique over distributed datasets. In this work, we propose to adopt federated learning for QA with the special concern on the statistical heterogeneity of the QA data. Here the heterogeneity refers to the fact that annotated QA data are typically with non-identical and independent distribution (non-IID) and unbalanced sizes in practice. Traditional federated learning methods may sacrifice the accuracy of individual models under the heterogeneous situation. To tackle this problem, we propose a novel Federated Matching framework for QA, named FedMatch, with a backbone-patch architecture. The shared backbone is to distill the common knowledge of all the participants while the private patch is a compact and efficient module to retain the domain information for each participant. To facilitate the evaluation, we build a benchmark collection based on several QA datasets from different domains to simulate the heterogeneous situation in practice. Empirical studies demonstrate that our model can achieve significant improvements against the baselines over all the datasets. © 2021, CC BY-NC-ND.

关键词： Deep learning

B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Ma, Xinyu Guo, Jiafeng Zhang, Ruqing Fan, Yixing Li, Yingyan Cheng, Xueqi CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences Beijing China University of Chinese Academy of Sciences Beijing China

Pre-training and fine-tuning have achieved remarkable success in many downstream natural language processing (NLP) tasks. Recently, pre-training methods tailored for information retrieval (IR) have also been explored, and the latest success is the PROP method which has reached new SOTA on a variety of ad-hoc retrieval benchmarks. The basic idea of PROP is to construct the representative words prediction (ROP) task for pre-training inspired by the query likelihood model. Despite its exciting performance, the effectiveness of PROP might be bounded by the classical unigram language model adopted in the ROP task construction process. To tackle this problem, we propose a bootstrapped pre-training method (namely B-PROP) based on BERT for ad-hoc retrieval. The key idea is to use the powerful contextual language model BERT to replace the classical unigram language model for the ROP task construction, and re-train BERT itself towards the tailored objective for IR. Specifically, we introduce a novel contrastive method, inspired by the divergence-from-randomness idea, to leverage BERT's self-attention mechanism to sample representative words from the document. By further fine-tuning on downstream ad-hoc retrieval tasks, our method achieves significant improvements over PROP and other baselines, and further pushes forward the SOTA on a variety of ad-hoc retrieval tasks. Copyright © 2021, The Authors. All rights reserved.

关键词： Natural language processing systems

DDistill-SR: Reparameterized Dynamic Distillation network for Lightweight Image Super-Resolution

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Yan Su, Tongtong Li, Yusen Cao, Jiuwen Wang, Gang Liu, Xiaoguang Nankai-Baidu Joint Lab School of Computer Science Nankai University Tianjin300350 China Tianjin Key Laboratory of Network and Data Security Technology Tianjin300071 China Machine Learning and I-health International Cooperation Base of Zhejiang Province and Artificial Intelligence Institute Hangzhou Dianzi University 300018 China

Recent research on deep convolutional neural networks (CNNs) has provided a significant performance boost on efficient super-resolution (SR) tasks by trading off the performance and applicability. However, most existing methods focus on subtracting feature processing consumption to reduce the parameters and calculations without refining the immediate features, which leads to inadequate information in the restoration. In this paper, we propose a lightweight network termed DDistill- SR, which significantly improves the SR quality by capturing and reusing more helpful information in a static-dynamic feature distillation manner. Specifically, we propose a plug-in reparameterized dynamic unit (RDU) to promote the performance and inference cost trade-off. During the training phase, the RDU learns to linearly combine multiple reparameterizable blocks by analyzing varied input statistics to enhance layerlevel representation. In the inference phase, the RDU is equally converted to simple dynamic convolutions that explicitly capture robust dynamic and static feature maps. Then, the information distillation block is constructed by several RDUs to enforce hierarchical refinement and selective fusion of spatial context information. Furthermore, we propose a dynamic distillation fusion (DDF) module to enable dynamic signals aggregation and communication between hierarchical modules to further improve performance. Empirical results show that our DDistill- SR outperforms the baselines and achieves state-of-the-art results on most super-resolution domains with much fewer parameters and less computational overhead. We have released the code of DDistill-SR at https://***/icandle/DDistill-SR. Copyright © 2023, The Authors. All rights reserved.

关键词： Convolution

Link Prediction on n-ary relational data based on relatedness evaluation

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Guan, Saiping Jin, Xiaolong Guo, Jiafeng Wang, Yuanzhuo Cheng, Xueqi School of Computer Science and Technology University of Chinese Academy of Sciences Beijing100049 China CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences Beijing100864 China

With the overwhelming popularity of Knowledge Graphs (KGs), researchers have poured attention to link prediction to fill in missing facts for a long time. However, they mainly focus on link prediction on binary relational data, where facts are usually represented as triples in the form of (head entity, relation, tail entity). In practice, n-ary relational facts are also ubiquitous. When encountering such facts, existing studies usually decompose them into triples by introducing a multitude of auxiliary virtual entities and additional triples. These conversions result in the complexity of carrying out link prediction on n-ary relational data. It has even proven that they may cause loss of structure information. To overcome these problems, in this paper, we represent each n-ary relational fact as a set of its role and role-value pairs. We then propose a method called NaLP to conduct link prediction on n-ary relational data, which explicitly models the relatedness of all the role and role-value pairs in an n-ary relational fact. We further extend NaLP by introducing type constraints of roles and role-values without any external type-specific supervision, and proposing a more reasonable negative sampling mechanism. Experimental results validate the effectiveness and merits of the proposed methods. Copyright © 2021, The Authors. All rights reserved.

关键词： Knowledge graph