检索结果-内蒙古大学图书馆

12th International Conference on Learning Representations, ICLR 2024

作者： Nahshan, Yury Kampeas, Joseph Haleva, Emir Distributed and Parallel Software Lab Huawei Technologies China

Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models. © 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Enhancing Code Representation Learning for Code Search with Abstract Code Semantics

Enhancing Code Representation Learning for Code Search with ...

引用

International Joint Conference on Neural Networks (IJCNN)

作者： Zhang, Shaojie Ding, Yiwei Hu, Enrui Yu, Yue Zhang, Yu Southern Univ Sci & Technol Dept Comp Sci & Engn Shenzhen Peoples R China Peng Cheng Lab Shenzhen Peoples R China Huawei Distributed & Parallel Software Lab Shenzhen Peoples R China

ISBN: (纸本)9798350359329;9798350359312

Code representation learning is an important way to encode the semantics of source code through pre-training. The learned representation supports a variety of downstream tasks, such as natural language code search and code defect detection. Inspired by pre-trained models for natural language representation learning, existing approaches often treat the source code or its structural information (e.g., Abstract Syntax Tree or AST) as a plain token sequence. Unlike natural language, programming language has its unique code unit information (e.g., identifiers and expressions) and logic information (e.g., the functionality of a code snippet). To further explore those properties, we propose Abstract Code Embedding (AbCE), a selfsupervised learning method that considers the abstract semantics of code logic. Instead of scattered tokens, AbCE treats an entire node or a subtree in an AST as a basic code unit during pre-training, which preserves the entirety of a coding unit. Moreover, AbCE learns the abstract semantics of AST nodes via a self-distillation way. Experimental results show that it achieves significant improvements over state-of-the-art baselines on code search tasks and comparable performance on code clone detection and defect detection tasks even without using contrastive learning or curriculum learning.

关键词： Natural Language Processing Code Comprehension Self-distillation Code Search

来源：评论

学校读者我要写书评

暂无评论

Rotation Invariant Quantization for Model Compression

arXiv

引用

arXiv 2023年

作者： Kampeas, Joseph Nahshan, Yury Kremer, Hanoch Lederman, Gil Zaloshinski, Shira Li, Zheng Haleva, Emir Distributed and Parallel Software Lab Huawei Technologies China

Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study, we investigate the rate-distortion tradeoff for NN model compression. First, we suggest a Rotation-Invariant Quantization (RIQ) technique that utilizes a single parameter to quantize the entire NN model, yielding a different rate at each layer, i.e., mixed-precision quantization. Then, we prove that our rotation-invariant approach is optimal in terms of compression. We rigorously evaluate RIQ and demonstrate its capabilities on various models and tasks. For example, RIQ facilitates ×19.4 and ×52.9 compression ratios on pre-trained VGG dense and pruned models, respectively, with © 2023, CC BY-NC-ND.

关键词： Neural networks

来源：评论

学校读者我要写书评

暂无评论

LINEAR LOG-NORMAL ATTENTION WITH UNBIASED CONCENTRATION

arXiv

引用

arXiv 2023年

作者： Nahshan, Yury Kampeas, Joseph Haleva, Emir Distributed and Parallel Software Lab Huawei Technologies United States

关键词： Scalability

来源：评论

学校读者我要写书评

暂无评论

Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering 60

Hyperlink-induced Pre-training for Passage Retrieval in Open...

引用

60th Annual Meeting of the Association-for-Computational-Linguistics (ACL)

作者： Zhou, Jiawei Li, Xiaoguang Shang, Lifeng Luo, Lan Zhan, Ke Hu, Enrui Zhang, Xinyu Jiang, Hao Cao, Zhao Yu, Fan Jiang, Xin Liu, Qun Chen, Lei Hong Kong Univ Sci & Technol Hong Kong Peoples R China Huawei Noahs Ark Lab Shenzhen Peoples R China Huawei Distributed & Parallel Software Lab Shenzhen Peoples R China

ISBN: (纸本)9781955917216

To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the HyperLink-induced Pre-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multihop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by more than 10 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios.

关键词： Zero-shot learning

来源：评论

学校读者我要写书评

暂无评论

PANGU-Σ: TOWARDS TRILLION PARAMETER LANGUAGE MODEL WITH SPARSE HETEROGENEOUS COMPUTING

arXiv

引用

arXiv 2023年

作者： Ren, Xiaozhe Zhou, Pingyi Meng, Xinfan Huang, Xinjing Wang, Yadao Wang, Weichao Li, Pengfei Zhang, Xiaoda Podolskiy, Alexander Arshinov, Grigory Bout, Andrey Piontkovskaya, Irina Wei, Jiansheng Jiang, Xin Su, Teng Liu, Qun Yao, Jun Noah’s Ark Lab Huawei Technologies China Distributed and Parallel Software Lab Huawei Technologies China

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors 2 and MindSpore framework 3, and present the language model with 1.085T parameters named PanGu-Σ. With parameter inherent from PanGu-α [1], we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation (ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-Σ provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation. Copyright © 2023, The Authors. All rights reserved.

关键词： Zero-shot learning

来源：评论

学校读者我要写书评

暂无评论

Dual Mutual Learning for Cross-Modality Person Re-Identification

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2022年第8期32卷 5361-5373页

作者： Zhang, Demao Zhang, Zhizhong Ju, Ying Wang, Cong Xie, Yuan Qu, Yanyun Xiamen Univ Sch Informat Xiamen 361005 Peoples R China East China Normal Univ Sch Comp Sci & Software Engn Shanghai 200050 Peoples R China Huawei Techol Distributed & Parallel Software Lab 2012 Labs Shenzhen 518129 Peoples R China

Cross-modality person re-identification (Re-ID) is more challenging than traditional visible Re-ID due to the huge cross-modality gap from heterogeneous images. To alleviate this problem, existing methods often utilize a dual path learning framework equipped with metric loss to learn discriminative features. Despite effectiveness, the inevitable degeneration of intra-modality discrimination by taking cross-modality discrimination into consideration is unsolvable. Such degeneration substantially hinders the model's capability of further improving feature representations. To mitigate this degeneration, we propose a Dual Mutual Learning (DML) method for cross-modality Re-ID which conducts mutual learning between the cross-modality and each of two single modalities. We design a triple-branch deep model containing the RGB and IR branches and the cross-modality branch. The cross-modality branch is designed to learn modality-invariant feature subspace for appearance similarity measurement. Both the RGB branch and IR branch provide attention supervision information to the cross-modality branch for attention feature alignment so as to enhance the intra-modality discrimination. Experimental results on two standard benchmarks demonstrate DML is superior to state-of-the-art methods.

关键词： Training Task analysis Surveillance Sensors Representation learning Knowledge engineering Convolutional neural networks Cross-modality person re-identification mutual learning attention alignment

来源：评论

学校读者我要写书评

暂无评论

Flora: Dual-Frequency LOss-Compensated ReAl-Time Monocular 3D Video Reconstruction 37

Flora: Dual-Frequency LOss-Compensated ReAl-Time Monocular 3...

引用

37th AAAI Conference on Artificial Intelligence (AAAI) / 35th Conference on Innovative Applications of Artificial Intelligence / 13th Symposium on Educational Advances in Artificial Intelligence

作者： Wang, Likang Gong, Yue Wang, Qirui Zhou, Kaixuan Chen, Lei Hong Kong Univ Sci & Technol Dept Comp Sci & Engn Hong Kong Peoples R China Hong Kong Univ Sci & Technol Data Sci & Analyt Thrust Guangzhou Peoples R China Huawei Technol Distributed & Parallel Software Lab Shenzhen Peoples R China Huawei Technol Riemann Lab Shenzhen Peoples R China Huawei Technol Fundamental Software Innovat Lab Shenzhen Peoples R China

ISBN: (纸本)9781577358800

In this work, we propose a real-time monocular 3D video reconstruction approach named Flora for reconstructing delicate and complete 3D scenes from RGB video sequences in an end-to-end manner. Specifically, we introduce a novel method with two main contributions. Firstly, the proposed feature aggregation module retains both color and reliability in a dual-frequency form. Secondly, the loss compensation module solves missing structure by correcting losses for falsely pruned voxels. The dual-frequency feature aggregation module enhances reconstruction quality in both precision and recall, and the loss compensation module benefits the recall. Notably, both proposed contributions achieve great results with negligible inferencing overhead. Our state-of-the-art experimental results on real-world datasets demonstrate Flora's leading performance in both effectiveness and efficiency. The code is available at https://***/NoOneUST/Flora.

关键词： Image reconstruction

来源：评论

学校读者我要写书评

暂无评论

Triple-Fact Retriever: An explainable reasoning retrieval model for multi-hop QA problem 38

Triple-Fact Retriever: An explainable reasoning retrieval mo...

引用

38th IEEE International Conference on Data Engineering (ICDE)

作者： Wu, Chengmin Hu, Enrui Zhan, Ke Luo, Lan Zhang, Xinyu Jiang, Hao Wang, Qirui Cao, Zhao Yu, Fan Chen, Lei Hong Kong Univ Sci & Technol Dept Comp Sci & Engn Hong Kong Peoples R China Huawei Distributed & Parallel Software Lab Shenzhen Peoples R China

ISBN: (数字)9781665408837

ISBN: (纸本)9781665408837

Nowadays, multi-hop question answer (QA) problem is challenging and not well solved in the QA community. The dominant bottleneck of the multi-hop QA problem is the need for a reasoning retriever to fetch a document path from an open-domain corpus (e.g., Wikipedia). A reasoning retriever aims to collect an evidence document from large corpora at one hop retrieval and aggregate the evidence for subsequent hop retrieval, which yields a document path after multi-hop retrieval. There exist two challenges, (1) to fetch the evidence document in an efficient and explainable way at one hop retrieval and (2) to update the question information by aggregating the evidence from the retrieved document after each hop retrieval. To address these two challenges, we propose a triple-fact-based retrieval model to effectively retrieve a related document path in an explainable way for each question. We extract a structured representation from the unstructured document and utilize the knowledge of pre-trained language model (PLM) to do the semantic-level matching between the question and document. We evaluate the proposed Triple-fact Retriever model on the recently proposed open-domain multi-hop QA dataset, HotpotQA, and a cross-document multi-step Reading Comprehension dataset, Wikihop. The results1 demonstrate that the Triple-fact retriever outperforms the existing baseline retrieval works.

关键词： Multi-hop QA a reasoning retriever triple-fact based model

来源：评论

学校读者我要写书评

暂无评论

IS-MVSNet:Importance Sampling-Based MVSNet 17th

IS-MVSNet:Importance Sampling-Based MVSNet

引用

17th European Conference on Computer Vision (ECCV)

作者： Wang, Likang Gong, Yue Ma, Xinjun Wang, Qirui Zhou, Kaixuan Chen, Lei Hong Kong Univ Sci & Technol Dept Comp Sci & Engn Clear Water Bay Hong Kong Peoples R China Huawei Technol Distributed & Parallel Software Lab Shenzhen Peoples R China Huawei Technol Riemann Lab Shenzhen Peoples R China Huawei Technol Fundamental Software Innovat Lab Shenzhen Peoples R China

ISBN: (纸本)9783031198236;9783031198243

This paper presents a novel coarse-to-fine multi-view stereo (MVS) algorithm called importance-sampling-based MVSNet (IS-MVSNet) to address a crucial problem of limited depth resolution adopted by current learning-based MVS methods. We proposed an importance-sampling module for sampling candidate depth, effectively achieving higher depth resolution and yielding better point-cloud results while introducing no additional cost. Furthermore, we proposed an unsupervised error distribution estimation method for adjusting the density variation of the importance-sampling module. Notably, the proposed sampling module does not require any additional training and works reasonably well with the pre-trained weights of the baseline model. Our proposed method leads to up to 20x promotion on the most refined depth resolution, thus significantly benefiting most scenarios and excellently superior on fine details. As a result, IS-MVSNet outperforms all the published papers on TNT's intermediate benchmark with an F-score of 62.82%. Code is available at ***/NoOneUST/IS-MVSNet.

关键词： 3D reconstruction Multi-view stereo Importance sampling

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：