检索结果-内蒙古大学图书馆

Multi-Task Transformer with Relation-Attention and Type-Attention for Named Entity Recognition

学校读者我要写书评

暂无评论

Multi-Task Transformer with Relation-Attention and Type-Atte...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Ying Mo Hongyin Tang Jiahao Liu Qifan Wang Zenglin Xu Jingang Wang Wei Wu Zhoujun Li State Key Lab of Software Development Environment Beihang University Beijing China Meituan Beijing China Meta AI New York United States Harbin Institute of Technology Shenzhen China

Named entity recognition (NER) is an important research problem in natural language processing. There are three types of NER tasks, including flat, nested and discontinuous entity recognition. Most previous sequential labeling models are task-specific, while recent years have witnessed the rising of generative models due to the advantage of unifying all NER tasks into the seq2seq model framework. Although achieving promising performance, our pilot studies demonstrate that existing generative models are ineffective at detecting entity boundaries and estimating entity types. This paper proposes a multi-task Transformer, which incorporates an entity boundary detection task into the named entity recognition task. More concretely, we achieve entity boundary detection by classifying the relations between tokens within the sentence. To improve the accuracy of entity-type mapping during decoding, we adopt an external knowledge base to calculate the prior entity-type distributions and then incorporate the information into the model via the self and cross-attention mechanisms. We perform experiments on an extensive set of NER benchmarks, including two flat, three nested, and three discontinuous NER datasets. Experimental results show that our approach considerably improves the generative NER model’s performance.

关键词： Speech recognition Signal processing Predictive models Transformers Multitasking Natural language processing Decoding

MULTI-TASK TRANSFORMER WITH RELATION-ATTENTION AND TYPE-ATTENTION FOR NAMED ENTITY RECOGNITION

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Mo, Ying Tang, Hongyin Liu, Jiahao Wang, Qifan Xu, Zenglin Wang, Jingang Wu, Wei Li, Zhoujun State Key Lab of Software Development Environment Beihang University Beijing China Meituan Beijing China Meta AI New York United States Harbin Institute of Technology Shenzhen China

关键词： Knowledge based systems

Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wei, Xiuying Zhang, Yunchen Li, Yuhang Zhang, Xiangguo Gong, Ruihao Guo, Jinyang Liu, Xianglong State Key Lab of Software Development Environment Beihang University China SenseTime Research China School of Computer and Communication Sciences EPFL Switzerland Yale University United States UESTC China

Post-training quantization (PTQ) of transformer language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outliers are concentrated in specific channels and are asymmetric across channels. To address this issue, we propose the Outlier Suppression+ (OS+) framework, which contains the channel-wise shifting for asymmetry and channel-wise scaling for concentration. We show that these operations can be seamlessly migrated into subsequent modules while maintaining equivalence. Second, we propose a fast and stable scheme to calculate effective shifting and scaling values. The channel-wise shifting aligns the center of each channel for removal of outlier asymmetry. The channel-wise scaling quantitatively evaluates changes brought by migration and quantization for better quantization burden balance. We validate our OS+ under both standard and fine-grained quantization settings with models including BERT, OPT, BLOOM, BLOOMZ, and LLaMA. Comprehensive results across various tasks demonstrate the superiority of our approach. Especially, with standard quantization, OS+ can achieve near-floating-point performance on both small models and large language models on 8-bit and 6-bit. Besides, we establish a new state-of-the-art for 4-bit BERT with 15.5% improvement. Our code is available at https://***/ModelTC/Outlier_Suppression_Plus. Copyright © 2023, The Authors. All rights reserved.

关键词： Digital arithmetic

Effective Reinforcement Learning Based on Structural Information Principles

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zeng, Xianghua Peng, Hao Su, Dingli Li, Angsheng State Key Laboratory of Software Development Environment School of Computer Science and Engineering Beihang University Beijing100191 China School of Cyber Science and Technology Beihang University Beijing100191 China

Although Reinforcement Learning (RL) algorithms acquire sequential behavioral patterns through interactions with the environment, their effectiveness in noisy and high-dimensional scenarios typically relies on specific structural priors. In this paper, we propose a novel and general Structural Information principles-based framework for effective Decision-Making, namely SIDM, approached from an information-theoretic perspective. This paper presents a specific unsupervised partitioning method that forms vertex communities in the state and action spaces based on their feature similarities. An aggregation function, which utilizes structural entropy as the vertex weight, is devised within each community to obtain its embedding, thereby facilitating hierarchical state and action abstractions. By extracting abstract elements from historical trajectories, a directed, weighted, homogeneous transition graph is constructed. The minimization of this graph’s high-dimensional entropy leads to the generation of an optimal encoding tree. An innovative two-layer skill-based learning mechanism is introduced to compute the common path entropy of each state transition as its identified probability, thereby obviating the requirement for expert knowledge. Moreover, SIDM can be flexibly incorporated into various single-agent and multi-agent RL algorithms, enhancing their performance. Finally, extensive evaluations on challenging benchmarks demonstrate that, compared with SOTA baselines, our framework significantly and consistently improves the policy’s quality, stability, and efficiency up to 32.70%, 88.26%, and 64.86%, respectively. Copyright © 2024, The Authors. All rights reserved.

关键词： Entropy

Effective Training of PINNs by Combining CMA-ES with Gradient Descent

学校读者我要写书评

暂无评论

Effective Training of PINNs by Combining CMA-ES with Gradien...

Congress on Evolutionary Computation

作者： Lin Liu Yuan Yuan School of Computer Science and Engineering Beihang University Beijing China State Key Laboratory of Software Development Environment Zhongguancun Laboratory School of Computer Science and Engineering Beihang University Beijing China

ISBN: (数字)9798350308365

ISBN: (纸本)9798350308372

Physics-Informed Neural Networks (PINNs) have recently received increasing attention, however, optimizing the loss function of PINNs is notoriously difficult, where the landscape of the loss function is often highly non-convex and rugged. Local optimization methods based on gradient information can converge quickly but are prone to being trapped in local minima for training PINNs. Evolutionary algorithms (EAs) are well known for the global search ability, which can help escape from local minima. It has been reported in the literature that EAs show some advantages over gradient-based methods in training PINNs. Inspired by the Memetic Algorithm, we combine global-search based EAs and local-search based batch gradient descent in order to make the best of both word. In addition, since the PINN loss function is composed of multiple terms, balancing these terms is also a challenging issue. Therefore, we also attempt to combine EAs with multiple-gradient descent algorithm for multi-objective optimization. Our experiments provide strong evidence for the superiority of the above algorithms.

关键词： Training Memetics Neural networks Evolutionary computation Pareto optimization Prediction algorithms Linear programming

Improving neural machine translation with soft template prediction 58

学校读者我要写书评

暂无评论

Improving neural machine translation with soft template pred...

58th Annual Meeting of the Association for Computational Linguistics, ACL 2020

作者： Yang, Jian Ma, Shuming Zhang, Dongdong Li, Zhoujun Zhou, Ming State Key Lab of Software Development Environment Beihang University Microsoft Research Asia

ISBN: (纸本)9781952148255

Although neural machine translation (NMT) has achieved significant progress in recent years, most previous NMT models only depend on the source text to generate translation. Inspired by the success of template-based and syntax-based approaches in other fields, we propose to use extracted templates from tree structures as soft target templates to guide the translation procedure. In order to learn the syntactic structure of the target sentences, we adopt the constituency-based parse tree to generate candidate templates. We incorporate the template information into the encoder-decoder framework to jointly utilize the templates and source text. Experiments show that our model significantly outperforms the baseline models on four benchmarks and demonstrate the effectiveness of soft target templates. © 2020 Association for Computational Linguistics

关键词： Neural machine translation

Multi-pretext attention network for few-shot learning with self-supervision

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Li, Hainan Tao, Renshuai Li, Jun Qin, Haotong Ding, Yifu Wang, Shuo Liu, Xianglong State Key Lab of Software Development Environment Beihang University China

Few-shot learning is an interesting and challenging study, which enables machines to learn from few samples like humans. Existing studies rarely exploit auxiliary information from large amount of unlabeled data. Self-supervised learning is emerged as an efficient method to utilize unlabeled data. Existing self-supervised learning methods always rely on the combination of geometric transformations for the single sample by augmentation, while seriously neglect the endogenous correlation information among different samples that is the same important for the task. In this work, we propose a Graph-driven Clustering (GC), a novel augmentation-free method for self-supervised learning, which does not rely on any auxiliary sample and utilizes the endogenous correlation information among input samples. Besides, we propose Multi-pretext Attention Network (MAN), which exploits a specific attention mechanism to combine the traditional augmentation-relied methods and our GC, adaptively learning their optimized weights to improve the performance and enabling the feature extractor to obtain more universal representations. We evaluate our MAN extensively on miniImageNet and tieredImageNet datasets and the results demonstrate that the proposed method outperforms the state-of-the-art (SOTA) relevant methods. © 2021, CC BY-NC-SA.

关键词： Machine learning

Online Task Offloading and Resource Allocation in Two-Tier Mobile-Edge Computing Network

学校读者我要写书评

暂无评论

Online Task Offloading and Resource Allocation in Two-Tier M...

第42届中国控制会议

作者： Xiaofeng Li Kexin Liu School of Automation Science and Electrical Engineering Beihang University School of Automation Science and Electrical Engineering State Key Laboratory of Software Development Environment Beihang University

This work focuses on the tasks offloading and resource allocation based on cloud-edge computing. We consider a two-tier multi-user mobile-edge computing(MEC) network, where wireless devices can offload the task data to edge servers or the cloud center. Our goal is to find optimal offloading and resource allocation decisions to maximize the average computation rate while reducing the average cost of edge and cloud services, subject to the stability of the task queue and average power ***, the problem is transformed into a deterministic problem for each time frame utilizing the Lyapunov optimization ***, an algorithm is proposed, combined with deep reinforcement learning(DRL), where DRL is used to make offloading decisions. This problem can be easily solved once offloading decisions are known. Finally, a numerical simulation is given to evaluate the algorithm's performance, which shows that our algorithm solves well and takes less time to compute.

关键词：

Hu-Fu: Efficient and Secure Spatial Queries over Data Federation 48th

学校读者我要写书评

暂无评论

Hu-Fu: Efficient and Secure Spatial Queries over Data Federa...

48th International Conference on Very Large Data Bases, VLDB 2022

作者： Tong, Yongxin Pan, Xuchen Zeng, Yuxiang Shi, Yexuan Xue, Chunbo Zhou, Zimu Zhang, Xiaofei Chen, Lei Xu, Yi Xu, Ke Lv, Weifeng State Key Laboratory of Software Development Environment Beihang University China Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing Beihang University China The Hong Kong University of Science and Technology Singapore Management University University of Memphis

Data isolation has become an obstacle to scale up query processing over big data, since sharing raw data among data owners is often prohibitive due to security concerns. A promising solution is to perform secure queries over a federation of multiple data owners leveraging secure multi-party computation (SMC) techniques, as evidenced by recent federation work over relational data. However, existing solutions are highly inefficient on spatial queries due to excessive secure distance operations for query processing and their usage of general-purpose SMC libraries for secure operation implementation. In this paper, we propose Hu-Fu, the first system for efficient and secure spatial query processing on a data federation. The idea is to decompose the secure processing of a spatial query into as many plaintext operations and as few secure operations as possible, where fewer secure operators are involved and all secure operators are implemented dedicatedly. As a working system, Hu-Fu supports not only query input in native SQL, but also heterogeneous spatial databases (e.g., PostGIS, Simba, GeoMesa, and SpatialHadoop) at the backend. Extensive experiments show that Hu-Fu usually outperforms the state-of-the-arts in running time and communication cost while guaranteeing security. © 2022, American Mathematical Society. All rights reserved.

关键词：