检索结果-内蒙古大学图书馆

Knowledge distillation of heterogeneous teacher-student model with intermediate layer loss 23

学校读者我要写书评

暂无评论

23rd Chinese National Conference on Computational Linguistics, CCL 2024

作者： Zhai, Feiyan Wang, Renzhi Li, Piji College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing211106 China

ISBN: (纸本)9780000000002

As a cutting-edge model compression strategy in the era of large language models, knowledge distillation technology significantly reduces the parameter scale and computational cost of models by effectively transferring the knowledge of complex models to simple models. However, the current mainstream generative large language model distillation algorithms mainly focus on optimizing the loss of the final output layer between the teacher-student models, and neglect the exploration of the model middle layer. In addition, the research on interlayer distillation often has strict requirements on the consistency of the structure of the teacher-student model, and can not deal with the distillation problem between heterogeneous models, so there are obvious limitations. To solve these problems, we propose a new knowledge distillation algorithm, which introduces the heterogeneous generative teacher-student large language model knowledge distillation algorithm with intermediate layer distillation loss. The algorithm first extracts the middle layer information of the teacher-student model as the distillation object, and then realizes the knowledge alignment and loss calculation between heterogeneous models based on the middle layer through the specially designed middle layer mapping rules and alignment module. Finally, the ratio of each distillation loss is optimized jointly. Through experimental validation on five relevant data sets, our method shows significant advantages in improving the distillation efficiency. © 2024 China National Conference on Computational Linguistics.

关键词： Students

ALiPy: Active learning in python

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Tang, Ying-Peng Li, Guo-Xiang Huang, Sheng-Jun College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing211106 China

Supervised machine learning methods usually require a large set of labeled examples for model training. However, in many real applications, there are plentiful unlabeled data but limited labeled data;and the acquisition of labels is costly. Active learning (AL) reduces the labeling cost by iteratively selecting the most valuable data to query their labels from the annotator. This article introduces a Python toobox ALiPy1 for active learning. ALiPy provides a module based implementation of active learning framework, which allows users to conveniently evaluate, compare and analyze the performance of active learning methods. In the toolbox, multiple options are available for each component of the learning framework, including data process, active selection, label query, results visualization, etc. In addition to the implementations of more than 20 state-of-the-art active learning algorithms, ALiPy also supports users to easily configure and implement their own approaches under different active learning settings, such as AL for multi-label data, AL with noisy annotators, AL with different costs and so on. The toolbox is well-documented and open-source on Github2, and can be easily installed through PyPI. Copyright © 2019, The Authors. All rights reserved.

关键词： Python

uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Li, Piji College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Jiangsu Nanjing China

The task of Chinese Spelling Check (CSC) is aiming to detect and correct spelling errors that can be found in the text. While manually annotating a high-quality dataset is expensive and time-consuming, thus the scale of the training dataset is usually very small (e.g., SIGHAN151 only contains 2339 samples for training), therefore supervised-learning based models usually suffer the data sparsity limitation and over-fitting issue, especially in the era of big language models. In this paper, we are dedicated to investigating the unsupervised paradigm to address the CSC problem and we propose a framework named uChecker to conduct unsupervised spelling error detection and correction. Masked pretrained language models such as BERT are introduced as the backbone model considering their powerful language diagnosis capability. Benefiting from the various and flexible MASKing operations, we propose a Confusionset-guided masking strategy to fine-train the masked language model to further improve the performance of unsupervised detection and correction. Experimental results on standard datasets demonstrate the effectiveness of our proposed model uChecker in terms of character-level and sentence-level Accuracy, Precision, Recall, and F1-Measure on tasks of spelling error detection and correction respectively. Copyright © 2022, The Authors. All rights reserved.

关键词： Error detection

Noise-Robust Bidirectional Learning with Dynamic Sample Reweighting

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zong, Chen-Chen Cao, Zheng-Tao Guo, Hong-Tao Du, Yun Xie, Ming-Kun Li, Shao-Yuan Huang, Sheng-Jun College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing211106 China

Deep neural networks trained with standard cross-entropy loss are more prone to memorize noisy labels, which degrades their performance. Negative learning using complementary labels is more robust when noisy labels intervene but with an extremely slow model convergence speed. In this paper, we first introduce a bidirectional learning scheme, where positive learning ensures convergence speed while negative learning robustly copes with label noise. Further, a dynamic sample reweighting strategy is proposed to globally weaken the effect of noise-labeled samples by exploiting the excellent discriminatory ability of negative learning on the sample probability distribution. In addition, we combine self-distillation to further improve the model performance. The code is available at https://***/chenchenzong/BLDR. © 2022, CC BY.

关键词： Distillation

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts, prompting the formulation of lifelong model editing task. While recent years have witnessed the development of various techniques for single and batch editing, these methods either fail to apply or perform sub-optimally when faced with lifelong editing. In this paper, we introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing. We first analyze the factors influencing the effectiveness of conventional MoE adaptor in lifelong editing, including catastrophic forgetting, inconsistent routing and order sensitivity. Based on these insights, we propose a tailored module insertion method to achieve lifelong editing, incorporating a novel KV anchor routing to enhance routing consistency between training and inference stage, along with a concise yet effective clustering-based editing order planning. Experimental results demonstrate the effectiveness of our method in lifelong editing, surpassing previous model editing techniques while maintaining outstanding performance in batch editing task. Our code will be available. © 2024, CC BY.

关键词： Computational linguistics

A Multi-Scale Multi-Hop Graph Convolution Network for Predicting Fluid intelligence via Functional Connectivity

学校读者我要写书评

暂无评论

A Multi-Scale Multi-Hop Graph Convolution Network for Predic...

IEEE International Conference on Bioinformatics and Biomedicine (BIBM)

作者： Xuyun Wen Qumei Cao Daoqiang Zhang College of Computer Science and Technology MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing University of Aeronautics and Astronautics Nanjing Jiangsu China

ISBN: (数字)9781665468190

ISBN: (纸本)9781665468206

Predicting fluid intelligence via neuroimaging data is important to understand neural mechanisms underlying diverse complex cognitive tasks in human brain. Functional connectivity (FC) reflects interactions among brain regions providing rich information of brain organization, which has been widely used in various behavior predictions. With the success of deep neural networks, graph convolutional network (GCN) is regarded as a promising feature learning method in FC networks (FCNs). However, as a challenging task, the existing GCN models cannot achieve a satisfactory performance in fluid intelligence predication due to the insufficient information utilization of brain connectivity and the limitation of graph convolution layer. To tackle these problems, this paper developed a Multi-Scale Multi-Hop GCN (MS-MH-GCN) to estimate fluid intelligence score by using FC. In the proposed method, we considered the hierarchy of brain system and thus utilized FCs from multiple spatial scales as input for the subsequent feature learning to achieve a complete characterize of brain organization for each individual. We also designed a new multi-hop graph convolution layer that uses multi-hop neighbors instead of l-hop neighbor in traditional GCN to guide message passing of nodal feature at every step. The introduction of high-order graph information benefits to the model learning ability improvement. Additionally, it is also worth emphasizing that, during feature learning process, we added contrast constraint to multi-scale FCNs to improve the similarity of feature representations across different spatial scales within a subject. Experimental results showed that our proposed method performed much better than the other four art-of-the-state methods.

关键词： Representation learning Neuroimaging Fluids Convolution Message passing Neural networks Spread spectrum communication

5W1H Extraction With Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Cao, Yang Lan, Yangsong Zhai, Feiyan Li, Piji College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics China MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing China

The extraction of essential news elements through the 5W1H framework (What, When, Where, Why, Who, and How) is critical for event extraction and text summarization. The advent of Large language models (LLMs) such as ChatGPT presents an opportunity to address language-related tasks through simple prompts without fine-tuning models with much time. While ChatGPT has encountered challenges in processing longer news texts and analyzing specific attributes in context, especially answering questions about What, Why, and How. The effectiveness of extraction tasks is notably dependent on high-quality human-annotated datasets. However, the absence of such datasets for the 5W1H extraction increases the difficulty of finetuning strategies based on open-source LLMs. To address these limitations, first, we annotate a high-quality 5W1H dataset based on four typical news corpora (CNN/DailyMail, XSum, NYT, RA-MDS);second, we design several strategies from zero-shot/few-shot prompting to efficient fine-tuning to conduct 5W1H aspects extraction from the original news documents. The experimental results demonstrate that the performance of the fine-tuned models on our labelled dataset is superior to the performance of ChatGPT. Furthermore, we also explore the domain adaptation capability by testing the source-domain (e.g. NYT) models on the target domain corpus (e.g. CNN/DailyMail) for the task of 5W1H extraction. © 2024, CC0.

关键词： Extraction

Tumor Micro-Environment Interactions Guided Graph Learning for Survival analysis of Human Cancers from Whole-Slide Pathological Images

学校读者我要写书评

暂无评论

Tumor Micro-Environment Interactions Guided Graph Learning f...

Conference on Computer Vision and pattern Recognition (CVPR)

作者： Wei Shao YangYang Shi Daoqiang Zhang JunJie Zhou Peng Wan Ministry of Education College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics Key Laboratory of Brain-Machine Intelligence Technology Nanjing China MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing China

ISBN: (数字)9798350353006

ISBN: (纸本)9798350353013

The recent advance of deep learning technology brings the possibility of assisting the pathologist to predict the patients' survival from whole-slide pathological images (WSIs). However, most of the prevalent methods only worked on the sampled patches in specifically or randomly selected tumor areas of WSIs, which has very limited capability to capture the complex interactions between tumor and its surrounding micro-environment components. As a matter of fact, tumor is supported and nurtured in the heterogeneous tumor micro-environment(TME), and the detailed analysis of TME and their correlation with tumors are important to in-depth analyze the mechanism of cancer development. In this paper, we considered the spatial interactions among tumor and its two major TME components (i.e., lymphocytes and stromal fibrosis) and presented a Tumor Micro-environment Interactions Guided Graph Learning (TMEGL) algorithm for the prognosis prediction of human cancers. Specifically, we firstly selected different types of patches as nodes to build graph for each WSI. Then, a novel TME neighborhood organization guided graph embedding algorithm was proposed to learn node representations that can preserve their topological structure information. Finally, a Gated Graph Attention Network is applied to capture the survival-associated intersections among tumor and different TME components for clinical outcome prediction. We tested TMEGL on three cancer cohorts derived from The Cancer Genome Atlas (TCGA), and the experimental results indicated that TMEGL not only outperforms the existing WSI-based survival analysis models, but also has good explainable ability for survival prediction.

关键词： Pathology Analytical models Organizations Predictive models Semisupervised learning Prediction algorithms pattern recognition

MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors

学校读者我要写书评

暂无评论

arXiv 2024年

Model editing aims to efficiently alter the behavior of Large Language Models (LLMs) within a desired scope, while ensuring no adverse impact on other inputs. Recent years have witnessed various model editing methods been proposed. However, these methods either exhibit poor overall performance or struggle to strike a balance between generalization and locality. We propose MEMoE, a model editing adapter utilizing a Mixture of Experts (MoE) architecture with a knowledge anchor routing strategy. MEMoE updates knowledge using a bypass MoE structure, keeping the original parameters unchanged to preserve the general ability of LLMs. And, the knowledge anchor routing ensures that inputs requiring similar knowledge are routed to the same expert, thereby enhancing the generalization of the updated knowledge. Experimental results show the superiority of our approach over both batch editing and sequential batch editing tasks, exhibiting exceptional overall performance alongside outstanding balance between generalization and locality. Our code will be available. © 2024, CC BY.

关键词：