检索结果-内蒙古大学图书馆

OLM2: Automatic Optimal Strategy Generating for Large-Scale Model Training with Limited-Memory

学校读者我要写书评

暂无评论

OLM2: Automatic Optimal Strategy Generating for Large-Scale ...

IEEE International Conference on Joint Cloud computing (JCC)

作者： Zhilin Yang Yu Tang Linbo Qiao Xi Yang Zhen Huang National Key Laboratory of Parallel and Distributed Computing College of Computer Science National University of Defense Technology Changsha 410073 China

The scale of model parameters and the amount of training data is exponentially increasing. It requires more GPU memory with the exponential increasement of model parameters. Recomputation and swapping are two main memory optimization methods that have been extensively studied, and there are also optimization strategies that combine the two methods. However, most of them are based on heuristic search strategies, which do not explore the complete solution space and can’t guarantee the optimality of the solution results. An optimal search strategy with tensor-level recomputation and swapping is expected in large-scale model training. In this paper, we propose an optimal strategy searching algorithm combining tensor-based recomputation and swapping. Specifically, the memory swapping strategy is reformulated as an optimization problem, which converts the memory constraints into mixed integer programming, to find the optimal memory optimization strategy. By leveraging the advantages of both recomputation and swapping, this approach minimizes computation consumption without exceeding the available memory limitation. Experimental results show that our method exhibits about 60% reduction in memory requirements during the training process. Furthermore, our method can reduce the overall training time beyond the existing algorithms. Compared to Checkmate, our approach achieves about 0.3–0.9% reduction in computation cost per iteration.

关键词：

KC-GenRe: A Knowledge-constrained Generative Re-ranking Method Based on Large Language Models for Knowledge Graph Completion

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Yilin Hu, Minghao Huang, Zhen Li, Dongsheng Yang, Dong Lu, Xicheng Defense Innovation Institute Academy of Military Sciences China Information Research Center of Military Science China National Key Laboratory of Parallel and Distributed Computing China

The goal of knowledge graph completion (KGC) is to predict missing facts among entities. Previous methods for KGC re-ranking are mostly built on non-generative language models to obtain the probability of each candidate. Recently, generative large language models (LLMs) have shown outstanding performance on several tasks such as information extraction and dialog systems. Leveraging them for KGC re-ranking is beneficial for leveraging the extensive pre-trained knowledge and powerful generative capabilities. However, it may encounter new problems when accomplishing the task, namely mismatch, misordering and omission. To this end, we introduce KC-GenRe, a knowledge-constrained generative re-ranking method based on LLMs for KGC. To overcome the mismatch issue, we formulate the KGC re-ranking task as a candidate identifier sorting generation problem implemented by generative LLMs. To tackle the misordering issue, we develop a knowledge-guided interactive training method that enhances the identification and ranking of candidates. To address the omission issue, we design a knowledge-augmented constrained inference method that enables contextual prompting and controlled generation, so as to obtain valid rankings. Experimental results show that KG-GenRe achieves state-of-the-art performance on four datasets, with gains of up to 6.7% and 7.7% in the MRR and Hits@1 metric compared to previous methods, and 9.0% and 11.1% compared to that without re-ranking. Extensive analysis demonstrates the effectiveness of components in KG-GenRe. Copyright © 2024, The Authors. All rights reserved.

关键词： Knowledge graph

Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： He, Yongquan Huang, Xuancheng Tang, Minghao Meng, Lingxun Li, Xiang Lin, Wei Zhang, Wenyuan Gao, Yifu Meituan China Institute of Information Engineering Chinese Academy of Sciences China National Key Laboratory of Parallel and Distributed Computing National University of Defense Technology China

Instruction tuning for large language models (LLMs) can drive them to produce results consistent with human goals in specific downstream tasks. However, the process of continual instruction tuning (CIT) for LLMs may bring about the catastrophic forgetting (CF) problem, where previously learned abilities are degraded. Recent methods try to alleviate the CF problem by modifying models or replaying data, which may only remember the surface-level pattern of instructions and get confused on held-out tasks. In this paper, we propose a novel continual instruction tuning method based on Key-part Information Gain (KPIG). Our method computes the information gain on masked parts to dynamically replay data and refine the training objective, which enables LLMs to capture task-aware information relevant to the correct response and alleviate overfitting to general descriptions in instructions. In addition, we propose two metrics, P-score and V-score, to measure the generalization and instruction-following abilities of LLMs. Experiments demonstrate our method achieves superior performance on both seen and held-out tasks. Copyright © 2024, The Authors. All rights reserved.

关键词： Digital storage

A Survey on Talking Head Generation: The Methods, Status and Challenges

学校读者我要写书评

暂无评论

SSRN

SSRN 2023年

作者： Cai, Yali Qiao, Peng Li, Dongsheng National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology Changsha410073 China

The talking head generation aims to synthesize a speech video of the source identity from a driving video or audio or text data irrelevant to the source identity. It can not only be applied to games and virtual reality applications, but also provide data for fake data detection. In recent years, the research of talking head generation is widely popular, and the authenticity of the generated results has also been greatly improved. However, the synthetic results still have great room for progress. We summarize the existing researches in this paper, hoping to offer assistance for later researchers. Furthermore, we divide these methods into three categories according to the input data type, namely video, audio and text driven talking head generation methods, and analyze them in detail. In addition, we also summarize the data sets commonly used in this kind of research and explore the evaluation criteria for measuring the performance of the method. Finally, the shortcomings of the existing methods in this field and the future direction are presented in last section. © 2023, The Authors. All rights reserved.

关键词： Deep learning

Deep Time Series Anomaly Detection with Local Temporal Pattern Learning

学校读者我要写书评

暂无评论

Deep Time Series Anomaly Detection with Local Temporal Patte...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Yizhou Li Yijie Wang Hongzuo Xu Xiaohui Zhou National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology Changsha China Intelligent Game and Decision Lab (IGDL) Beijing China

ISBN: (数字)9798350368741

ISBN: (纸本)9798350368758

Self-supervised time series anomaly detection (TSAD) demonstrates remarkable performance improvement by extracting high-level data semantics through proxy tasks. Nonetheless, most existing self-supervised TSAD techniques rely on manual- or neural-based transformations when designing proxy tasks, overlooking the intrinsic temporal patterns of time series. This paper proposes a local temporal pattern learning-based time series anomaly detection (LTPAD). LTPAD first generates sub-sequences. Pairwise sub-sequences naturally manifest proximity relationships along the time axis, and such correlations can be used to construct supervision and train neural networks to facilitate the learning of temporal patterns. Time intervals between two sub-sequences serve as labels for sub-sequence pairs. By classifying these labeled data pairs, our model captures the local temporal patterns of time series, thereby modeling the temporal pattern-aware "normality". Abnormal scores of testing data are acquired by evaluating their conformity to these learned patterns shared in training data. Extensive experiments show that LTPAD significantly outperforms state-of-the-art competitors.

关键词： Time series analysis Semantics Neural networks Training data Manuals Signal processing Data models Speech processing Anomaly detection Testing

A New Method for Evaluating Core Developers In Open Source Software 13

学校读者我要写书评

暂无评论

A New Method for Evaluating Core Developers In Open Source S...

13th IEEE International Conference on Software Engineering and Service Science, ICSESS 2022

作者： Song, Yulin Wang, Tao Shen, Yang Chang, Junsheng National University of Defense Technology National Laboratory for Parallel and Distributed Processing Hunan Province Changsha China National University of Defense Technology Institute for Quantum Information State Key Laboratory of High Performance Computing Hunan Province Changsha China

ISBN: (数字)9781665410328

ISBN: (纸本)9781665410311

The high-quality development of Open Source Software(OSS) benefits from the contributions of developers. An important research question is how to truly assess the extent of contribution of all developers in open source software and effectively identify the core developers to avoid their loss. In this paper, we use the undirected weighted network model to build a Developer Collaboration Network(DCN) based on developers and their contributions to the GitHub community's open source software Rails, analyse the characteristics of developer partnerships through the Developer Collaboration Network, and propose a Developer Personal Impact Factor(DPIF) to evaluate the importance of developer, select the top 10 core developers in Rails, focus on their participation in the open source software process, and predicting the changes of their contributions. The experimental results show that core developers with extensive partnerships are more likely to associate with developers with fewer partnerships. The average degree of developer contribution does not increase with more developers. Developer partnerships are characterized by close localization and loose overall relationships. The Developer Personal Impact Factor identifies the core developers in open source software well. The Developer Personal Impact Factor gives a ranking in which the core team members rank in the top 1.17% on average. © 2022 IEEE.

关键词： Open source software

Meta-Tsallis-Entropy Minimization: A New Self-Training Approach for Domain Adaptation on Text Classification

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Lu, Menglong Huang, Zhen Tian, Zhiliang Zhao, Yunxiang Fei, Xuanyu Li, Dongsheng National Key Laboratory of Parallel and Distributed Computing National University of Defense Technology China Beijing Institute of Biotechnology China Suiren Information China

Text classification is a fundamental task for natural language processing, and adapting text classification models across domains has broad applications. Self-training generates pseudo-examples from the model’s predictions and iteratively train on the pseudo-examples, i.e., mininizes the loss on the source domain and the Gibbs entropy on the target domain. However, Gibbs entropy is sensitive to prediction errors, and thus, self-training tends to fail when the domain shift is large. In this paper, we propose Meta-Tsallis Entropy minimization (MTEM), which applies meta-learning algorithm to optimize the instance adaptive Tsallis entropy on the target domain. To reduce the computation cost of MTEM, we propose an approximation technique to approximate the Second-order derivation involved in the meta-learning. To efficiently generate pseudo labels, we propose an annealing sampling mechanism for exploring the model’s prediction probability. Theoretically, we prove the convergence of the meta-learning algorithm in MTEM and analyze the effectiveness of MTEM in achieving domain adaptation. Experimentally, MTEM improves the adaptation performance of BERT with an average of 4 percent on the benchmark dataset. © 2023, CC BY-NC-SA.

关键词： Learning algorithms

XGrad: Boosting Gradient-Based Optimizers With Weight Prediction

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Guan, Lei Li, Dongsheng Shi, Yanqi Meng, Jian The Department of Mathematics National University of Defense Technology Changsha China The National Key Laboratory of Parallel and Distributed Computing National University of Defense Technology China

In this paper, we propose a general deep learning training framework XGrad which introduces weight prediction into the popular gradient-based optimizers to boost their convergence and generalization when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, the future weights are predicted according to the update rule of the used optimizer and are then applied to both the forward pass and backward propagation. In this way, during the whole training period, the optimizer always utilizes the gradients w.r.t. the future weights to update the DNN parameters, making the gradient-based optimizer achieve better convergence and generalization compared to the original optimizer without weight prediction. XGrad is rather straightforward to implement yet pretty effective in boosting the convergence of gradient-based optimizers and the accuracy of DNN models. Empirical results concerning five popular optimizers including SGD with momentum, Adam, AdamW, AdaBelief, and AdaM3 demonstrate the effectiveness of our proposal. The experimental results validate that XGrad can attain higher model accuracy than the baseline optimizers when training the DNN models. The code of XGrad will be available at: https://***/guanleics/XGrad. Copyright © 2023, The Authors. All rights reserved.

关键词： Forecasting

PTQ-SO: A Scale Optimization-based Approach for Post-training Quantization of Edge computing

学校读者我要写书评

暂无评论

PTQ-SO: A Scale Optimization-based Approach for Post-trainin...

International Conference on Computer Supported Cooperative Work in Design

作者： Kangkang Liu Ningjiang Chen College of Computer and Electronic Information Guangxi University Nanning China Education Department of Guangxi Zhuang Autonomous Region Key Laboratory of Parallel Distributed and Intelligent Computing Guangxi University Nanning China

ISBN: (数字)9798350349184

ISBN: (纸本)9798350349191

With the increasing performance of deep convolutional neural networks, they have been widely used in many computer vision tasks. However, a huge convolutional neural network model requires a lot of memory and computing resources, which makes it difficult to meet the requirements of low latency and reliability of edge computing when the model is deployed locally on resource-limited devices in edge environments. Quantization is a kind of model compression technology, which can effectively reduce model size, calculation cost and inference delay, but the quantization noise will cause the accuracy of the quantization model to decrease. Aiming at the problem of precision loss caused by model quantization, this paper proposes a post-training quantization method based on scale optimization. By reducing the influence of redundant parameters in the model on the quantization parameters in the process of model quantization, the scale factor optimization is realized to reduce the quantization error and thus improve the accuracy of the quantized model, reduce the inference delay and improve the reliability of edge applications. The experimental results show that under different quantization strategies and different quantization bit widths, the proposed method can improve the accuracy of the quantized model, and the absolute accuracy of the optimal quantization model is improved by 1.36%. The improvement effect is obvious, which is conducive to the application of deep neural network in edge environment.

关键词： Quantization (signal) Accuracy Computational modeling Computer network reliability Delays Reliability Convolutional neural networks