检索结果-内蒙古大学图书馆

KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation

KnowLA: Enhancing Parameter-efficient Finetuning with Knowle...

2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024

作者： Luo, Xindi Sun, Zequn Zhao, Jing Zhao, Zhe Hu, Wei State Key Laboratory for Novel Software Technology Nanjing University China National Institute of Healthcare Data Science Nanjing University China Tencent AI Lab China

ISBN: (纸本)9798891761148

Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that KnowLA can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts. © 2024 Association for Computational Linguistics.

关键词： Knowledge graph

来源：评论

学校读者我要写书评

暂无评论

AI Computing Systems for Large Language Models Training

引用

计算机科学技术学报（英文版） 2025年第1期40卷 6-41页

作者：张振兴文渊博吕涵祺刘畅张蕊李夏青王超杜子东郭崎李玲周学海陈云霁 1School of Computer Science and Technology University of Science and Technology of China Hefei 230026 China 2State Key Laboratory of Processors Institute of Computing Technology Chinese Academy of Sciences Beijing 100190 China 3Cambricon Technologies Beijing 100191 China 4Shanghai Innovation Center for Processor Technologies Shanghai 201210 China 5Intelligent Software Research Center Institute of Software Chinese Academy of Sciences Beijing 100190 China 6University of Chinese Academy of Sciences Beijing 101408 China

In this paper,we present a comprehensive overview of artificial intelligence(AI)computing systems for large language models(LLMs)*** rapid advancement of LLMs in recent years,coupled with the widespread adoption of algorithms and applications such as BERT,ChatGPT,and DeepSeek,has sparked significant interest in this *** classify LLMs into encoder-only,encoder-decoder,and decoder-only models,and briefly analyze their training and infer-ence processes to emphasize their substantial need for computational *** operations depend heavily on AI-specific accelerators like GPUs(graphics processing units),TPUs(tensor processing units),and MLUs(machine learning units).However,as the gap widens between the increasing complexity of LLMs and the current capabilities of accelerators,it becomes essential to adopt heterogeneous computing systems optimized for distributed environments to manage the growing computational and memory requirements of *** delve into the execution and scheduling of LLM algo-rithms,underlining the critical role of distributed computing strategies,memory management enhancements,and boosting computational *** paper clarifies the complex relationship between algorithm design,hardware infrastructure,and software optimization,and provides an in-depth understanding of both the software and hardware infrastructure sup-porting LLMs training,offering insights into the challenges and potential avenues for future development and deployment.

关键词： artificial intelligence(AI)chip large language model(LLM) AI computing system accelerator

来源：评论

学校读者我要写书评

暂无评论

TSCMR:Two-Stage Cross-Modal Retrieval 9th

TSCMR:Two-Stage Cross-Modal Retrieval

引用

19th International Conference on Advanced Data Mining and Applications, ADMA 2023

作者： Chen, Zhihao Wang, Hongya School of Computer Science and Technology Donghua University Shanghai China State Key Laboratory of Computer Architecture Institute of Computing Technology CAS Beijing China Shanghai Key Laboratory of Computer Software Evaluating and Testing Shanghai China

ISBN: (纸本)9783031466731

Currently, large-scale vision and language models has significantly improved the performances of cross-modal retrieval tasks. However, large-scale models require a substantial amount of computing resources, so the execution of these models on devices with limited resources is challenging. Thus, it is paramount to reduce the model size and minimize computing costs of a model without sacrificing its performance. In this paper, we improved TERAN by dividing cross-modal retrieval into two stages: image-text coarse-grained matching and image-text fine-grained matching. Specifically, we present a novel approach called Two-Stage Cross-Modal Retrieval network(TSCMR). To reduce model size after model training, our approach utilized a new knowledge distillation method for Transformer-based models. Experiments have shown that our approach maintains a performance comparable to TERAN on the MS-COCO 1K test set, while being 2x smaller and 3.1x faster on inference. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.

关键词： Distillation

来源：评论

学校读者我要写书评

暂无评论

Formally Verified C Code Generation from Hybrid Communicating Sequential Processes

arXiv

引用

arXiv 2024年

作者： Wang, Shuling Ji, Zekun Zhan, Bohua Xu, Xiong Gao, Qiang Zhan, Naijun State Key Lab. of Computer Science Institute of Software Chinese Academy of Sciences Beijing China

Hybrid Communicating Sequential Processes (HCSP) is a formal model for hybrid systems, including primitives for evolution along an ordinary differential equation (ODE), communication, and parallel composition. Code generation is needed to convert HCSP models into code that can be executed in practice, and the correctness of this conversion is essential to ensure that the generated code accurately reflects the formal model. In this paper, we propose a code generation algorithm from HCSP to C with POSIX library for concurrency. The main difficulties include how to bridge the gap between the synchronized communication model in HCSP and the use of mutexes for synchronization in C, and how to discretize evolution along ODEs and support interrupt of ODE evolution by communication. To prove the correctness of code generation, we define a formal semantics for POSIX C, and build transition system models for both HCSP and C programs. We then define an approximate bisimulation relation between traces of transition systems, and show that under certain robustness conditions for HCSP, the generated C program is approximately bisimilar to the original model. Finally, we evaluate the code generation algorithm on a detailed model for automatic cruise control, showing its utility on real-world examples. Copyright © 2024, The Authors. All rights reserved.

关键词： Hybrid systems

来源：评论

学校读者我要写书评

暂无评论

Enhancing Offensive Language Detection with Data Augmentation and Knowledge Distillation

引用

Research 2024年第2期6卷 29-40页

作者： Jiawen Deng Zhuang Chen Hao Sun Zhexin Zhang Jincenzi Wu Satoshi Nakagawa Fuji Ren Minlie Huang The CoAl group DCSTInstitute for Artificial InteligenceState Key Lab of Intelligent Technology and SystemsBeijing National Research Center for Information Science and TechnologyTsinghua UniversityBeijing 100084China Graduate School of Information Science&Technology The University of Tokyo.Tokyo 1138654Japan School of Computer Science and Engineering University of Electronic Science and Technology of ChinaChengduSichuan611731China

Offensive language detection has received important attention and plays a crucial role in promoting healthy communication on social platforms,as well as promoting the safe deployment of large language *** data is the basis for developing detectors;however,the available offense-related dataset in Chinese is severely limited in terms of data scale and coverage when compared to English *** significantly affects the accuracy of Chinese offensive language detectors in practical applications,especially when dealing with hard cases or out-of-domain *** alleviate the limitations posed by available datasets,we introduce AugCOLD(Augmented Chinese Offensive Language Dataset),a large-scale unsupervised dataset containing 1 million samples gathered by data crawling and model ***,we employ a multiteacher distillation framework to enhance detection performance with unsupervised *** is,we build multiple teachers with publicly accessible datasets and use them to assign soft labels to *** soft labels serve as a bridge for knowledge to be distilled from both AugCOLD and multiteacher to the student network,i.e.,the final offensive *** conduct experiments on multiple public test sets and our well-designed hard tests,demonstrating that our proposal can effectively improve the generalization and robustness of the offensive language detector.

关键词： COLD dealing limitations

来源：评论

学校读者我要写书评

暂无评论

Bug localization based on syntactical and semantic information of source code

引用

Journal of Systems Engineering and Electronics 2023年第1期34卷 236-246页

作者： YAN Xuefeng CHENG Shasha GUO Liqin College of Computer Science Technology Nanjing University of Aeronautics and AstronauticsNanjing 211106China Collaborative Innovation Center of Novel Software Technology and Industrialization Nanjing 211106China State Key Laboratory of Intelligent Manufacturing System Technology Beijing Institute of Electronic System EngineeringBeijing 100854China

The existing software bug localization models treat the source file as natural language, which leads to the loss of syntactical and structure information of the source file. A bug localization model based on syntactical and semantic information of source code is proposed. Firstly, abstract syntax tree(AST) is divided based on node category to obtain statement sequence. The statement tree is encoded into vectors to capture lexical and syntactical knowledge at the statement ***, the source code is transformed into vector representation by the sequence naturalness of the statement. Therefore,the problem of gradient vanishing and explosion caused by a large AST size is obviated when using AST to the represent source code. Finally, the correlation between bug reports and source files are comprehensively analyzed from three aspects of syntax, semantics and text to locate the buggy code. Experiments show that compared with other standard models, the proposed model improves the performance of bug localization, and it has good advantages in mean reciprocal rank(MRR), mean average precision(MAP) and Top N Rank.

关键词： bug report abstract syntax tree code representation software bug localization

来源：评论

学校读者我要写书评

暂无评论

Differentially Testing Database Transactions for Fun and Profit 22

Differentially Testing Database Transactions for Fun and Pro...

引用

Proceedings of the 37th IEEE/ACM International Conference on Automated software Engineering

作者： Ziyu Cui Wensheng Dou Qianwang Dai Jiansen Song Wei Wang Jun Wei Dan Ye State Key Lab of Computer Science Institute of Software Chinese Academy of Science China

ISBN: (纸本)9781450394758

Database Management Systems (DBMSs) utilize transactions to ensure the consistency and integrity of data. Incorrect transaction implementations in DBMSs can lead to severe consequences, e.g., incorrect database states and query results. Therefore, it is critical to ensure the reliability of transaction implementations. In this paper, we propose DT2, an approach for automatically testing transaction implementations in DBMSs. We first randomly generate a database and a group of concurrent transactions operating the database, which can support complex features in DBMSs, e.g., various database schemas and cross-table queries. We then leverage differential testing to compare transaction execution results on multiple DBMSs to find discrepancies. The non-determinism of concurrent transactions can affect the effectiveness of our method. Therefore, we propose a transaction test protocol to ensure the deterministic execution of concurrent transactions. We evaluate DT2 on three widely-used MySQL-compatible DBMSs: MySQL, MariaDB and TiDB. In total, we have detected 10 unique transaction bugs and 88 transaction-related compatibility issues from the observed discrepancies. Our empirical study on these compatibility issues shows that DBMSs suffer from various transaction-related compatibility issues, although they claim that they are compatible. These compatibility issues can also lead to serious consequences, e.g., inconsistent database states among DBMSs.

关键词： compatibility issue Database transaction isolation level differential testing

来源：评论

学校读者我要写书评

暂无评论

Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting 38

Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynami...

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Xu, Jiawei Fan, Zexin Yang, Jian Xie, Jin PCA Lab VCIP College of Computer Science Nankai University China State Key Laboratory for Novel Software Technology Nanjing University Nanjing China School of Intelligence Science and Technology Nanjing University Suzhou China

Recently, Gaussian splatting has received more and more attention in the field of static scene rendering. Due to the low computational overhead and inherent flexibility of explicit representations, plane-based explicit methods are popular ways to predict deformations for Gaussian-based dynamic scene rendering models. However, plane-based methods rely on the inappropriate low-rank assumption and excessively decompose the space-time 4D encoding, resulting in overmuch feature overlap and unsatisfactory rendering quality. To tackle these problems, we propose Grid4D, a dynamic scene rendering model based on Gaussian splatting and employing a novel explicit encoding method for the 4D input through the hash encoding. Different from plane-based explicit representations, we decompose the 4D encoding into one spatial and three temporal 3D hash encodings without the low-rank assumption. Additionally, we design a novel attention module that generates the attention scores in a directional range to aggregate the spatial and temporal features. The directional attention enables Grid4D to more accurately fit the diverse deformations across distinct scene components based on the spatial encoded features. Moreover, to mitigate the inherent lack of smoothness in explicit representation methods, we introduce a smooth regularization term that keeps our model from the chaos of deformation prediction. Our experiments demonstrate that Grid4D significantly outperforms the state-of-the-art models in visual quality and rendering speed. Project page: https://***/Grid4D-web/. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Deep Learning Based Take-Over Performance Prediction and Its Application on Intelligent Vehicles

IEEE Transactions on Intelligent Vehicles

引用

IEEE Transactions on Intelligent Vehicles 2024年 1-15页

作者： Liu, Weimin Li, Qingkun Wang, Wenjun Wang, Zhenyuan Zeng, Chao Cheng, Bo State Key Laboratory of Automotive Safety and Energy Center for Intelligent Connected Vehicles and Transportation School of Vehicle and Mobility Tsinghua University Beijing China Beijing Key Laboratory of Human-Computer Interaction Institute of Software Chinese Academy of Sciences Beijing China College of Information Science and Engineering Henan University of Technology Zhengzhou China

Take-over performance plays a significant role in evaluating drivers' state, and serves as a crucial reference for enhancing control transitions in the context of conditionally automated driving. In this study, we aim to predict minimum anticipated collision time (min ACT), an indicator of drivers' take-over performance, in expectation of promoting safer take-overs via deep learning, so that drivers' state detriment of take-over safety could be adjusted accordingly with intelligent human-machine interaction algorithms predictably. By incorporating multi-source information including drivers' state, drivers' demographics, surrounding traffic features as well as driver-vehicle interaction characteristics, network model “ACTNet” was proposed to facilitate continuous estimation. Depthwise separable convolution and non-local self-attention were utilized to prevent overfitting and establish spatial dependency over fixation heatmap, respectively. To overcome data distribution imbalance, class balanced loss was used in conjunction with regression loss to realize more accurate predictions. Driving simulator experiment was conducted with dataset collected for the subsequent verification of the proposed algorithm. Potentialities of deep learning methods were highlighted for take-over studies, contributing to the design of intelligent human-machine interaction systems in conditional automation. Our findings present a valid method of deep learning in predicting drivers' take-over performance and meanwhile have implications for the development of intelligent adaptive take-over time budget regulation and dynamic drivers' state adjustment algorithms. IEEE

关键词： Automation

来源：评论

学校读者我要写书评

暂无评论

Detecting Double JPEG Compressed Color Images via an Improved Approach

引用

computers, Materials & Continua 2023年第4期75卷 1765-1781页

作者： Xiaojie Zhao Xiankui Meng Ruyong Ren Shaozhang Niu Zhenguang Gao School of Sciences Beijing University of Posts and TelecommunicationsBeijing100876China Beijing Key Lab of Intelligent Telecommunication Software and Multimedia Beijing University of Posts and TelecommunicationsBeijing100876China Department of Computer Science Framingham State UniversityFraminghamMA01772USA

Detecting double Joint Photographic Experts Group (JPEG) compressionfor color images is vital in the field of image forensics. In previousresearches, there have been various approaches to detecting double JPEGcompression with different quantization matrices. However, the detectionof double JPEG color images with the same quantization matrix is stilla challenging task. An effective detection approach to extract features isproposed in this paper by combining traditional analysis with ConvolutionalNeural Networks (CNN). On the one hand, the number of nonzero pixels andthe sum of pixel values of color space conversion error are provided with 12-dimensional features through experiments. On the other hand, the roundingerror, the truncation error and the quantization coefficient matrix are used togenerate a total of 128-dimensional features via a specially designed CNN. Insuch aCNN, convolutional layers with fixed kernel of 1×1 and Dropout layersare adopted to prevent overfitting of the model, and an average pooling layeris used to extract local characteristics. In this approach, the Support VectorMachine (SVM) classifier is applied to distinguishwhether a given color imageis primarily or secondarily compressed. The approach is also suitable for thecase when customized needs are considered. The experimental results showthat the proposed approach is more effective than some existing ones whenthe compression quality factors are low.

关键词： Color image forensics double JPEG compression detection the same quantization matrix CNN

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：