检索结果-内蒙古大学图书馆

Xinyu: An Efficient LLM-based System for Commentary Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wu, Yiquan Tang, Bo Xi, Chenyang Yu, Yu Wang, Pengyu Liu, Yifei Kuang, Kun Deng, Haiying Li, Zhiyu Xiong, Feiyu Hu, Jie Cheng, Peng Wang, Zhonghao Wang, Yi Luo, Yi Yang, Mingchuan Zhejiang University Hangzhou China University of Science and Technology of China Hefei China Institute for Advanced Algorithms Research Shanghai China Northeastern University Shenyang China State Key Laboratory of Media Convergence Production Technology and Systems Beijing China Research Institute of China Telecom Beijing China

Commentary provides readers with a deep understanding of events by presenting diverse arguments and evidence. However, creating commentary is a time-consuming task, even for skilled commentators. Large language models (LLMs) have simplified the process of natural language generation, but their direct application in commentary creation still faces challenges due to unique task requirements. These requirements can be categorized into two levels: 1) fundamental requirements, which include creating well-structured and logically consistent narratives, and 2) advanced requirements, which involve generating quality arguments and providing convincing evidence. In this paper, we introduce Xinyu, an efficient LLM-based system designed to assist commentators in generating Chinese commentaries. To meet the fundamental requirements, we deconstruct the generation process into sequential steps, proposing targeted strategies and supervised fine-tuning (SFT) for each step. To address the advanced requirements, we present an argument ranking model for arguments and establish a comprehensive evidence database that includes up-to-date events and classic books, thereby strengthening the substantiation of the evidence with retrieval augmented generation (RAG) technology. To evaluate the generated commentaries more fairly, corresponding to the two-level requirements, we introduce a comprehensive evaluation metric that considers five distinct perspectives in commentary generation. Our experiments confirm the effectiveness of our proposed system. We also observe a significant increase in the efficiency of commentators in real-world scenarios, with the average time spent on creating a commentary dropping from 4 hours to 20 minutes. Importantly, such an increase in efficiency does not compromise the quality of the commentaries. © 2024, CC BY.

关键词： Economic and social effects

GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Tang, Chen Lv, Bo Zheng, Zifan Yang, Bohao Zhao, Kun Liao, Ning Wang, Xiaoxing Xiong, Feiyu Li, Zhiyu Liu, Nayu Jiang, Jingchi Institute for Advanced Algorithms Research Shanghai China Institute of Computing Technology Chinese Academy of Sciences China The University of Manchester United Kingdom The University of Pittsburgh United States National Key Laboratory of Smart Farm Technologies and Systems Harbin Institute of Technology China University of Sydney Australia

Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo Graph MoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models. Copyright © 2025, The Authors. All rights reserved.

关键词： Network architecture

Internal Consistency and Self-Feedback in Large Language Models: A Survey

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Liang, Xun Song, Shichao Zheng, Zifan Wang, Hanyu Yu, Qingchen Li, Xunkai Li, Rong-Hua Wang, Yi Wang, Zhonghao Xiong, Feiyu Li, Zhiyu The School of Information Renmin University of China Beijing China The Large Language Model Center Institute for Advanced Algorithms Research Shanghai China The School of Computer Science and Technology Beijing Institute of Technology Beijing China The State Key Laboratory of Media Convergence Production Technology and Systems Xinhua News Agency Beijing China

Large language models (LLMs) often exhibit deficient reasoning or generate hallucinations. To address these, studies prefixed with "Self-" such as Self-Consistency, Self-Improve, and Self-Refine have been initiated. They share a commonality: involving LLMs evaluating and updating themselves. Nonetheless, these efforts lack a unified perspective on summarization, as existing surveys predominantly focus on categorization. In this paper, we use a unified perspective of internal consistency, offering explanations for reasoning deficiencies and hallucinations. Internal consistency refers to the consistency in expressions among LLMs' latent, decoding, or response layers based on sampling methodologies. Then, we introduce an effective theoretical framework capable of mining internal consistency, named Self-Feedback. This framework consists of two modules: Self-Evaluation and Self-Update. The former captures internal consistency signals, while the latter leverages the signals to enhance either the model's response or the model itself. This framework has been employed in numerous studies. We systematically classify these studies by tasks and lines of work;summarize relevant evaluation methods and benchmarks;and delve into the concern, "Does Self-Feedback Really Work?" We also propose several critical viewpoints, including the "Hourglass Evolution of Internal Consistency", "Consistency Is (Almost) Correctness" hypothesis, and "The Paradox of Latent and Explicit Reasoning". The relevant resources are open-sourced at https://***/IAAR-Shanghai/ICSFSurvey. Copyright © 2024, The Authors. All rights reserved.

关键词： Benchmarking

DPA-2:a large atomic model as a multitask learner

学校读者我要写书评

暂无评论

npj Computational Materials 2024年第1期10卷 185-199页

作者： Duo Zhang Xinzijian Liu Xiangyu Zhang Chengqian Zhang Chun Cai Hangrui Bi Yiming Du Xuejian Qin Anyang Peng Jiameng Huang Bowen Li Yifan Shan Jinzhe Zeng Yuzhi Zhang Siyuan Liu Yifan Li Junhan Chang Xinyan Wang Shuo Zhou Jianchuan Liu Xiaoshan Luo Zhenyu Wang Wanrun Jiang Jing Wu Yudi Yang Jiyuan Yang Manyi Yang Fu-Qiang Gong Linshuang Zhang Mengchao Shi Fu-Zhi Dai Darrin M.York Shi Liu Tong Zhu Zhicheng Zhong Jian Lv Jun Cheng Weile Jia Mohan Chen Guolin Ke Weinan E Linfeng Zhang Han Wang AI for Science Institute BeijingP.R.China DP Technology BeijingP.R.China Academy for Advanced Interdisciplinary Studies Peking UniversityBeijingP.R.China State Key Lab of Processors Institute of Computing TechnologyChinese Academy of SciencesBeijingP.R.China University of Chinese Academy of Sciences BeijingP.R.China HEDPS CAPTCollege of EngineeringPeking UniversityBeijingP.R.China Ningbo Institute of Materials Technology and Engineering Chinese Academy of SciencesNingboP.R.China CAS Key Laboratory of Magnetic Materials and Devices and Zhejiang Province Key Laboratory of Magnetic Materials and Application Technology Chinese Academy of SciencesNingboP.R.China School of Electronics Engineering and Computer Science Peking UniversityBeijingP.R.China Shanghai Engineering Research Center of Molecular Therapeutics&New Drug Development School of Chemistry and Molecular EngineeringEast China Normal UniversityShanghaiP.R.China Laboratory for Biomolecular Simulation Research Institute for Quantitative Biomedicine and Department of Chemistry and Chemical BiologyRutgers UniversityPiscatawayNJUSA Department of Chemistry Princeton UniversityPrincetonNJUSA College of Chemistry and Molecular Engineering Peking UniversityBeijingP.R.China Yuanpei College Peking UniversityBeijingP.R.China School of Electrical Engineering and Electronic Information Xihua UniversityChengduP.R.China State Key Laboratory of Superhard Materials College of PhysicsJilin UniversityChangchunP.R.China Key Laboratory of Material Simulation Methods&Software of Ministry of Education College of PhysicsJilin UniversityChangchunP.R.China International Center of Future Science Jilin UniversityChangchunP.R.China Key Laboratory for Quantum Materialsof Zhejiang Province Department of PhysicsSchool of ScienceWestlake UniversityHangzhouP.R.China Atomistic Simulations Italian Institute of TechnologyGenovaItaly State Key Laboratory of Physical Chemistry of Solid Surface iChEMCollege of Chemistry and Chemical EngineeringXiame

The rapid advancements in artificial intelligence(AI)are catalyzing transformative changes in atomic modeling,simulation,and ***-driven potential energy models havedemonstrated the capability to conduct large-scale,long-duration simulations with the accuracy of ab initio electronic structure ***,the model generation process remains a bottleneck for large-scale *** propose a shift towards a model-centric ecosystem,wherein a large atomic model(LAM),pretrained across multiple disciplines,can be efficiently fine-tuned and distilled for various downstream tasks,thereby establishing a new framework for molecular *** this study,we introduce the DPA-2 architecture as a prototype for ***-trained on a diverse array of chemical and materials systemsusing a multi-task approach,DPA-2demonstrates superior generalization capabilities across multiple downstream tasks compared to the traditional single-task pre-training and fine-tuning *** approach sets the stage for the development and broad application of LAMs in molecular and materials simulation research.

关键词： DPA establishing thereby

REASONING BIAS OF NEXT TOKEN PREDICTION TRAINING

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Lin, Pengxiao Zhang, Zhongwang Xu, Zhi-Qin John Institute of Natural Sciences MOE-LSC Shanghai Jiao Tong University China School of Mathematical Sciences Shanghai Jiao Tong University China School of Artificial Intelligence Shanghai Jiao Tong University China Key Laboratory of Marine Intelligent Equipment and System Ministry of Education China Center for LLM Institute for Advanced Algorithms Research Shanghai China

Since the inception of Large Language Models (LLMs), the quest to efficiently train them for superior reasoning capabilities has been a pivotal challenge. The dominant training paradigm for LLMs is based on next token prediction (NTP). Alternative methodologies, called Critical Token Prediction (CTP), focused exclusively on specific critical tokens (such as the answer in Q&A dataset), aiming to reduce the overfitting of extraneous information and noise. Contrary to initial assumptions, our research reveals that despite NTP’s exposure to noise during training, it surpasses CTP in reasoning ability. We attribute this counterintuitive outcome to the regularizing influence of noise on the training dynamics. Our empirical analysis shows that NTP-trained models exhibit enhanced generalization and robustness across various benchmark reasoning datasets, demonstrating greater resilience to perturbations and achieving flatter loss minima. These findings illuminate that NTP is instrumental in fostering reasoning abilities during pretraining, whereas CTP is more effective for finetuning, thereby enriching our comprehension of optimal training strategies in LLM development. © 2025, CC BY.

关键词： Prediction models

AN ANALYSIS FOR REASONING BIAS OF LANGUAGE MODELS WITH SMALL INITIALIZATION

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Yao, Junjie Zhang, Zhongwang Xu, Zhi-Qin John Institute of Natural Sciences MOE-LSC Shanghai Jiao Tong University China School of Mathematical Sciences Shanghai Jiao Tong University China School of Artificial Intelligence Shanghai Jiao Tong University China Key Laboratory of Marine Intelligent Equipment and System Ministry of Education China Center for LLM Institute for Advanced Algorithms Research Shanghai China

Transformer-based Large Language Models (LLMs) have revolutionized Natural Language Processing by demonstrating exceptional performance across diverse tasks. This study investigates the impact of the parameter initialization scale on the training behavior and task preferences of LLMs. We discover that smaller initialization scales encourage models to favor reasoning tasks, whereas larger initialization scales lead to a preference for memorization tasks. We validate this reasoning bias via real datasets and meticulously designed anchor functions. Further analysis of initial training dynamics suggests that specific model components, particularly the embedding space and self-attention mechanisms, play pivotal roles in shaping these learning biases. We provide a theoretical framework from the perspective of model training dynamics to explain these phenomena. Additionally, experiments on real-world language tasks corroborate our theoretical insights. This work enhances our understanding of how initialization strategies influence LLM performance on reasoning tasks and offers valuable guidelines for training models. © 2025, CC BY.

关键词： Natural language processing systems

Low-dose Dynamic Cerebral Perfusion CT Reconstruction Method Based on Voxel-level TAC Correction

学校读者我要写书评

暂无评论

Low-dose Dynamic Cerebral Perfusion CT Reconstruction Method...

第十七届中国体视学与图像分析学术会议

作者： Zixiang Chen Ying Huang Zhenxing Huang Guotao Quan Xiang Li Xin Liu Hairong Zheng Dong Liang Hu Zhanli Lauterbur Research Center for Biomedical Imaging Shenzhen Institute of Advanced TechnologyChinese Academy of Sciences Chinese Academy of Sciences Key Laboratory of Health Informatics Department of CT Physics & Algorithms United Imaging Healthcare Group

Background Dynamic cerebral perfusion computed tomography(DCP-CT) is a advanced imaging technique that help in clinical diagnosis of cerebrovascular ***,the radiation dose deposition during the repeated CT scans serious choke off its clinical *** this study,we proposed a DCP-CT images reconstruction method that is able to reconstruct images from low-dose scan *** Raw DCP-CT data were acquired by uCT-960+(United Imaging Healthcare Inc.),and data corresponding to the middle slice of the detector were used in our ***-CT images were reconstructed from raw data using a least square-based iterative method accompanied with voxel-level time-attenuation curve(TAC) correction(LS-VC).Low-dose CT imaging protocol was simulated by data down-sampling(representing the mode of sparse-view) and adding gaussian noise to the down-sampled sinograms(representing the mode of reduced kVp).During the reconstruction iterations,dynamic image noise was eliminated from the temporal domain via the application of the theory of third-order Hermite interpolation(THI).Quantitative evaluations of the reconstructed images were carried out by using the peak signal-to-noise ratio(PSNR) and structural similarity(SSIM) referred to normal-dose DCPCT *** maps were calculated from the reconstructed images for imaging *** conventional FBP and the famous PICCS methods were involved for ***:Superior performance of the proposed PSVC method is demonstrated by the remarkable visual effect of image denoising and the better PSNR and SSIM values compared to several state-of-the-art ***,LSVC yields the best hemodynamic maps among all the compared ***:The proposed LSVC method has very advanced performance in DCP-CT image reconstruction with raw data acquired with a low-dose scan *** may effectively promote the clinical application of DCP-CT.

关键词： Low-dose dynamic cerebral perfusion CT Iterative reconstruction Voxel-level TAC correction Thirdorder Hermite interpolation Hemodynamic maps

A Second-Order Time-Accurate, Linearly Fully Decoupled Unconditionally Energy Stable Discontinuous Galerkin Method for Tumor Growth Model

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Guo, Penghao Wang, Bo Zou, Guang-An School of Mathematics and Statistics Henan University Kaifeng475004 China Henan Key Laboratory of Earth System Observation and Modeling Henan University Kaifeng475004 China The Academy for Advanced Interdisplinary Studies Henan University Zhengzhou450046 China School of Mathematics and Statistics Henan Engineering Research Center for Artificial Intelligence Theory and Algorithms Henan University Kaifeng475004 China

In this paper, we firstly derive a tumor growth model based on free energy. For this model, the scalar auxiliary variable (SAV) is used to handle the nonlinear term. Combining the second-order backward Euler (BDF2) method and the discontinuous Galerkin (DG) method to discretize the model, we propose a second-order time-accurate, linear, fully decoupled and unconditionally energy stable numerical scheme. After giving the unique solvability, we rigorously prove the unconditional energy stability and optimal error estimates of the scheme. Finally, several numerical experiments are performed to verify the energy stability and validity of the proposed scheme. © 2024, The Authors. All rights reserved.

关键词： Free energy

Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Zhongwang Lin, Pengxiao Wang, Zhiwei Zhang, Yaoyu Xu, Zhi-Qin John Institute of Natural Sciences MOE-LSC Shanghai Jiao Tong University China School of Mathematical Sciences Shanghai Jiao Tong University China Key Laboratory of Marine Intelligent Equipment and System Ministry of Education China Shanghai Seres Information Technology Co. Ltd Shanghai China Center for LLM Institute for Advanced Algorithms Research Shanghai China

Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the mechanisms of how transformers behave on unseen compositional tasks. We discover that the parameter initialization scale plays a critical role in determining whether the model learns inferential (reasoning-based) solutions, which capture the underlying compositional primitives, or symmetric (memory-based) solutions, which simply memorize mappings without understanding the compositional structure. By analyzing the information flow and vector representations within the model, we reveal the distinct mechanisms underlying these solution types. We further find that inferential (reasoning-based) solutions exhibit low complexity bias, which we hypothesize is a key factor enabling them to learn individual mappings for single anchors. We validate our conclusions on various real-world datasets. Our findings provide valuable insights into the role of initialization scale in tuning the reasoning and memorizing ability and we propose the initialization rate γ to be a convenient tunable hyper-parameter in common deep learning frameworks, where 1/dγin is the standard deviation of parameters of the layer with din Copyright © 2024, The Authors. All rights reserved.

关键词： Mapping