检索结果-内蒙古大学图书馆

T4SEfinder: a bioinformatics tool for genome-scale prediction of bacterial type IV secreted effectors using pre-trained protein language model

引用

BRIEFINGS IN BIOINFORMATICS 2022年第1期23卷 bbab420页

作者： Zhang, Yumeng Zhang, Yangming Xiong, Yi Wang, Hui Deng, Zixin Song, Jiangning Ou, Hong-Yu Shanghai Jiao Tong Univ Sch Life Sci & Biotechnol Shanghai 200030 Peoples R China Monash Univ Biomed Discovery Inst Melbourne Vic 3800 Australia Monash Univ Dept Biochem & Mol Biol Melbourne Vic 3800 Australia Beijing Inst Microbiol & Epidemiol State Key Lab Pathogens & Biosecur Beijing Peoples R China

Bacterial type IV secretion systems (T4SSs) are versatile and membrane-spanning apparatuses, which mediate both genetic exchange and delivery of effector proteins to target eukaryotic cells. The secreted effectors (T4SEs) can affect gene expression and signal transduction of the host cells. As such, they often function as virulence factors and play an important role in bacterial pathogenesis. Nowadays, T4SE prediction tools have utilized various machine learning algorithms, but the accuracy and speed of these tools remain to be improved. In this study, we apply a sequence embedding strategy from a pre-trained language model of protein sequences (TAPE) to the classification task of T4SEs. The training dataset is mainly derived from our updated type IV secretion system database SecReT4 with newly experimentally verified T4SEs. An online web server termed T4SEfinder is developed using TAPE and a multi-layer perceptron (MLP) for T4SE prediction after a comprehensive performance comparison with several candidate models, which achieves a slightly higher level of accuracy than the existing prediction tools. It only takes about 3 minutes to make a classification for 5000 protein sequences by T4SEfinder so that the computational speed is qualified for whole genome-scale T4SEs detection in pathogenic bacteria. T4SEfinder might contribute to meet the increasing demands of re-annotating secretion systems and effector proteins in sequenced bacterial genomes. T4SEfinder is freely accessible at https://***/T4SEfinder_TAPE/.

关键词： Type IV secreted effectors pre-trained language model sequence analysis deep learning

来源：评论

学校读者我要写书评

暂无评论

LightToken: A Task and model-agnostic Lightweight Token Embedding Framework for pre-trained language models 23

LightToken: A Task and Model-agnostic Lightweight Token Embe...

引用

29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

作者： Wang, Haoyu Li, Ruirui Jiang, Haoming Wang, Zhengyang Tang, Xianfeng Bi, Bin Cheng, Monica Yin, Bing Wang, Yaqing Zhao, Tuo Gao, Jing Purdue Univ W Lafayette IN 47907 USA Amazon com Inc Seattle WA USA Georgia Inst Technol Atlanta GA 30332 USA

ISBN: (纸本)9798400701030

pre-trained language models (PLMs) such as BERT, RoBERTa, and DeBERTa have achieved state-of-the-art performance on various downstream tasks. The enormous sizes of PLMs hinder their deployment in resource-constrained scenarios, e.g., on edge and mobile devices. To address this issue, many model compression approaches have been proposed to reduce the number of model parameters. This paper focuses on compressing the token embedding matrices of PLMs, which typically make up a large proportion (around 20-30%) of the entire model parameters. Existing efforts to compress token embedding usually require the introduction of customized compression architectures or the optimization of model compression processes for individual downstream tasks, limiting their applicability in both model and task dimensions. To overcome these limitations and adhere to the principle of "one-for-all", we propose a lightweight token embedding framework named LightToken, which is able to produce compressed token embedding in a task and model-agnostic fashion. LightToken is generally compatible with different architectures and applicable to any downstream task. Specifically, through an integration of low-rank approximation, novel residual binary autoencoder, and a new compression loss function, LightToken can significantly improve the model compression ratio. To demonstrate the effectiveness of LightToken, we conduct comprehensive experiments on natural language understanding and question answering tasks. In particular, LightToken improves the state-of-the-art token embedding compression ratio from 5 to 25 and outperforms the existing token embedding compression approaches by 11% and 5% on GLUE and SQuAD v1.1 benchmarks, respectively.

关键词： pre-trained language model Compression

来源：评论

学校读者我要写书评

暂无评论

A diachronic language model for long-time span classical Chinese

引用

INFORMATION PROCESSING & MANAGEMENT 2025年第1期62卷

作者： Wei, Yuting Li, Meiling Zhu, Yangfu Xu, Yuanxing Li, Yuqing Wu, Bin Beijing Univ Posts & Telecommun Beijing Key Lab Intelligent Telecommun Software & Beijing 100876 Peoples R China

Classical Chinese literature, with its long history spanning thousands of years, serves as an invaluable resource for historical and humanistic studies. previous classical Chinese language models have achieved significant progress in semantic understanding. However, they largely neglected the dynamic evolution of language across different historical eras. In this paper, we introduce a novel diachronic pre-trained language model tailored for classical Chinese texts. This model utilizes a time-based transformer architecture that captures the continuous evolution of semantics over time. Moreover, it adeptly balances the contextual and temporal information, minimizing semantic ambiguities from excessive time-related inputs. A high-quality diachronic corpus for classical Chinese is developed for training. This corpus spans from the pre-Qin dynasty to the Qing dynasty and includes a diverse array of genres. We validate its effectiveness by enriching a well-known classical Chinese word sense disambiguation dataset with additional temporal annotations. The results demonstrate the state-of-the-art performance of our model in discerning classical Chinese word meanings across different historical periods. Our research helps linguists to rapidly grasp the extent of semantic changes across different periods from vast corpora.1 1

关键词： Classical Chinese understanding Semantic shift Diachronic word representation pre-trained language model Word sense disambiguation

来源：评论

学校读者我要写书评

暂无评论

Historical facts learning from Long-Short Terms with language model for Temporal Knowledge Graph Reasoning

引用

INFORMATION PROCESSING & MANAGEMENT 2025年第3期62卷

作者： Xu, Wenjie Liu, Ben Peng, Miao Jiang, Zihao Jia, Xu Liu, Kai Liu, Lei Peng, Min Wuhan Univ Sch Comp Sci Wuhan 430072 Hubei Peoples R China Hong Kong Univ Sci & Technol Guangzhou Data Sci & Analyt Thrust Guangzhou 511458 Guangdong Peoples R China Hebei Normal Univ Coll Comp & Cyber Secur Shijiazhuang 050024 Hebei Peoples R China

Temporal Knowledge Graph Reasoning (TKGR) aims to reason the missing parts in TKGs based on historical facts from different time periods. Traditional GCN-based TKGR models depend on structured relations between entities. To utilize the rich linguistic information in TKGs, some models have focused on applying pre-trained language models (PLMs) to TKGR. However, previous PLM-based models still face some issues: (1) they did not mine the associations in relations;(2) they did not differentiate the impact of historical facts from different time periods. (3) they introduced external knowledge to enhance the performance without fully utilizing the inherent reasoning capabilities of PLMs. To deal with these issues, we propose HFL: H istorical F acts L earning from Long-Short Terms with language model for TKGR. Firstly, we construct time tokens for different types of time intervals to use timestamps and input the historical facts relevant to the query into the PLMs to learn the associations in relations. Secondly, we take a multi-perspective sampling strategy to learn from different time periods and use the original text information in TKGs or even no text information to learn reasoning abilities without any external knowledge. Finally, we perform HFL on four TKGR benchmarks, and the experiment results demonstrate that HFL has great competitiveness compared to both graph-based and PLMbased models. Additionally, we design a variant that applies HFL to LLMs and evaluate the performance of different LLMs.

关键词： Knowledge graph Temporal knowledge graph reasoning pre-trained language model Large-scale language model

来源：评论

学校读者我要写书评

暂无评论

Enhancing pre-trained language models with Chinese character morphological knowledge

引用

INFORMATION PROCESSING & MANAGEMENT 2025年第1期62卷

作者： Zheng, Zhenzhong Wu, Xiaoming Liu, Xiangzhi Qilu Univ Technol Shandong Acad Sci Shandong Comp Sci Ctr Key Lab Comp Power Network & Informat SecurMinist Jinan 250014 Peoples R China Shandong Fundamental Res Ctr Comp Sci Shandong Prov Key Lab Comp Networks Jinan 250014 Peoples R China

pre-trained language models (PLMs) have demonstrated success in Chinese natural language processing (NLP) tasks by acquiring high-quality representations through contextual learning. However, these models tend to neglect the glyph features of Chinese characters, which contain valuable semantic knowledge. To address this issue, this paper introduces a self-supervised learning strategy, named SGBERT, aiming to learn high-quality semantic knowledge from Chinese Character morphology to enhance PLMs' understanding of natural language. Specifically, the learning process of SGBERT can be divided into two stages. In the first stage, we preheat the glyph encoder by constructing contrastive learning between glyphs, enabling it to obtain preliminary glyph coding capabilities. In the second stage, we transform the glyph features captured by the glyph encoder into context-sensitive representations through a glyph-aware window. These representations are then contrasted with the character representations generated by the PLMs, leveraging the powerful representation capabilities of the PLMs to guide glyph learning. Finally, the glyph knowledge is fused with the pre-trained model representations to obtain semantically richer representations. We conduct experiments on ten datasets covering six Chinese NLP tasks, and the results demonstrate that SGBERT significantly enhances commonly used Chinese PLMs. On average, the introduction of SGBERT resulted in a performance improvement of 1.36% for BERT and 1.09% for RoBERTa.

关键词： Nature language processing Self-supervised learning Contrastive learning pre-trained language model BERT Glyph

来源：评论

学校读者我要写书评

暂无评论

K-Bloom: unleashing the power of pre-trained language models in extracting knowledge graph with predefined relations

引用

KNOWLEDGE AND INFORMATION SYSTEMS 2025年第5期67卷 4487-4521页

作者： Vo, Trung Luu, Son T. Nguyen, Le-Minh Japan Adv Inst Sci & Technol Nomi Ishikawa Japan VNU Univ Informat Technol Ho Chi Minh City Vietnam

pre-trained language models have become popular in natural language processing tasks, but their inner workings and knowledge acquisition processes remain unclear. To address this issue, we introduce K-Bloom-a refined search-and-score mechanism tailored for seed-guided exploration in pre-trained language models, ensuring both accuracy and efficiency in extracting relevant entity pairs and relationships. Specifically, our crawling procedure is divided into two sub-tasks. Using a few seed entity pairs to minimize the need for extensive manual effort or predefined knowledge, we expand the knowledge graph with new entity pairs around these seeds. To evaluate the effectiveness of our proposed model, we conducted experiments on two datasets that cover the general domain. Our resulting knowledge graphs serve as symbolic representations of the source pre-trained language models, providing valuable insights into their knowledge capacities. Additionally, they enhance our understanding of the pre-trained language models' capabilities when automatically evaluated on large language models. The experimental results demonstrate that our method outperforms the baseline approach by up to 5.62% in terms of accuracy in various settings of the two benchmarks. We believe that our approach offers a scalable and flexible solution for knowledge graph construction and can be applied to different domains and novel contexts.

关键词： pre-trained language model Knowledge graph extraction Knowledge discovery Knowledge representation

来源：评论

学校读者我要写书评

暂无评论

Memory-Tuning: A Unified Parameter-Efficient Tuning Method for pre-trained language models

引用

IEEE TRANSACTIONS ON AUDIO SPEECH AND language PROCESSING 2025年 33卷 1-10页

作者： Qi, Wang Liu, Rui Zuo, Yuan Li, Fengzhi Chen, Yong Wu, Junjie Beihang Univ Sch Comp Sci & Engn Beijing 100191 Peoples R China Beihang Univ Sch Econ & Management Beijing 100191 Peoples R China MIIT Key Lab Data Intelligence & Management Beijing 100191 Peoples R China Beijing Univ Posts & Telecommun Sch Comp Sci Beijing 100876 Peoples R China

Conventional fine-tuning encounters increasing difficulties given the size of current pre-trained language models, which makes parameter-efficient tuning become the focal point of frontier research. Recent advances in this field is the unified tuning methods that aim to tune the representations of both multi-head attention (MHA) and fully connected feed-forward network (FFN) simultaneously, but they rely on existing tuning methods and do not explicitly model domain knowledge for downstream tasks. In this work, we propose memory-tuning, a novel unified parameter-efficient tuning method with task-specific knowledge learning, for both MHA and FFN components in Transformer blocks. We also prove that the well-known prefix tuning is also a kind of memory tuning, which further ensures memory tuning is a genuine unified tuning method. Experiments on eight benchmark data sets including both sentence- and token-level tasks demonstrate that our method outperforms the state-of-the-art baselines even full-tuning in most cases.

关键词： Tuning Transformers Vectors Adaptation models Transfer learning Speech processing Natural language processing Computer science Computational modeling Benchmark testing pre-trained language model parameter-efficient tuning memory mechanism

来源：评论

学校读者我要写书评

暂无评论

ASRLM: ASR-Robust language model pre-training via Generative and Discriminative Learning 13th

ASRLM: ASR-Robust Language Model Pre-training via Generative...

引用

13th International Conference on Natural language Processing and Chinese Computing

作者： Hu, Qian Han, Xue Wang, Yiting Wang, Yitong Deng, Chao Feng, Junlan China Mobile Res Inst JiuTian Team Beijing Peoples R China

ISBN: (纸本)9789819794362;9789819794379

The rise of voice interface applications has renewed interest in improving the robustness of spoken language understanding(SLU). Many advances have come from end-to-end speech-language joint training, such as inferring semantics directly from speech signals and post-editing automatic speech recognition (ASR) output. Despite their performance achievements, these methods either suffer from the unavailability of a large number of paired error-prone ASR transcriptions and ground-truth annotations or are computationally costly. To mitigate these issues, we propose an ASR-robust pre-trained language model (ASRLM), which involves a generator generating simulated ASR transcriptions from ground-truth annotations and a sample-efficient discriminator distinguishing reasonable ASR errors from unrealistic ones. Experimental results demonstrate that ASRLM improves performance on a wide range of SLU tasks in the presence of ASR errors while saving 27% of the computation cost compared to baselines. Analysis also shows that our proposed generator is better than other simulation methods, including both BERT and GPT4-based, at simulating real-world ASR error situations.

关键词： Spoken language understanding Automatic speech recognition pre-trained language model

来源：评论

学校读者我要写书评

暂无评论

FedBM: Stealing knowledge from pre-trained language models for heterogeneous federated learning

引用

MEDICAL IMAGE ANALYSIS 2025年 102卷 103524页

作者： Zhu, Meilu Yang, Qiushi Gao, Zhifan Yuan, Yixuan Liu, Jun City Univ Hong Kong Dept Mech Engn Hong Kong Peoples R China City Univ Hong Kong Dept Elect Engn Hong Kong Peoples R China Sun Yat Sen Univ Sch Biomed Engn Guangzhou Peoples R China Univ Hong Kong Dept Data & Syst Engn Hong Kong Peoples R China

Federated learning (FL) has shown great potential in medical image computing since it provides a decentralized learning paradigm that allows multiple clients to train a model collaboratively without privacy leakage. However, current studies have shown that data heterogeneity incurs local learning bias in classifiers and feature extractors of client models during local training, leading to the performance degradation of a federation system. To address these issues, we propose a novel framework called Federated Bias eliMinating (FedBM) to get rid of local learning bias in heterogeneous federated learning (FL), which mainly consists of two modules, i.e., Linguistic Knowledge-based Classifier Construction (LKCC) and Concept-guided Global Distribution Estimation (CGDE). Specifically, LKCC exploits class concepts, prompts and pre-trained language models (PLMs) to obtain concept embeddings. These embeddings are used to estimate the latent concept distribution of each class in the linguistic space. Based on the theoretical derivation, we can rely on these distributions to pre-construct a high-quality classifier for clients to achieve classification optimization, which is frozen to avoid classifier bias during local training. CGDE samples probabilistic concept embeddings from the latent concept distributions to learn a conditional generator to capture the input space of the global model. Three regularization terms are introduced to improve the quality and utility of the generator. The generator is shared by all clients and produces pseudo data to calibrate updates of local feature extractors. Extensive comparison experiments and ablation studies on public datasets demonstrate the superior performance of FedBM over state-of-the-arts and confirm the effectiveness of each module, respectively. The code is available at https://***/CUHKAIM-Group/FedBM.

关键词： Federated learning Medical Image Classification pre-trained language model

来源：评论

学校读者我要写书评

暂无评论

Comprehensive Study on Zero-Shot Text Classification Using Category Mapping

引用

IEEE ACCESS 2025年 13卷 23526-23546页

作者： Zhang, Kai Zhang, Qiuxia Wang, Chung-Che Jang, Jyh-Shing Roger Yiwu Ind & Commercial Coll Yiwu 322000 Zhejiang Peoples R China Natl Taiwan Univ Grad Inst Network & Multimedia Taipei 106 Taiwan Natl Taiwan Univ Dept Comp Sci & Informat Engn Taipei 106 Taiwan

Existing zero-shot text classification methods based on large pre-trained models with added prompts exhibit strong representational capacity and scalability but have relatively poor commercial applicability. Approaches that fine-tune smaller models using label mappings and existing datasets for zero-shot classification are simpler but suffer from weaker generalization capabilities. This paper employs three strategies to improve the accuracy and generalization of pre-trained models in zero-shot text classification tasks: 1) Utilizing a pre-trained model that transforms inputs into a standardized multiple-choice format. 2) Constructing a text classification training set using Wikipedia text data to fine-tune the pre-trained model;3) Proposing a zero-shot category mapping method based on GloVe text similarity, using Wikipedia categories as substitutes for text labels. Without fine-tuning on the target labels, this method achieves performance comparable to the best models fine-tuned with target labels.

关键词： Vectors Context modeling Text categorization Training Encoding Bidirectional control Computational modeling Data models predictive models Decoding Natural language processing pre-trained language model zero-shot text classification classification GloVe

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：