检索结果-内蒙古大学图书馆

pre-trained language model for code-mixed text in Indonesian, Javanese, and English using transformer

SOCIAL NETWORK ANALYSIS AND MINING 2025年第1期15卷 1-17页

作者： Hidayatullah, Ahmad Fathan Apong, Rosyzie Anna Lai, Daphne Teck Ching Qazi, Atika Univ Brunei Darussalam Sch Digital Sci Jalan Tungku Link BE-1410 Gadong Bandar Seri Beg Brunei Univ Islam Indonesia Dept Informat Jalan Kaliurang Km 14-5 Sleman 55584 Yogyakarta Indonesia Univ Brunei Darussalam Ctr Lifelong Learning Jalan Tungku Link BE-1410 Gadong Bandar Seri Beg Brunei

pre-trained language models (PLMs) have become increasingly popular due to their ability to achieve state-of-the-art performance on various natural language processing tasks with less training data and time. However, they struggle when dealing with code-mixed data, characterized by colloquial language and inconsistent linguistic forms. This limitation arises because most available PLMs are trained monolingually or for individual languages. Furthermore, the availability of PLMs specifically designed for code-mixed text remains limited, especially for low-resource languages like Indonesian (ID) and Javanese (JV). Despite the significant number of speakers of these languages, both languages are underrepresented in the field of natural language processing. To address these issues, this study introduces IndoJavE, a series of pre-trained language models specifically designed for mixing languages between Indonesian (ID), Javanese (JV), and English (EN) texts. We developed four transformer-based models, IndoJavE-BERT, IndoJavE-RoBERTa, IndoJavE-IndoBERTweet, and IndoJavE-IndoBERT, using two approaches: training from scratch and transfer learning. Our results show that transfer learning is more efficient and effective than training from scratch. The IndoJavE models outperformed multilingual and monolingual models in various downstream NLP tasks, highlighting the importance of specialized pre-trained models for handling code-mixed texts. This study paves the way for future research directions, including exploring the models' performance in diverse NLP tasks and developing more versatile pre-trained language models that can adapt to a broader range of code-mixed languages and dialects.

关键词： pre-trained language model Code-mixing Transformer Fine-tuning

来源：评论

学校读者我要写书评

暂无评论

PPTopicPLM: plug-and-play topic-enhanced pre-trained language model for short-text rumor detection

引用

JOURNAL OF SUPERCOMPUTING 2025年第1期81卷 1-20页

作者： Zeng, Jiangfeng Li, Xinyu Ma, Xiao Cent China Normal Univ Sch Informat Management Luoyu St Wuhan 430079 Hubei Peoples R China Zhongnan Univ Econ & Law Sch Informat Engn Nanhu Ave Wuhan 430073 Hubei Peoples R China

Recently, lots of pre-trained language models (PLMs) have been investigated for rumor detection and obtained superior results. However, existing PLM-based approaches are challenged and limited when addressing short texts which are often sparse, noisy, and semantically ambiguous. To remedy this, we propose a Plug-and-Play Topic-enhanced pre-trained language model for short-text rumor detection, dubbed PPTopicPLM, aiming to combine topic models with PLMs without modifying the core network structure of PLMs. First, the pre-trained RoBERTa is used to represent the short texts. Then, we utilize the semantics-assisted non-negative factorization (SeaNMF) model to explore topics in short texts since SeaNMF has proven to be effective in incorporating the word-context semantic correlations. Instead of simple but computationally expensive methods like word frequency statistics, we build the semantic correlation matrix based on the semantic similarity between the dynamic word embeddings yielded by pre-trained PLMs to mitigate the semantic information loss especially encountered by short texts. Afterward, we employ the multimodal factorized bilinear pooling technique to thoroughly aggregate topics and text semantics. Finally, substantial experiments conducted on four real-world datasets show that the proposed method achieves significant performance improvements due to its successfully joining topic models and PLMs for short-text rumor detection. We further empirically demonstrate the wide application prospect of our approach on the semantic similarity detection task. The codes are open sourced at https://***/zjfgh2015/PPTopicPLM.

关键词： Plug-and-play Rumor detection pre-trained language model Topic model

来源：评论

学校读者我要写书评

暂无评论

Diachronic semantic encoding based on pre-trained language model for temporal knowledge graph reasoning

引用

KNOWLEDGE-BASED SYSTEMS 2025年 318卷

作者： Deng, Yunteng Song, Jia Yang, Zhongliang Long, Yilin Zeng, Li Zhou, Linna Beijing Univ Posts & Telecommun Sch Cyberspace Secur Beijing 100876 Peoples R China Shenzhen Stock Exchange Shenzhen 518010 Guangdong Peoples R China

Temporal Knowledge Graph Reasoning (TKGR) aims to infer missing facts at specific timestamps. However, most existing methods primarily focus on the local and global evolutionary characteristics of temporal knowledge graphs (TKG), often neglecting the inherent semantic information of historical facts. The oversight limits the understanding of the diachronic evolution of facts, thereby limiting the ability to predict future missing facts. To address these issues, we propose a TKGR model with Diachronic Semantic Encoding based on a pre-trained language model, called DSEP. It uses a pre-trained language model (PLM) to learn the evolutionary characteristics of historical related facts of the entity or relation to be predicted, so as to enhance the understanding of historical facts by the graph encoder used to capture the local evolutionary characteristics of the temporal knowledge graph. Additionally, to further narrow the prediction scope, DSEP incorporates historical fact correlation matrix in its prediction results. Experimental results on four benchmark datasets demonstrate that DSEP significantly improves the performance of relation prediction in temporal knowledge graphs, with an average improvement of 20.9% in MRR1.

关键词： Temporal knowledge graph Graph neural network pre-trained language model

来源：评论

学校读者我要写书评

暂无评论

Talent Supply and Demand Matching Based on Prompt Learning and the pre-trained language model

引用

APPLIED SCIENCES-BASEL 2025年第5期15卷 2536-2536页

作者： Li, Kunping Liu, Jianhua Zhuang, Cunbo Beijing Inst Technol Sch Mech Engn Lab Digital Mfg Beijing 100081 Peoples R China Beijing Inst Technol Tangshan Res Inst Hebei Key Lab Intelligent Assembly & Detect Techno Tangshan 063000 Peoples R China

In the context of the accelerating new technological revolution and industrial transformation, the issue of talent supply and demand matching has become increasingly urgent. precise matching talent supply and demand is a critical factor in expediting the implementation of technological innovations. However, traditional methods relying on interpersonal networks for talent ability collection, demand transmission, and matching suffer from inefficiency and are often influenced by the subjective intentions of intermediaries, posing significant limitations. To address this challenge, we propose a novel approach named TSDM for talent supply and demand matching. TSDM leverages prompt learning with pre-trained large language models to extract detailed expressions of talent ability and demand from unstructured documents while utilizing the powerful text comprehension capabilities of pre-trained models for feature embedding. Furthermore, TSDM employs talent-specific and demand-specific encoding networks to perform deep learning on talent and demand features, capturing their comprehensive representations. In a series of comparative experiments, we validated the effectiveness of the proposed model. The results demonstrate that TSDM significantly enhances the accuracy of talent supply and demand matching, offering a promising approach to optimize human resource allocation.

关键词： talent supply and demand matching prompt learning pre-trained language model deep learning

来源：评论

学校读者我要写书评

暂无评论

Explainable reasoning over temporal knowledge graphs by pre-trained language model

引用

INFORMATION PROCESSING & MANAGEMENT 2025年第1期62卷

作者： Li, Qing Wu, Guanzhong Northwestern Polytech Univ Sch Comp Sci Xian 710000 Shaanxi Peoples R China

Temporal knowledge graph reasoning (TKGR) has been considered as a crucial task for modeling the evolving knowledge, aiming to infer the unknown connections between entities at specific times. Traditional TKGR methods try to aggregate structural information between entities and evolve representations of entities over distinct snapshots, while some other methods attempt to extract temporal logic rules from historical interactions. However, these methods fail to address the continuously emerging unseen entities over time and ignore the historical dependencies between entities and relations. To overcome these limitations, we propose a novel method, termed TPNet, which introduces historical information completion strategy (HICS) and pre-trained language model (PLM) to conduct explainable inductive reasoning over TKGs. Specifically, TPNet extracts reliable temporal logical paths from historical subgraphs using a temporal-correlated search strategy. For unseen entities, we utilize HICS to sample or generate paths to supplement their historical information. Besides, a PLM and a time-aware encoder are introduced to jointly encode the temporal paths, thereby comprehensively capturing dependencies between entities and relations. Moreover, the semantic similarity between the query quadruples and the extracted paths is evaluated to simultaneously optimize the representations of entities and relations. Extensive experiments on entity and relation prediction tasks are conducted to evaluate the performance of TPNet. The experimental results on four benchmark datasets demonstrate the superiority of TPNet over state-of-the-art TKGR methods, achieving improvements of 14.35%, 23.08%, 6.75% and 5.38% on MRR, respectively.

关键词： Knowledge graph reasoning pre-trained language model Logical reasoning Temporal knowledge graph Multi-hop paths Graph representation learning

来源：评论

学校读者我要写书评

暂无评论

Efficient word segmentation for enhancing Chinese spelling check in pre-trained language model

引用

KNOWLEDGE AND INFORMATION SYSTEMS 2025年第1期67卷 603-632页

作者： Li, Fangfang Jiang, Jie Tang, Dafu Shan, Youran Duan, Junwen Zhang, Shichao Cent South Univ Comp Sci & Engn Changsha 410083 Peoples R China Natl Univ Def Technol Coll Syst Engn Changsha 410073 Peoples R China Zhejiang Coll Secur Technol Coll Artificial Intelligence Wenzhou 325016 Peoples R China Guangxi Normal Univ Key Lab Educ Blockchain & Intelligent Technol Minist Educ Guilin 541004 Peoples R China Guangxi Normal Univ Guangxi Key Lab Multisource Informat Min & Secur Guilin 541004 Peoples R China

In existing pre-trained language models, Chinese spelling check (CSC) often considers phonetic and graphic details at the character level, and ignores the essential role of word segmentation. To address this issue, an efficient word segmentation is studied for CSC in this paper, referred to Word Segmentation-Enhanced Speller (WOSES). The WOSES comprises two distinct models, the Word Speller (WSpeller) and the Hierarchical Word Speller (H-WSpeller), designed both for mitigating the often-ignored word boundary errors in CSC. The WOSES framework outperforms existing benchmarks on standard datasets SIGHAN13 and SIGHAN15, attributed to its innovative use of word segmentation and the specialized pre-trained model, W-MLM. Notably, the WSpeller model within the WOSES framework achieves F1 score improvements of 3.3 and 2.1% on SIGHAN13 and SIGHAN15, respectively, compared to existing methods. In this paper, the importance of word segmentation is not only underscored in CSC, but also a novel performance standard is proposed in the domain.

关键词： Chinese spelling check pre-trained language model Word segmentation

来源：评论

学校读者我要写书评

暂无评论

Few-shot medical relation extraction via prompt tuning enhanced pre-trained language model

引用

NEUROCOMPUTING 2025年 633卷

作者： He, Guoxiu Huang, Chen East China Normal Univ Sch Econ & Management Shanghai 200062 Peoples R China Singapore Univ Technol & Design Singapore 487372 Singapore

Medical relation extraction is crucial for developing structured information to support intelligent healthcare systems. However, acquiring large volumes of labeled medical data is challenging due to the specialized nature of medical knowledge and privacy constraints. To address this, we propose a prompt-enhanced few- shot relation extraction (FSRE) model that leverages few-shot and prompt learning techniques to improve performance with minimal data. Our approach introduces a hard prompt concatenated to the original input, enabling contextually enriched learning. We calculate prototype representations by averaging the intermediate states of each relation class in the support set, and classify relations by finding the shortest distance between the query instance and class prototypes. We evaluate our model against existing deep learning based FSRE models using three biomedical datasets: the 2010 i2b2/VA challenge dataset, the CHEMPROT corpus, and the BioRED dataset, focusing on few-shot scenarios with limited training data. Our model demonstrates exceptional performance, achieving the highest accuracy across all datasets inmost training configurations under a 3-way- 5-shot condition and significantly surpassing the current state-of-the-art. Particularly, it achieves improvements ranging from 1.25% to 11.25% on the 2010 i2b2/VA challenge dataset, 3.4% to 20.2% on the CHEMPROT dataset, and 2.73% to 10.98% on the BioRED dataset compared to existing models. These substantial gains highlight the model's robust generalization ability, enabling it to effectively handle previously unseen relations during testing. The demonstrated effectiveness of this approach underscores its potential for diverse medical applications, particularly in scenarios where acquiring extensive labeled data is challenging.

关键词： Information extraction Medical relation extraction Few-shot learning Prompt tuning pre-trained language model

来源：评论

学校读者我要写书评

暂无评论

Graph-aware pre-trained language model for political sentiment analysis in Filipino social media

引用

ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE 2025年 146卷

作者： Aquino, Jean Aristide Liew, Di Jie Chang, Yung-Chun Taipei Med Univ Grad Inst Data Sci Taipei Taiwan Taipei Med Univ Hosp Clin Big Data Res Ctr Taipei Taiwan

Elections are emotionally and sentimentally charged events that offer unique opportunities for analysis of sentiments not typically observed during non-election periods. Unlike recurring phenomena, elections are inherently singular events, with each election shaped by distinct political, social, and cultural contexts. In the digital age, social media has become a direct channel for politicians and political parties to engage with voters, making it a critical platform for sentiment analysis. However, challenges such as imbalanced datasets, the prevalence of noisy non-text elements (e.g., emojis, hashtags, user mentions), and the need for effective integration of graphbased learning remain significant hurdles in sentiment prediction. To address these challenges, we constructed an imbalanced dataset of 8035 manually annotated tweets and approximately 516,000 weakly labeled Filipino tweets related to the 2022 Philippine National Election. Leveraging these datasets, we designed a Bidirectional Encoder Representations from Transformers (BERT) and Graph Convolution Network (GCN) model, which uniquely incorporates emojis, hashtags, and user mentions as features to enhance semantic understanding. Differing from the prior literature that focused solely on textual data or discarded non-textual elements, our model integrates these features to achieve a robust performance that outperforms baseline models with a macrorecall score of 64.73% and a macro F1-score of 68.72% on the imbalanced dataset. Additionally, we introduce a topic modeling framework that combines BERT embeddings with Latent Dirichlet Allocation (LDA) and LogLikelihood Ratio (LLR) to yield more distinct topic clusters for deeper sentiment analysis. Our work therefore contributes two novel datasets in Filipino as well as methodologies that bridge sentiment prediction and analysis, and in so doing, provides valuable resources for future research.

关键词： Politics Sentiment analysis Social media analytics pre-trained language model Graph convolution network Topic modeling

来源：评论

学校读者我要写书评

暂无评论

MCFC: A Momentum-Driven Clicked Feature Compressed pre-trained language model for Information Retrieval 13th

MCFC: A Momentum-Driven Clicked Feature Compressed Pre-train...

引用

13th International Conference on Natural language Processing and Chinese Computing

作者： Li, Dongyang Ding, Ruixue Xie, Pengjun He, Xiaofeng East China Normal Univ Shanghai Peoples R China Alibaba Grp Hangzhou Peoples R China

ISBN: (纸本)9789819794300;9789819794317

Information Retrieval (IR) pre-trained language models are trained from large-scale retrieval-based corpora to promote the task-specific knowledge capacity. previous works focus on general retrieval pre-trained datasets, which cover inter-document data and intra-document data, paying less attention to the important asset of clicked data which is commonly adopted in recommendation domain. However, the utilization of easily accessible clicked data is a non-trivial operation due to its characteristics of large volume and insufficient refinement, which affect model learning efficiency and imply the risk of distorting learning directions. In this paper, we propose a Momentum-Driven Clicked Feature Compressed pre-trained language models for Information Retrieval (MCFC). Specifically, to tackle the effective learning pace on large amounts of data, we generalize multiple similar feature instances and compress the dispersed knowledge together at the query granularity, named Multi-Instance Information Integration. Meanwhile, more relevant detection between queries and documents is eager in coarse clicked data background, we leverage a momentum-driven adjusting mechanism to refine the text representations, named Continuous Debiasing Calibration. Extensive experiments on downstream datasets validate the superiority of our work to other recent strong baselines.

关键词： pre-trained language model Information Retrieval Clicked Feature

来源：评论

学校读者我要写书评

暂无评论

Prompting disentangled embeddings for knowledge graph completion with pre-trained language model

引用

EXPERT SYSTEMS WITH APPLICATIONS 2025年 268卷

作者： Geng, Yuxia Chen, Jiaoyan Zeng, Yuhang Chen, Zhuo Zhang, Wen Pan, Jeff Z. Wang, Yuxiang Xu, Xiaoliang Powerchina Huadong Engn Corp Ltd Hangzhou 311112 Peoples R China Hangzhou Dianzi Univ Sch Comp Sci Hangzhou 310018 Peoples R China UNIV MANCHESTER Dept Comp Sci MANCHESTER M13 9PL England Hangzhou Dianzi Univ HDU ITMO Joint Inst Hangzhou 310018 Peoples R China Zhejiang Univ Coll Comp Sci & Technol Hangzhou 310028 Peoples R China Zhejiang Univ Sch Software Technol Ningbo 315048 Peoples R China Univ Edinburgh Sch Informat Edinburgh EH8 9AB Scotland

Both graph structures and textual information play a critical role in Knowledge Graph Completion (KGC). With the success of pre-trained language models (PLMs) such as BERT, they have been applied for text encoding for KGC. However, the current methods mostly prefer to fine-tune PLMs, leading to huge training costs and limited scalability to larger PLMs. In contrast, we propose to utilize prompts and perform KGC on a frozen PLM with only the prompts trained. Accordingly, we propose a new KGC method named PDKGC with two prompts - a hard task prompt which is to adapt the KGC task to the PLM pre-training task of token prediction, and a disentangled structure prompt which learns disentangled graph representation so as to enable the PLM to combine more relevant structure knowledge with the text information. With the two prompts, PDKGC builds a textual predictor and a structural predictor, respectively, and their combination leads to more comprehensive entity prediction. Solid evaluation on three widely used KGC datasets has shown that PDKGC often outperforms the baselines including the state-of-the-art, and its components are all effective. Our codes and data are available at https://***/genggengcss/PDKGC.

关键词： Knowledge graph completion pre-trained language model Prompt tuning Disentangled embedding

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：