检索结果-内蒙古大学图书馆

pre-trained language model-based Retrieval and Ranking forWeb Search

ACM TRANSACTIONS ON THE WEB 2023年第1期17卷 1-36页

作者： Zou, Lixin Lu, Weixue Liu, Yiding Cai, Hengyi Chu, Xiaokai Ma, Dehong Shi, Daiting Sun, Yu Cheng, Zhicong Gu, Simiu Wang, Shuaiqiang Yin, Dawei Baidu Inc Beijing Peoples R China

pre-trained language representation models (PLMs) such as BERT and Enhanced Representation through kNowledge IntEgration (ERNIE) have been integral to achieving recent improvements on various downstream tasks, including information retrieval. However, it is nontrivial to directly utilize these models for the large-scale web search due to the following challenging issues: (1) the prohibitively expensive computations ofmassive neural PLMs, especially for long texts in the web document, prohibit their deployments in the web search system that demands extremely low latency;(2) the discrepancy between existing task-agnostic pre-training objectives and the ad hoc retrieval scenarios that demand comprehensive relevance modeling is another main barrier for improving the online retrieval and ranking effectiveness;and (3) to create a significant impact on real-world applications, it also calls for practical solutions to seamlessly interweave the resultant PLM and other components into a cooperative system to serve web-scale data. Accordingly, we contribute a series of successfully applied techniques in tackling these exposed issues in this work when deploying the state-of-the-art Chinese pre-trained language model, i.e., ERNIE, in the online search engine system. We first present novel practices to perform expressive PLM-based semantic retrieval with a flexible poly-interaction scheme and cost-efficiently contextualize and rank web documents with a cheap yet powerful Pyramid-ERNIE architecture. We then endow innovative pre-training and fine-tuning paradigms to explicitly incentivize the query-document relevance modeling in PLM-based retrieval and ranking with the large-scale noisy and biased post-click behavioral data. We also introduce a series of effective strategies to seamlessly interwoven the designed PLM-based models with other conventional components into a cooperative system. Extensive offline and online experimental results show that our proposed techniques are cruci

关键词： pre-trained language model web retrieval ranking

来源：评论

学校读者我要写书评

暂无评论

pre-trained language model augmented adversarial training network for Chinese clinical event detection

引用

MATHEMATICAL BIOSCIENCES AND ENGINEERING 2020年第4期17卷 2825-2841页

作者： Zhang, Zhichang Zhang, Minyu Zhou, Tong Qiu, Yanlong Northwest Normal Univ Coll Comp Sci & Engn 967 Anning East Rd Lanzhou 730070 Peoples R China

Clinical event detection (CED) is a hot topic and essential task in medical artificial intelligence, which has attracted the attention from academia and industry over the recent years. However, most studies focus on English clinical narratives. Owing to the limitation of annotated Chinese medical corpus, there is a lack of relevant research about Chinese clinical narratives. The existing methods ignore the importance of contextual information in semantic understanding. Therefore, it is urgent to research multilingual clinical event detection. In this paper, we present a novel encoder-decoder structure based on pre-trained language model for Chinese CED task, which integrates contextual representations into Chinese character embeddings to assist model in semantic understanding. Compared with existing methods, our proposed strategy can help model harvest a language inferential skill. Besides, we introduce the punitive weight to adjust the proportion of loss on each category for coping with class imbalance problem. To evaluate the effectiveness of our proposed model, we conduct a range of experiments on test set of our manually annotated corpus. We compare overall performance of our proposed model with baseline models on our manually annotated corpus. Experimental results demonstrate that our proposed model achieves the best precision of 83.73%, recall of 86.56% and F1-score of 85.12%. Moreover, we also evaluate the performance of our proposed model with baseline models on minority category samples. We discover that our proposed model obtains a significant increase on minority category samples.

关键词： medical artificial intelligence Chinese clinical narratives Chinese clinical event detection semantic understanding class imbalance problem transfer learning pre-trained language model adversarial training network

来源：评论

学校读者我要写书评

暂无评论

pre-trained language model for Biomedical Question Answering 19th

Pre-trained Language Model for Biomedical Question Answering

引用

European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD)

作者： Yoon, Wonjin Lee, Jinhyuk Kim, Donghyeon Jeong, Minbyul Kang, Jaewoo Korea Univ Seoul South Korea

ISBN: (纸本)9783030438876;9783030438869

The recent success of question answering systems is largely attributed to pre-trained language models. However, as language models are mostly pre-trained on general domain corpora such as Wikipedia, they often have difficulty in understanding biomedical questions. In this paper, we investigate the performance of BioBERT, a pre-trained biomedical language model, in answering biomedical questions including factoid, list, and yes/no type questions. BioBERT uses almost the same structure across various question types and achieved the best performance in the 7th BioASQ Challenge (Task 7b, Phase B). BioBERT pre-trained on SQuAD or SQuAD 2.0 easily outperformed previous state-of-the-art models. BioBERT obtains the best performance when it uses the appropriate pre-/post-processing strategies for questions, passages, and answers.

关键词： Biomedical question answering pre-trained language model Transfer learning

来源：评论

学校读者我要写书评

暂无评论

pre-trained language model based Ranking in Baidu Search 21

Pre-trained Language Model based Ranking in Baidu Search

引用

27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)

作者： Zou, Lixin Zhang, Shengqiang Cai, Hengyi Ma, Dehong Cheng, Suqi Wang, Shuaiqiang Shi, Daiting Cheng, Zhicong Yin, Dawei Baidu Inc Beijing Peoples R China

ISBN: (纸本)9781450383325

As the heart of a search engine, the ranking system plays a crucial role in satisfying users' information demands. More recently, neural rankers fine-tuned from pre-trained language models (PLMs) establish state-of-the-art ranking effectiveness. However, it is non-trivial to directly apply these PLM-based rankers to the large-scale web search system due to the following challenging issues: (1) the prohibitively expensive computations of massive neural PLMs, especially for long texts in the web-document, prohibit their deployments in an online ranking system that demands extremely low latency;(2) the discrepancy between existing ranking-agnostic pre-training objectives and the ad-hoc retrieval scenarios that demand comprehensive relevance modeling is another main barrier for improving the online ranking system;(3) a real-world search engine typically involves a committee of ranking components, and thus the compatibility of the individually fine-tuned ranking model is critical for a cooperative ranking system. In this work, we contribute a series of successfully applied techniques in tackling these exposed issues when deploying the stateof-the-art Chinese pre-trained language model, i.e., ERNIE, in the online search engine system. We first articulate a novel practice to cost-efficiently summarize the web document and contextualize the resultant summary content with the query using a cheap yet powerful Pyramid-ERNIE architecture. Then we endow an innovative paradigm to finely exploit the large-scale noisy and biased post-click behavioral data for relevance-oriented pre-training. We also propose a human-anchored fine-tuning strategy tailored for the online ranking system, aiming to stabilize the ranking signals across various online components. Extensive offline and online experimental results show that the proposed techniques significantly boost the search engine's performance.

关键词： pre-trained language model Learning to Rank

来源：评论

学校读者我要写书评

暂无评论

Construction and Implementation of Knowledge Enhancement pre-trained language model for Text Sentiment Analysis

引用

Systems and Soft Computing 2025年

作者： Lan Cui Liaoning University of International Business and Economics Dalian 116502 China

With the in-depth development of natural language processing technology, text sentiment analysis has shown great potential in public opinion monitoring, market analysis and other fields. However, traditional methods have limitations in dealing with complex semantic and emotional diversity. Therefore, this study proposes a knowledge-enhanced pre-trained language model for text sentiment analysis. The model effectively improves the model's ability to understand emotional semantics by integrating external knowledge bases, such as emotional dictionaries and domain-specific knowledge. In the pre-training stage, we adopted a large-scale Chinese text dataset and combined emotional labels for joint training. The experimental results show that compared with the baseline model, the accuracy of the proposed model in the sentiment classification task is improved by 8.3%, and the F1 score is improved by 7.5%. The model performed equally well in the fine-grained sentiment analysis task, with accuracy and F1 scores improving by 6.2% and 5.8%, respectively. In addition, the model shows stronger robustness when dealing with long texts and complex emotional expressions. Further analysis shows that the knowledge enhancement module effectively improves the model's ability to recognize emotional vocabulary and tendencies. This study provides a new technical path for text sentiment analysis and a valuable exploration for applying pre-trained language models in specific fields.

关键词： Text sentiment analysis Knowledge enhancement pre-trained language model Emotion classification Fine-grained sentiment analysis

来源：评论

学校读者我要写书评

暂无评论

MEM-KGC: Masked Entity model for Knowledge Graph Completion With pre-trained language model

引用

IEEE ACCESS 2021年 9卷 132025-132032页

作者： Choi, Bonggeun Jang, Daesik Ko, Youngjoong Sungkyunkwan Univ Dept Elect & Comp Engn Suwon 16419 Gyeonggi Do South Korea Sungkyunkwan Univ Dept Comp Sci & Engn Suwon 16419 Gyeonggi Do South Korea

The knowledge graph completion (KGC) task aims to predict missing links in knowledge graphs. Recently, several KGC models based on translational distance or semantic matching methods have been proposed and have achieved meaningful results. However, existing models have a significant shortcoming-they cannot train entity embedding when an entity does not appear in the training phase. As a result, such models use randomly initialized embeddings for entities that are unseen in the training phase and cause a critical decrease in performance during the test phase. To solve this problem, we propose a new approach that performs KGC task by utilizing the masked language model (MLM) that is used for a pre-trained language model. Given a triple (head entity, relation, tail entity), we mask the tail entity and consider the head entity and the relation as a context for the tail entity. The model then predicts the masked entity from among all entities. Then, the task is conducted by the same process as an MLM, which predicts a masked token with a given context of tokens. Our experimental results show that the proposed model achieves significantly improved performances when unseen entities appear during the test phase and achieves state-of-the-art performance on the WN18RR dataset.

关键词： Task analysis predictive models Training Bit error rate Semantics Micromechanical devices Licenses Knowledge graph completion link prediction masked language model pre-trained language model

来源：评论

学校读者我要写书评

暂无评论

Knowledge-Aware Collaborative Filtering With pre-trained language model for Personalized Review-Based Rating prediction

引用

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2024年第3期36卷 1170-1182页

作者： Wang, Quanxiu Cao, Xinlei Wang, Jianyong Zhang, Wei East China Normal Univ Shanghai Inst AI Educ Sch Comp Sci & Technol Shanghai 200062 Peoples R China East China Normal Univ Shanghai Inst AI Educ Sch Comp Sci & Technol KLATASDS MOE Shanghai Peoples R China Tsinghua Univ Dept Comp Sci & Technol Beijing 100086 Peoples R China Jiangsu Normal Univ Jiangsu Collaborat Innovat Ctr Language Abil Xuzhou 221009 Peoples R China

Personalized review-based rating prediction aims at leveraging existing reviews to model user interests and item characteristics for rating prediction. Most of the existing studies mainly encounter two issues. First, the rich knowledge contained in the fine-grained aspects of each review and the knowledge graph is rarely considered to complement the pure text for better modeling user-item interactions. Second, the power of pre-trained language models is not carefully studied for personalized review-based rating prediction. To address these issues, we propose an approach named Knowledge-aware Collaborative Filtering with pre-trained language model (KCF-PLM). For the first issue, to utilize rich knowledge, KCF-PLM develops a transformer network to model the interactions of the extracted aspects w.r.t. a user-item pair. For the second issue, to better represent users and items, KCF-PLM takes all the historical reviews of a user or an item as input to pre-trained language models. Moreover, KCF-PLM integrates the transformer network and the pre-trained language models through representation propagation on the knowledge graph and user-item guided attention of the aspect representations. Thus KCF-PLM combines review text, aspect, knowledge graph, and pre-trained language models together for review-based rating prediction. We conduct comprehensive experiments on several public datasets, demonstrating the effectiveness of KCF-PLM.

关键词： Review-based rating prediction pre-trained language model collaborative filtering

来源：评论

学校读者我要写书评

暂无评论

Electric Power Audit Text Classification With Multi-Grained pre-trained language model

引用

IEEE ACCESS 2023年 11卷 13510-13518页

作者： Meng, Qinglin Song, Yan Mu, Jian Lv, Yuanxu Yang, Jiachen Xu, Liang Zhao, Jin Ma, Junwei Yao, Wei Wang, Rui Xiao, Maoxiang Meng, Qingyu State Grid Tianjin Elect Power Co Comprehens Serv Ctr Tianjin 300010 Peoples R China State Grid Tianjin Elect Power Co Tianjin 300010 Peoples R China State Grid Corp China Beijing 100031 Peoples R China Tianjin Univ Sch Elect & Informat Engn Tianjin 300072 Peoples R China Tianjin Tianyuan Power Engn Co Ltd State Grid Tianjin Elect Power Co Baodi Power Supply Branch Tianjin 301800 Peoples R China State Grid Shanxi Elect Power Co Elect Power Res Inst Taiyuan 003001 Peoples R China State Grid Shanxi Elect Power Co Informat & Commun Branch Taiyuan 030012 Peoples R China State Grid Shanxi Elect Power Co Taiyuan Power Supply Co Taiyuan 003000 Shanxi Peoples R China State Grid Tianjin Elect Power Co Chengxi Power Supply Branch Tianjin 300190 Peoples R China State Grid Tianjin Elect Power Co Ningdongshengyuan Elect Power Engn Co Ltd Ninghe Power Supply Branch Tianjin 301500 Peoples R China State Grid Jibei Elect Power Co Ltd Zhangjiakou Wanquan Dist Power Supply Branch Zhangjiakou 076261 Heibei Peoples R China

Electric power audit text classification is one of the important research problem in electric power systems. Recently, kinds of automatic classification methods for these texts based on machine learning or deep learning models have been applied. At present, the development of computing technology makes "pre-training and fine-tuning " the newest paradigm of text classification, which achieves better results than previous fully-supervised models. Based on pre-training theory, domain-related pre-training tasks can enhance the performance of downstream tasks in the specific domain. However, existing pre-training models usually use general corpus for pre-training, and do not use texts related to the field of electric power, especially electric power audit texts. This results in that the model does not learn too much electric-power-related morphology or semantics in the pre-training stage, so that less information can be used in the fine-tuning stage. Based on the research status, in this paper, we propose EPAT-BERT, a BERT-based model pre-trained by two-granularity pre-training tasks: word-level masked language model and entity-level masked language model. These two tasks predict word and entity in electric-power-related texts to learn abundant morphology and semantics about electric power. We then fine-tune EPAT-BERT for electric power audit text classification task. The experimental results show that, compared with fully supervised machine learning models, neural network models, and general pre-trained language models, EPAT-BERT can significantly outperform existing models in a variety of evaluation metrics. Therefore, EPAT-BERT can be further applied to electric power audit text classification. We also conduct ablation studies to prove the effectiveness of each component in EPAT-BERT to further illustrate our motivations.

关键词： Power systems Task analysis Text categorization Bit error rate Data models Computational modeling Natural language processing pre-trained language model text classification electric power audit text natural language processing masked language model

来源：评论

学校读者我要写书评

暂无评论

ARoBERT: An ASR Robust pre-trained language model for Spoken language Understanding

引用

IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND language PROCESSING 2022年 30卷 1207-1218页

作者： Wang, Chengyu Dai, Suyang Wang, Yipeng Yang, Fei Qiu, Minghui Chen, Kehan Zhou, Wei Huang, Jun Zhejiang Lab Hangzhou 311121 Zhejiang Peoples R China Alibaba Grp Hangzhou 311121 Zhejiang Peoples R China

Spoken language Understanding (SLU) aims to interpret the meanings of human speeches in order to support various human-machine interaction systems. A key technique for SLU is Automatic Speech Recognition (ASR), which transcribes speech signals into text contents. As the output texts of modern ASR systems unavoidably contain errors, mainstream SLU models either trained or tested on texts transcribed by ASR systems would not be sufficiently error robust. We present ARoBERT, an ASR Robust BERT model, which can be fine-tuned to solve a variety of SLU tasks with noisy inputs. To guarantee the robustness of ARoBERT, during pretraining, we decrease the fluctuations of language representations when some parts of the input texts are replaced by homophones or synophones. Specifically, we propose two novel self-supervised pre-training tasks for ARoBERT, namely Phonetically-aware Masked language modeling (PMLM) and ASR model-adaptive Masked language modeling (AMMLM). The PMLM task explicitly fuses the knowledge of word phonetic similarities into the pre-training process, which forces homophones and synophones to share similar representations. In AMMLM, a data-driven algorithm is further introduced to mine typical ASR errors such that ARoBERT can tolerate ASR model errors. In the experiments, we evaluate ARoBERT over multiple datasets. The results show the superiority of ARoBERT, which consistently outperforms strong baselines. We have also shown that ARoBERT outperforms state-of-the-arts on a public benchmark. Currently, ARoBERT has been deployed in an online production system with significant improvements.

关键词： Task analysis Transformers Phonetics Robustness predictive models Bit error rate Adaptation models ASR robust representation learning pre-trained language model spoken language understanding

来源：评论

学校读者我要写书评

暂无评论

Satellite and instrument entity recognition using a pre-trained language model with distant supervision

引用

International Journal of Digital Earth 2022年第1期15卷 1290-1304页

作者： Ming Lin Meng Jin Yufu Liu Yuqi Bai Department of Earth System Science Institute for Global Change StudiesMinistry of Education Ecological Field Station for East Asian Migratory BirdsTsinghua UniversityBeijingPeople’s Republic of China

Earth observations,especially satellite data,have produced a wealth of methods and results in meeting global challenges,often presented in unstructured texts such as papers or *** extraction of satellite and instrument entities from these unstructured texts can help to link and reuse Earth observation *** direct use of an existing dictionary to extract satellite and instrument entities suffers from the problem of poor matching,which leads to low *** this study,we present a named entity recognition model to automatically extract satellite and instrument entities from unstructured *** to the lack of manually labeled data,we apply distant supervision to automatically generate labeled training ***,we fine-tune the pre-trained language model with early stopping and a weighted cross-entropy loss *** propose the dictionary-based self-training method to correct the incomplete annotations caused by the distant supervision *** demonstrate that our method achieves significant improvements in both precision and recall compared to dictionary matching or standard adaptation of pre-trained language models.

关键词： Earth observation named entity recognition pre-trained language model distant supervision dictionary-based self-training

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：