检索结果-内蒙古大学图书馆

Improving Source Code Pre-Training via Type-Specific Masking

ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY 2025年第3期34卷 1-34页

作者： Zou, Wentao Li, Qi Li, Chuanyi Ge, Jidong Chen, Xiang Huang, Liguo Luo, Bin Nanjing Univ Natl Key Lab Novel Software Technol Nanjing Peoples R China Nantong Univ Sch Artificial Intelligence & Comp Sci Nantong Peoples R China Southern Methodist Univ Dept Comp Sci Dallas TX USA

The masked language modeling (MLM) task is widely recognized as one of the most effective pre-training tasks and currently derives many variants in the Software Engineering (SE) field. However, most of these variants mainly focus on code representation without distinguishing between different code token types, while some focus on a specific type, such as code identifiers. Indeed, various code token types exist, and there is no evidence that only identifiers can improve PTMs. Thus, to improve PTMs through different types, we conducted an extensive study to evaluate how different type-specific masking tasks can affect PTMs. First, we extract five code token types, convert them into type-specific masking tasks, and generate their combinations. Second, we pre-train CodeBERT and PLBART using combinations and fine-tuned them on four SE downstream tasks. Experimental results show that type-specific masking tasks can enhance CodeBERT and PLBART on all downstream tasks. Furthermore, we discuss topics related to low-resource datasets, conflicting PTMs that original pre-training tasks conflict with our methods, the cost and performance of our methods, factors that impact the performance of our methods, and applying our methods on state-of-the-art PTMs. These discussions comprehensively analyze the strengths and weaknesses of different type-specific masking tasks. CCS Concepts: center dot Software and its engineering- Software creation and management;center dot Computing methodologies- Artificial intelligence

关键词： code representation learning pre-trained model pre-training tasks masked language modeling

来源：评论

学校读者我要写书评

暂无评论

Argument Structure Mining Based on Effective Usage of Contextual Information

引用

KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS 2025年第1期19卷 1-16页

作者： Xu, Menglong Zhang, Yanliang Yang, Yapu Henan Polytech Univ Sch Phys & Elect Informat Engn Jiaozuo 454000 Peoples R China Xu Ji Elect Co Ltd Xuchang 461000 Peoples R China

Argument Structure Extraction (ASE) is increasingly prominent for its role in identifying discourse structure within documents. Many pioneering works have demonstrated that the contextual information in the document is vital for the final performance of ASE. Traditional context-aware methods relying on concatenation of contextual sentences have proven insufficient, introducing noise, inefficiency, and bias due to the reliance on discourse markers. To overcome these issues, we introduce Efficient Argument Structure Extraction (E-ASE), which eschews sentence concatenation in favor of encoding sentences separately and applying sentence-level attention to integrate context. To mitigate discourse marker bias, E-ASE employs a novel data augmentation technique, substituting discourse markers with a [MASK] token and leveraging masked language modeling (MLM) loss. Our empirical research, conducted across five diverse datasets, demonstrates E-ASE's state-of-the-art (SOTA) performance, save for on the ECHR dataset, marking a significant advancement in the field of ASE by optimizing contextual information usage and enhancing both the training and inference processes.

关键词： Argument structure extraction Contextual information Discourse markers masked language modeling Data augmentation

来源：评论

学校读者我要写书评

暂无评论

Written Term Detection Improves Spoken Term Detection

引用

IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND language PROCESSING 2024年 32卷 3213-3223页

作者： Yusuf, Bolaji Saraclar, Murat Bogazici Univ Dept Elect & Elect Engn TR-34342 Istanbul Turkiye Brno Univ Technol Fac Informat Technol SpeechFIT Brno 61200 Czech Republic

End-to-end (E2E) approaches to keyword search (KWS) are considerably simpler in terms of training and indexing complexity when compared to approaches which use the output of automatic speech recognition (ASR) systems. This simplification however has drawbacks due to the loss of modularity. In particular, where ASR-based KWS systems can benefit from external unpaired text via a language model, current formulations of E2E KWS systems have no such mechanism. Therefore, in this paper, we propose a multitask training objective which allows unpaired text to be integrated into E2E KWS without complicating indexing and search. In addition to training an E2E KWS model to retrieve text queries from spoken documents, we jointly train it to retrieve text queries from masked written documents. We show empirically that this approach can effectively leverage unpaired text for KWS, with significant improvements in search performance across a wide variety of languages. We conduct analysis which indicates that these improvements are achieved because the proposed method improves document representations for words in the unpaired text. Finally, we show that the proposed method can be used for domain adaptation in settings where in-domain paired data is scarce or nonexistent.

关键词： Training Hidden Markov models Computational modeling Feature extraction Keyword search Complexity theory Task analysis spoken term detection keyword spotting end-to-end keyword search multitask learning domain adaptation masked language modeling

来源：评论

学校读者我要写书评

暂无评论

SNP-S³: Shared Network Pre-Training and Significant Semantic Strengthening for Various Video-Text Tasks

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2024年第4期34卷 2525-2535页

作者： Dong, Xingning Guo, Qingpei Gan, Tian Wang, Qing Wu, Jianlong Ren, Xiangyuan Cheng, Yuan Chu, Wei Shandong Univ Sch Comp Sci & Technol Qingdao 266237 Peoples R China Ant Grp Co Ltd Hangzhou 310023 Peoples R China Harbin Inst Technol Shenzhen Sch Comp Sci & Technol Shenzhen 518055 Peoples R China Fudan Univ Artificial Intelligence Innovat & Incubat Inst AI3 Shanghai 200433 Peoples R China

We present a framework for learning cross-modal video representations by directly pre-training on raw data to facilitate various downstream video-text tasks. Our main contributions lie in the pre-training framework and proxy tasks. First, based on the shortcomings of two mainstream pixel-level pre-training architectures (limited applications or less efficient), we propose Shared Network Pre-training (SNP). By employing one shared BERT-type network to refine textual and cross-modal features simultaneously, SNP is lightweight and could support various downstream applications. Second, based on the intuition that people always pay attention to several "significant words" when understanding a sentence, we propose the Significant Semantic Strengthening (S-3) strategy, which includes a novel masking and matching proxy task to promote the pre-training performance. Experiments conducted on three downstream video-text tasks and six datasets demonstrate that, we establish a new state-of-the-art in pixel-level video-text pre-training;we also achieve a satisfactory balance between the pre-training efficiency and the fine-tuning performance. The codebase and pre-trained models are available at https://***/dongxingning/SNPS3.

关键词： Video-text pre-training vision and language masked language modeling video-text matching

来源：评论

学校读者我要写书评

暂无评论

Improving edit-based unsupervised sentence simplification using fine-tuned BERT

引用

PATTERN RECOGNITION LETTERS 2023年第1期166卷 112-118页

作者： Rashid, Mohammad Amin Amirkhani, Hossein Univ Qom Dept Comp Engn Qom Iran

Word suggestion in unsupervised sentence simplification aims to replace complex words of a given sen-tence with their simpler alternatives. This is mostly done without considering their context within the input sentence. In this paper, we propose a technique that brings context awareness to word suggestion by merging pre-trained BERT models with a successful edit-based unsupervised sentence simplification model. More importantly, we show that only by fine-tuning the BERT model on simple English corpora, simplification results can be improved and even outperform some of the competing supervised methods. Finally, we introduce a framework that involves filtering an arbitrary amount of unlabeled in-domain text for tuning the model in situations where labeled data, as simple and complex, is scarce. This preprocess-ing step also speeds up the training process by ignoring fine-tuning on unnecessary samples.(c) 2023 Elsevier B.V. All rights reserved.

关键词： Unsupervised sentence simplification masked language modeling Domain adaptation BERT Data selection Natural language processing

来源：评论

学校读者我要写书评

暂无评论

Hypert: hypernymy-aware BERT with Hearst pattern exploitation for hypernym discovery

引用

JOURNAL OF BIG DATA 2023年第1期10卷 141页

作者： Yun, Geonil Lee, Yongjae Moon, A-Seong Lee, Jaesung Chung Ang Univ Dept Artificial Intelligence 84 Heukseok Ro Seoul 06974 South Korea S2W Inc 12 Pangyoyeok Ro192 beon Gil Seongnam 13524 Gyeonggi Do South Korea Chung Ang Univ AI ML Innovat Res Ctr 84 Heukseok Ro Seoul 06974 South Korea

Hypernym discovery is challenging because it aims to find suitable instances for a given hyponym from a predefined hypernym vocabulary. Existing hypernym discovery methods used supervised learning with word embedding from word2vec. However, word2vec embedding suffers from low embedding quality regarding unseen or rare noun phrases because entire noun phrases are embedded into a single vector. Recently, prompting methods have attempted to find hypernyms using pretrained language models with masked prompts. Although language models alleviate the problem of w embeddings, general-purpose language models are ineffective for capturing hypernym relationships. Considering the hypernym relationship to be a linguistic domain, we introduce Hypert, which is further pretrained using masked language modeling with Hearst pattern sentences. To the best of our knowledge, this is the first attempt in the hypernym relationship discovery field. We also present a fine-tuning strategy for training Hypert with special input prompts for the hypernym discovery task. The proposed method outperformed the comparison methods and achieved statistically significant results in three subtasks of hypernym discovery. Additionally, we demonstrate the effectiveness of the several proposed components through an in-depth analysis. The code is available at: https://***/Gun1Yun/Hypert.

关键词： Hypernym discovery Hypernym relationship language model masked language modeling Hearst pattern Natural language processing

来源：评论

学校读者我要写书评

暂无评论

GDP: Generic Document Pretraining to Improve Document Understanding 18th

GDP: Generic Document Pretraining to Improve Document Unders...

引用

18th International Conference on Document Analysis and Recognition (ICDAR)

作者： Trivedi, Akkshita Upadhyay, Akarsh Mukhopadhyay, Rudrabha Chaudhury, Santanu Indian Inst Technol Jodhpur Karwar India Int Inst Informat Technol Hyderabad India

ISBN: (纸本)9783031705328;9783031705335

In this paper, we propose a novel pretraining approach for document analysis that advances beyond conventional methods. The approach, called the GDPerformer, trains a suite of unique architectures to predict both masked OCR tokens and masked OCR bounding boxes, fostering the network to learn document semantics such as structure and language. Our experiments with GDPerformerv1 and GDPerformerv2 show enhanced performance on various downstream tasks, including Semantic Entity Recognition and Extraction and Multi-Modal Document Classification with minimal task-specific data and generalization to a wide range of documents. Furthermore, our pretrained features exhibit robustness in handling noisy documents and can be easily extended to multiple languages. Our experiments indicate that the proposed pretraining strategy requires only 50K document images, making it particularly beneficial for low-resource languages.

关键词： Pretraining Document Analysis masked language modeling Document Transformers Document Imaging Deep Learning masked Layout modeling

来源：评论

学校读者我要写书评

暂无评论

High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group masked language Model 25

High Fidelity Text-to-Speech Via Discrete Tokens Using Token...

引用

25th Interspeech Conference

作者： Lee, Joun Yeop Jeong, Myeonghun Kim, Minchan Lee, Ji-Hyun Choi, Hoon-Young Kim, Nam Soo Samsung Res Seoul South Korea Seoul Natl Univ Dept ECE & INMC Seoul South Korea

We propose a novel two-stage text-to-speech (TTS) framework with two types of discrete tokens, i.e., semantic and acoustic tokens, for high-fidelity speech synthesis. It features two core components: the Interpreting module, which processes text and a speech prompt into semantic tokens focusing on linguistic contents and alignment, and the Speaking module, which captures the timbre of the target voice to generate acoustic tokens from semantic tokens, enriching speech reconstruction. The Interpreting stage employs a transducer for its robustness in aligning text to speech. In contrast, the Speaking stage utilizes a Conformer-based architecture integrated with a Grouped masked language Model (G-MLM) to boost computational efficiency. Our experiments verify that this innovative structure surpasses the conventional models in the zero-shot scenario in terms of speech quality and speaker similarity.

关键词： speech synthesis neural transducer masked language modeling semantic token acoustic token

来源：评论

学校读者我要写书评

暂无评论

Exploring terminological relations between multi-word terms in distributional semantic models

引用

TERMINOLOGY 2024年第2期30卷 159-189页

作者： Wang, Yizhe Daille, Beatrice Hathout, Nabil Univ Toulouse Jean Jaures Toulouse France Nantes Univ Nantes France CNRS Lab Cognit Langues Langage Ergonomie CLLE Paris France Nantes Univ Dept Informat LS2N 2 Chemin HoussiniereBP 92208 F-44322 Nantes 3 France

A term is a lexical unit with specialized meaning in a particular domain. Terms may be simple (STs) or multi-word (MWTs). The organization of terms gives a representation of the structure of domain knowledge, which is based on the relationships between the concepts of the domain. However, relations between MWTs are often underrepresented in terminology resources. This work aims to explore distributional semantic models for capturing terminological relations between multi-word terms through lexical substitution and analogy. The experiments show that the results of the analogy-based method are globally better than those of the one based on lexical substitution and that analogy is well suited to the acquisition of synonymy, antonymy, and hyponymy while lexical substitution performs best for hypernymy.

关键词： terminology relations multi-word terms analogy lexical substitution FastText masked language modeling transformer models environment domain

来源：评论

学校读者我要写书评

暂无评论

TVD-BERT: A Domain-Adaptation Pre-trained Model for Textural Vulnerability Descriptions 20th

TVD-BERT: A Domain-Adaptation Pre-trained Model for Textural...

引用

20th International Conference on Intelligent Computing (ICIC)

作者： Wang, Ziyuan Liang, Xiaoyan Du, Ruizhong Tian, Junfeng Zhang, Siyi Hebei Univ Key Lab High Trusted Informat Syst Hebei Prov Baoding Peoples R China Hebei Univ Sch Cyber Secur & Comp Baoding Peoples R China

ISBN: (纸本)9789819756629;9789819756636

Textual Vulnerability Descriptions (TVD) serve as concise natural language summaries within databases like the National Vulnerability Database(NVD), elucidating key facets of software vulnerabilities including impacted products, causation, and impacts. Collecting and analyzing TVD holds significant importance in the field of cybersecurity. In order to better analyze TVD, identify potential trends in software vulnerabilities, and reduce the occurrence of software vulnerabilities, we present a domain-adaptive pre-trained model for textual vulnerability descriptions. First, we collected the most common words in the textual vulnerability descriptions through Term Frequency (TF) method and expanded the BERT vocabulary. Then we adapted the model to textual vulnerability descriptions language (TVDL) and named the trained model TVD-BERT. To further refine the model's adaptability and representational capabilities, we introduce an innovative approach named Double Lexicon Masking, underpinning our masked language modeling (MLM). Our experimental results on tasks related to TVD demonstrate that TVD-BERT can achieve state-of-the-art performance, underscoring its superiority over other methods.

关键词： TVD BERT Domain Adaptation masked language modeling

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：