检索结果-内蒙古大学图书馆

1st International Conference on Image Processing Machine Learning and Pattern Recognition

作者： Zhang, Chuancai Qu, Dan Du, Liming Yang, Kaiyuan Univ Informat Engn Sch Informat Syst Engn Zhengzhou Peoples R China Lab Adv Comp & Intelligence Engn Wuxi Jiangsu Peoples R China Zhengzhou Univ Sch Cyberspace Secur Zhengzhou Peoples R China

ISBN: (纸本)9798400707032

This study proposes an unsupervised machine translation method based on a dynamic adaptive masking strategy and multi-task learning. Firstly, a dynamic adaptive masking strategy is introduced in masked language modeling, dynamically adjusting the masking rate based on sentence complexity and contextual information to retain more important information in complex sentences while optimizing the masking effect. Secondly, a multi-task learning framework is adopted, incorporating tasks such as translation, summarization, and question answering into the unsupervised translation model. By using a shared encoder and task-specific decoders for joint training, the model enhances both generalization and task-specific capabilities. Experimental results demonstrate that the proposed approach significantly improves translation quality and model generalization across multiple translation tasks, providing new insights and application prospects for unsupervised machine translation research.

关键词： Unsupervised Machine Translation Dynamic Adaptive Masking Strategy Multi-Task Learning masked language modeling Translation Quality Improvement

来源：评论

学校读者我要写书评

暂无评论

Self-supervised Learning and masked language Model for Code-switching Automatic Speech Recognition

Self-supervised Learning and Masked Language Model for Code-...

引用

10th IEEE International Conference on Communications and Electronics (IEEE ICCE)

作者： Chen, Po-Kai Fu, Li-Yeh Chen, Cheng-Kai Lin, Yi-Xing Chen, Chih-Ping Huang, Chien-Lin Wang, Jia-Ching Natl Cent Univ Dept CSIE Taoyuan Taiwan Realtek Semicond Corp Hsinchu Taiwan Natl Cheng Kung Univ Dept CSIE Tainan Taiwan

ISBN: (纸本)9798350379808;9798350379792

Code-switching (CS) is a common linguistic phenomenon that poses significant challenges for automatic speech recognition systems due to the lack of corpus. In this paper, we propose a novel approach to address this challenge by leveraging self-supervised learning (SSL) and the masked language model (MLM) in speech recognition. Specifically, we use the wav2vec 2.0 pre-trained model to reduce the dependency on CS labeled data, and the MLM to rerank sentences generated using beam search decoding. Our proposed method is evaluated on the SEAME dataset, and experimental results show that it outperforms state-of-the-art CS speech recognition approaches by 15.6% and 19.9% in terms of token error rates (TER). Moreover, the proposed method is generalizable and can be extended to other CS languages. These results demonstrate the effectiveness of our approach and its potential for future research in the field of CS speech recognition.

关键词： code-switching speech recognition self-supervised learning masked language modeling

来源：评论

学校读者我要写书评

暂无评论

Extended Abstract: Predicting the Morality of a Character Using Character-Centric Embeddings

Extended Abstract: Predicting the Morality of a Character Us...

引用

IEEE International Conference on Big Data and Smart Computing (BigComp)

作者： Bae, Su Young Kim, Eun Chong Cheong, Yun Gyung Sungkyunkwan Univ Dept AI Suwon South Korea

ISBN: (纸本)9781665475785

Story generation and analysis have been researched for decades using AI and Natural language Processing technology. However, while ethics is becoming essential for developing AI applications, little research deals with morality in the narrative. We present the framework to build embeddings for representing characters in terms of the character's morality. In addition, we propose a morality judgment task using storybooks. We conduct experiments, and the results suggest that word embedding models can learn a character's morality.

关键词： Morals Story Character Embedding masked language modeling

来源：评论

学校读者我要写书评

暂无评论

POSPAN: Position-Constrained Span Masking for language Model Pre-training 23

POSPAN: Position-Constrained Span Masking for Language Model...

引用

32nd ACM International Conference on Information and Knowledge Management (CIKM)

作者： Zhang, Zhenyu Shen, Lei Zhao, Yuming Chen, Meng He, Xiaodong JD AI Res Beijing Peoples R China

ISBN: (纸本)9798400701245

Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies are critical to language understanding. Previous works only consider span length with some discrete distributions, while the dependencies among spans are ignored, i.e., assuming that the positions of masked spans are uniformly distributed. In this paper, we present POSPAN, a general framework to allow diverse position-constrained span masking strategies via the combination of span length distribution and position constraint distribution, which unifies all existing span-level masking methods. To verify the effectiveness of POSPAN in pre-training, we evaluate it on the datasets from several NLU benchmarks. Experimental results indicate that the position constraint is capable of enhancing span-level masking broadly, and our best POSPAN setting consistently outperforms its span-length-only counterparts and vanilla MLM. We also conduct theoretical analysis for the position constraint in masked language models to shed light on the reason why POSPAN works well, demonstrating the rationality and necessity of POSPAN.

关键词： masked language modeling pre-training span length distribution position constraint distribution

来源：评论

学校读者我要写书评

暂无评论

ASBERT: ASR-SPECIFIC SELF-SUPERVISED LEARNING WITH SELF-TRAINING

ASBERT: ASR-SPECIFIC SELF-SUPERVISED LEARNING WITH SELF-TRAI...

引用

IEEE Spoken language Technology Workshop (SLT)

作者： Kim, Hyung Yong Kim, Byeong-Yeol Yoo, Seung Woo Lim, Youshin Lim, Yunkyu Lee, Hanbin 42Dot Inc Seoul South Korea

ISBN: (纸本)9798350396904

Pre-training of self-supervised learning (SSL) generally shows a good performance on various speech processing tasks. However, this pre-training scheme may lead to a sub-optimal solution for fine-tuning a specific task, such as automatic speech recognition (ASR). In order to provide a more optimal pre-trained model for ASR, we introduce an ASR-Specific hidden-unit BERT with self-training, namely ASBERT. Motivated by self-training, we extract linguistic-related pseudo labels from the fine-tuned model, and these labels are used in the next pre-training procedure. Experimental results on LibriSpeech test-clean and test-other datasets show that ASBERT without language model (LM) outperforms the conventional SSL and self-training model, achieving a 6.3/2.0% and 15.4/13.2% relatively word error rate reduction (RERR). Moreover, without using pseudo-transcription, ASBERT yields comparable performance to the conventional self-training method.

关键词： self-supervised learning self-training automatic speech recognition masked language modeling hidden-unit BERT

来源：评论

学校读者我要写书评

暂无评论

A Comparative Study of BERT-Based Attention Flows Versus Human Attentions on Fill-in-Blank Task 24th

A Comparative Study of BERT-Based Attention Flows Versus Hum...

引用

24th International Conference on Human-Computer Interaction (HCII)

作者： Qian, Ming Lee, Ka Wai Pathfinders Translat & Interpretat Res Cary NC 27519 USA Univ Illinois Champaign IL 61820 USA

ISBN: (纸本)9783031217067;9783031217074

The purpose of this small-scale study is to compare BERT language model's attention flow-which quantifies the marginal contributions from each word and aggregated word groups towards the fill-in-blank prediction- with human evaluators' opinions on the same task. Based on a limited number of experiments performed, we have the following findings: (1) Compared with human evaluators, BERT base model pay less attention towards verbs, and more attention towards noun and other word types. That seems to agree with the natural partition hypothesis: nouns predominate over verbs in children's initial vocabularies because it is easy to understand the meanings of nouns. The premise of such hypothesis is that BERT base model performs like a human child. (2) As sentences become longer and more complex, human evaluators can distinguish the major logic relation and be less distracted by other components in the structure. The attention flow scores calculated using the BERT base model, on the other hand, amortize towards multiplewords andword groups as sentences become longer and more complex. (3) Amortized attention flow scores calculated using BERT base model provides a balanced global view towards different types of discourse relations embedded in long and complex sentences. For future works, more examples will be prepared for detailed and rigorous verifications on the findings.

关键词： Interpretability Explainability Attention Attention flow language modeling masked language modeling BERT Linguistics Fill-in-blank task Natural language processing

来源：评论

学校读者我要写书评

暂无评论

A New Approach for Text Style Transfer Based on Deletion and Generation

A New Approach for Text Style Transfer Based on Deletion and...

引用

Euro-Asia Conference on Frontiers of Computer Science and Information Technology (FCSIT)

作者： Liu, Guoqiang Xiao, Ruochen Li, Wenkai Zhang, Youcheng Blended Learning MIT Apple Sin Project Cambridge MA 02139 USA

ISBN: (纸本)9781665463539

We propose using the Sequence Classification modeling, SHAP algorithm and masked-language modeling (MLM) for the task of text style transfer. To tackle cases when no parallel source-target pairs are available, we train Sequence Classification model based on Bert model with SST-2 task of GLUE for both source and target domain;and we use SHAP values, which are computed based on Sequence Classification model we gained, to detect and then delete words associated with original attributes. The deleted tokens are replaced by MLM trained with the target domain to retrieve new phrases associated with the target attributes. Based on this, we detect the part of speech (POS) of each word in the sentence in order to replace the suitable positions without much impact on the semantics. Additionally, we use GloVe to determine semantic similarity between the word generated by MLM and the original word so that we can trade off content versus attribute by using grid search to gain their weighting percentage. The experiments show that our methods improve style conversion rate by 9.7% and get a semantic similarity compared to original contents 28.2% on average higher than best previous system.

关键词： masked language modeling SHAP algorithm Glove Spacy Sequence Classification modeling

来源：评论

学校读者我要写书评

暂无评论

Unsupervised Representation Learning of Player Behavioral Data with Confidence Guided Masking 22

Unsupervised Representation Learning of Player Behavioral Da...

引用

31st ACM Web Conference (WWW)

作者： Pu, Jiashu Lin, Jianshi Mao, Xiaoxi Tao, Jianrong Shen, Xudong Shang, Yue Wu, Runze NetEase Inc Fuxi AI Lab Hangzhou Peoples R China

ISBN: (纸本)9781450390965

Players of online games generate rich behavioral data during gaming. Based on these data, game developers can build a range of data science applications, such as bot detection and social recommendation, to improve the gaming experience. However, the development of such applications requires data cleansing, training sample labeling, feature engineering, and model development, which makes the use of such applications in small and medium-sized game studios still uncommon. While acquiring supervised learning data is costly, unlabeled behavioral logs are often continuously and automatically generated in games. Thus we resort to unsupervised representation learning of player behavioral data to optimize intelligent services in games. Behavioral data has many unique properties, including semantic complexity, excessive length, etc. A worth noting property within raw player behavioral data is that a lot of it is task-irrelevant. For these data characteristics, we introduce a BPE-enhanced compression method and propose a novel adaptive masking strategy called Masking by Token Confidence (MTC) for the masked language modeling (MLM) pre-training task. MTC is designed to increase the masking probabilities of task-relevant tokens. Experiments on four downstream tasks and successful deployment in a world-renowned Massively Multiplayer Online Role-Playing Game (MMORPG) prove the effectiveness of the MTC strategy1.

关键词： User modeling unsupervised Pre-trained Transformer Encoder sequence compression masked language modeling Online Games

来源：评论

学校读者我要写书评

暂无评论

Efficient Text Style Transfer Through Robust masked language Model and Iterative Inference

引用

IEEE ACCESS 2024年 12卷 182353-182373页

作者： Subhani Khan, Osama Iltaf, Naima Zia, Usman Latif, Rabia Shahida Mohd Jamail, Nor Natl Univ Sci & Technol Islamabad 44000 Pakistan Prince Sultan Univ Coll Comp & Informat Sci CCIS Artificial Intelligence & Data Analyt Lab Riyadh 11586 Saudi Arabia

Emergence of Large language Models (LLMs) have prompted researchers to exploit the abilities of such models for text style transfer (TST). However, these models are prone to hallucinations and suffer from problems of manually crafting prompts and high computation requirements. The purpose of TST is to edit text sequences so that their style is changed without hindering the meaning of their content. Owing to the scarcity of parallel data, existing approaches rely on various strategies to identify and replace style attributes or to edit a given sequence as a whole. Successful style transfer should be fluent and reflect original content. To address these challenges, we propose a novel technique leveraging explanations of prompt-free few-shot contrastive learning based lightweight classifier. First, we create a style-independent corpus of target style sequences by masking out style attributes and train a generator with masked language modeling objective that learns to predict target style tokens. Then, we apply an iterative mechanism to mask source style sequences and predict target style attributes until style is transferred gauged by a pre-trained evaluator model. We conduct experiments on two real world widely used product reviews sentiment datasets on both polarities, i.e., positive to negative and negative to positive. Comparison with various prompt-based as well as unsupervised learning based methods demonstrate state-of-the-art performance of our approach.

关键词： Iterative methods Accuracy Unsupervised learning Transformers Computational modeling Semantics Training Predictive models Generators Encoding Text style transfer generative AI contrastive learning masked language modeling explainable AI

来源：评论

学校读者我要写书评

暂无评论

TET: Text emotion transfer

引用

KNOWLEDGE-BASED SYSTEMS 2023年第1期262卷

作者： Baghmolaei, Rezvan Mohammadi Ahmadi, Ali KN Toosi Univ Technol Fac Comp Engn Tehran Iran

Text style transfer aims at transforming the style of a piece of text while keeping its primary content. The style of the text is usually defined as a particular writing tone in different categories, such as formality, politeness, sentiment, and political slant. Recently, most of the work in the area has been devoted to the problem of sentiment transfer, which tries to transfer an opinionated text into a positive or negative perspective. It has applications in marketing, political news, chatbots, writing tools, and many others. On the other hand, emotions as the basic forms of sentiments have brought many attentions to different tasks, including image style transfer but they are not well expressed in text style transfer yet. This article presents a text emotion transfer model that transforms the style of a text to each of the predefined 'anger', 'fear', 'joy', and 'sadness' emotions. Relying on masked language modeling and transfer learning, the proposed model can perform efficiently on limited amounts of emotion-annotated data. Moreover, the model shows promising experimental results against other existing models considering style transfer accuracy, content preservation, and fluency in the ISEAR and TEC emotion corpora.(c) 2022 Elsevier B.V. All rights reserved.

关键词： Natural language generation Text style transfer Emotion recognition Transfer learning masked language modeling Transformers

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：