检索结果-内蒙古大学图书馆

MindLLM: Pre-training Lightweight Large language Model from Scratch, Evaluations and Domain applications

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yang, Yizhe Sun, Huashan Li, Jiawei Liu, Runheng Li, Yinghao Liu, Yuhang Gao, Yang Huang, Heyan School of Computer Science Beijing Institute of Technology Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications Beijing Institute of Technology Southeast Academy of Information Technology China

Large language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing increasingly large-scale models, there could be another branch to develop lightweight custom models that better serve certain domains, taking into account the high cost of training and deploying LLMs and the scarcity of resources. In this paper, we present MindLLM, a novel series of bilingual lightweight large language models, trained from scratch, alleviating such burdens by offering models with 1.3 billion and 3 billion parameters. A thorough account of experiences accrued during large model development is given, covering every step of the process, including data construction, model architecture, evaluation, and applications. Such insights are hopefully valuable for fellow academics and developers. MindLLM consistently matches or surpasses the performance of other open-source larger models on some public benchmarks. We also introduce an innovative instruction tuning framework tailored for smaller models to enhance their capabilities efficiently. Moreover, we explore the application of MindLLM in specific vertical domains such as law and finance, underscoring the agility and adaptability of our lightweight models. Copyright © 2023, The Authors. All rights reserved.

关键词： Benchmarking

Incorporating Option and Out-of-domain Knowledge for Multi-choice Machine Reading Comprehension 7

学校读者我要写书评

暂无评论

Incorporating Option and Out-of-domain Knowledge for Multi-c...

7th IEEE International Conference on cloud computing and Intelligence Systems, CCIS 2021

作者： Xu, Yuan Shi, Shumin Huang, Heyan Beijing Institute of Technology School of Computer Science and Technology Beijing China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications Beijing China

ISBN: (纸本)9781665441490

Multi-choice Machine Reading Comprehension (MRC) requires the model to select the correct answer from a set of answer candidates given the corresponding passage and question. Previous studies mainly focus on complex matching networks to model the relationship among options, passage and question. However, these models obtain little improvement over the powerful Pre-trained language Models (PLMs). In this paper, we propose a simple method to incorporate option knowledge from PLMs and introduce out-of-domain knowledge by multi-task learning skillfully. Our approach obtains state-of-the-art results on Chinese multi-choice MRC dataset ReCO and also effectively improves the performance on C3. © 2021 IEEE.

关键词： Domain Knowledge

MindLLM: Lightweight Large language Model Pre-Training, Evaluation and Domain Application

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Yang, Yizhe Sun, Huashan Li, Jiawei Liu, Runheng Li, Yinghao Liu, Yuhang Gao, Yang Huang, Heyan School of Computer Science Beijing Institute of Technology Beijing China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications Beijing China Beijing Institute of Technology Southeast Academy of Information Technology Fujian Putian China

关键词： Artificial intelligence

Boosting legal case retrieval by query content selection with large language models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhou, Youchao Huang, Heyan Wu, Zhijing School of Computer Science and Technology Beijing Institute of Technology Southeast Academy of Information Technology Beijing Institute of Technology Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications China

Legal case retrieval, which aims to retrieve relevant cases to a given query case, benefits judgment justice and attracts increasing attention. Unlike generic retrieval queries, legal case queries are typically long and the definition of relevance is closely related to legal-specific elements. Therefore, legal case queries may suffer from noise and sparsity of salient content, which hinders retrieval models from perceiving correct information in a query. While previous studies have paid attention to improving retrieval models and understanding relevance judgments, we focus on enhancing legal case retrieval by utilizing the salient content in legal case queries. We first annotate the salient content in queries manually and investigate how sparse and dense retrieval models attend to those content. Then we experiment with various query content selection methods utilizing large language models (LLMs) to extract or summarize salient content and incorporate it into the retrieval models. Experimental results show that reformulating long queries using LLMs improves the performance of both sparse and dense models in legal case retrieval. © 2023, CC BY-NC-SA.

关键词： Computational linguistics

Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Chenxu Jian, Ping Huang, Mu School of Computer Science and Technology Beijing Institute of Technology Beijing China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications Beijing Institute of Technology Beijing China

Implicit Discourse Relation Recognition (IDRR), which infers discourse relations without the help of explicit connectives, is still a crucial and challenging task for discourse parsing. Recent works tend to exploit the hierarchical structure information from the annotated senses, which demonstrate enhanced discourse relation representations can be obtained by integrating sense hierarchy. Nevertheless, the performance and robustness for IDRR are significantly constrained by the availability of annotated data. Fortunately, there is a wealth of unannotated utterances with explicit connectives, that can be utilized to acquire enriched discourse relation features. In light of such motivation, we propose a Prompt-based Logical Semantics Enhancement (PLSE) method for IDRR. Essentially, our method seamlessly injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction. Furthermore, considering the prompt-based connective prediction exhibits local dependencies due to the deficiency of masked language model (MLM) in capturing global semantics, we design a novel self-supervised learning objective based on mutual information maximization to derive enhanced representations of logical semantics for IDRR. Experimental results on PDTB 2.0 and CoNLL16 datasets demonstrate that our method achieves outstanding and consistent performance against the current state-of-the-art models. Copyright © 2023, The Authors. All rights reserved.

关键词： Semantics

Unsupervised Style Control for Image Captioning

学校读者我要写书评

暂无评论

Unsupervised Style Control for Image Captioning

作者： Junyu Tian Zhikun Yang Shumin Shi School of Computer Science and Technology Beijing Institute of Technology Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications

We propose a novel unsupervised image captioning *** captioning involves two fields of deep learning,natural language processing and computer *** excessive pursuit of model evaluation results makes the caption style generated by the model too monotonous,which is difficult to meet people's demands for vivid and stylized image ***,we propose an image captioning model that combines text style transfer and image emotion recognition methods,with which the model can better understand images and generate controllable stylized *** proposed method can automatically judge the emotion contained in the image through the image emotion recognition module,better understand the image content,and control the description through the text style transfer method,thereby generating captions that meet people's *** our knowledge,this is the first work to use both image emotion recognition and text style control.

关键词：

Reducing Length Bias in Scoring Neural Machine Translation via a Causal Inference Method 20

学校读者我要写书评

暂无评论

Reducing Length Bias in Scoring Neural Machine Translation v...

20th China National Conference on Computational Linguistics, CCL 2021

作者： Shi, Xuewen Huang, Heyan Jian, Ping Tang, Yi-Kun School of Computer Science and Technology Beijing Institute of Technology Beijing100081 China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications Beijing100081 China

ISBN: (纸本)9783030841850

Neural machine translation (NMT) usually employs beam search to expand the searching space and obtain more translation candidates. However, the increase of the beam size often suffers from plenty of short translations, resulting in dramatical decrease in translation quality. In this paper, we handle the length bias problem through a perspective of causal inference. Specifically, we regard the model generated translation score S as a degraded true translation quality affected by some noise, and one of the confounders is the translation length. We apply a Half-Sibling Regression method to remove the length effect on S, and then we can obtain a debiased translation score without length information. The proposed method is model agnostic and unsupervised, which is adaptive to any NMT model and test dataset. We conduct the experiments on three translation tasks with different scales of datasets. Experimental results and further analyses show that our approaches gain comparable performance with the empirical baseline methods. © 2021, Springer Nature Switzerland AG.

关键词： Neural machine translation

Exploiting Knowledge Embedding to Improve the Description for Image Captioning 5th

学校读者我要写书评

暂无评论

Exploiting Knowledge Embedding to Improve the Description fo...

5th China Conference on Knowledge Graph, and Semantic computing, CCKS 2020

作者： Song, Dandan Peng, Cuimei Yang, Huan Liao, Lejian Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications Beijing Key Laboratory of Intelligent Information Technology School of Computer Science and Technology Beijing Institute of Technology Beijing China

ISBN: (纸本)9789811619632

Most existing methods for image captioning are based on the encoder-decoder framework which directly translates visual features into sentences, without exploiting commonsense knowledge available in the form of knowledge graph. Inspired by the success of information retrieval and question answering systems that leverage prior knowledge, we explore a knowledge embedding approach for image captioning. In this paper, we propose a Knowledge Embedding with Attention on Attention (KE-AoA) method for image captioning, which judges whether or how well the objects are related and augments semantic correlations and constraints between them. The KE-AoA method combines knowledge base method (TransE) and text method (Skip-gram), adding external knowledge graph information (triplets) into the language model to guide the learning of word vectors as the regularization term. Then it employs the AoA module to model the relations among different objects. As more inherent relations and commonsense knowledge are learned, the model can generate better image descriptions. The experiments on MSCOCO data sets achieve a significant improvement on the existing methods and validate the effectiveness of our prior knowledge-based approach. © 2021, Springer Nature Singapore Pte Ltd.

关键词： Knowledge graph

SciMRC: Multi-perspective Scientific Machine Reading Comprehension

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Xiao Zheng, Heqi Nie, Yuxiang Huang, Heyan Mao, Xian-Ling School of Computer Science and Technology Beijing Institute of Technology China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications China Southeast Academy of Information Technology Beijing Institute of Technology China

Scientific machine reading comprehension (SMRC) aims to understand scientific texts through interactions with humans by given questions. As far as we know, there is only one dataset focused on exploring full-text scientific machine reading comprehension. However, the dataset has ignored the fact that different readers may have different levels of understanding of the text, and only includes single-perspective question-answer pairs, leading to a lack of consideration of different perspectives. To tackle the above problem, we propose a novel multi-perspective SMRC dataset, called SciMRC, which includes perspectives from beginners, students and experts. Our proposed SciMRC is constructed from 741 scientific papers and 6,057 question-answer pairs. Each perspective of beginners, students and experts contains 3,306, 1,800 and 951 QA pairs, respectively. The extensive experiments on SciMRC by utilizing pre-trained models suggest the importance of considering perspectives of SMRC, and demonstrate its challenging nature for machine comprehension. © 2023, CC BY.

关键词：