检索结果-内蒙古大学图书馆

1st workshop on Mathematical natural language processing, MathNLP 2022

作者： Bueno, Mirelle Gemmell, Carlos Dalton, Jeffrey Lotufo, Roberto Nogueira, Rodrigo University of Campinas Brazil University of Glasgow United Kingdom University of Campinas NeuralMind Brazil

ISBN: (纸本)9781959429142

The ability to extrapolate, i.e., to make predictions on sequences that are longer than those presented as training examples, is a challenging problem for current deep learning models. Recent work shows that this limitation persists in state-of-the-art Transformer-based models. Most solutions to this problem use specific architectures or training methods that do not generalize to other tasks. We demonstrate that large language models can succeed in extrapolation without modifying their architecture or training procedure. Our experimental results show that generating step-by-step rationales and introducing marker tokens are both required for effective extrapolation. first, we induce a language model to produce step-by-step rationales before outputting the answer to effectively communicate the task to the model. However, as sequences become longer, we find that current models struggle to keep track of token positions. To address this issue, we interleave output tokens with markup tokens that act as explicit positional and counting symbols. Our findings show how these two complementary approaches enable remarkable sequence extrapolation and highlight a limitation of current architectures to effectively generalize without explicit surface form guidance. Code available at https://***/MirelleB/induced-rationales-markup-tokens. © 2022 Association for Computational Linguistics.

关键词： Extrapolation

来源：评论

学校读者我要写书评

暂无评论

Retrieval-augmented Generation across Heterogeneous Knowledge

Retrieval-augmented Generation across Heterogeneous Knowledg...

引用

Conference of the North-American-Chapter-of-the-Association-for-Computational-Linguistics (NAAACL) - Human language Technologies

作者： Yu, Wenhao Univ Notre Dame Notre Dame IN 46556 USA

ISBN: (纸本)9781955917735

Retrieval-augmented generation (RAG) methods have been receiving increasing attention from the NLP community and achieved state-of-the-art performance on many NLP downstream tasks. Compared with conventional pretrained generation models, RAG methods have remarkable advantages such as easy knowledge acquisition, strong scalability, and low training cost. Although existing RAG models have been applied to various knowledge-intensive NLP tasks, such as open-domain QA and dialogue systems, most of the work has focused on retrieving unstructured text documents from Wikipedia. In this paper, I first elaborate on the current obstacles to retrieving knowledge from a single-source homogeneous corpus. Then, I demonstrate evidence from both existing literature and my experiments, and provide multiple solutions on retrieval-augmented generation methods across heterogeneous knowledge.

关键词： natural language processing systems

来源：评论

学校读者我要写书评

暂无评论

Extend, don't rebuild: Phrasing conditional graph modification as autoregressive sequence labelling

Extend, don't rebuild: Phrasing conditional graph modificati...

引用

Conference on Empirical methods in natural language processing (EMNLP)

作者： Weber, Leon Garda, Samuele Muenchmeyer, Jannes Leser, Ulf Humboldt Univ Berlin Germany Max Delbriick Ctr Mol Med Berlin Germany GFZ German Res Ctr Geosci Potsdam Potsdam Germany

ISBN: (纸本)9781955917094

Deriving and modifying graphs from natural language text has become a versatile basis technology for information extraction with applications in many subfields, such as semantic parsing or knowledge graph construction. A recent work used this technique for modifying scene graphs (He et al., 2020), by first encoding the original graph and then generating the modified one based on this encoding. In this work, we show that we can considerably increase performance on this problem by phrasing it as graph extension instead of graph generation. We propose the first model for the resulting graph extension problem based on autoregressive sequence labelling. On three scene graph modification data sets, this formulation leads to improvements in accuracy over the state-of-the-art between 13 and 26 percentage points. Furthermore, we introduce a novel data set from the biomedical domain which has much larger linguistic variability and more complex graphs than the scene graph modification data sets. For this data set, the state-of-the art fails to generalize, while our model can produce meaningful predictions.

关键词： Knowledge graph

来源：评论

学校读者我要写书评

暂无评论

LLM in a flash: Efficient Large language Model Inference with Limited Memory 62

LLM in a flash: Efficient Large Language Model Inference wit...

引用

62nd Annual Meeting of the Association-for-Computational-Linguistics (ACL) / Student Research workshop (SRW)

作者： Alizadeh, Keivan Mirzadeh, Iman Belenko, Dmitry Khatamifard, S. Karen Cho, Minsik Del Mundo, Carlo C. Rastegari, Mohammad Farajtabar, Mehrdad Apple Cupertino CA 95014 USA

ISBN: (纸本)9798891760943

Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their substantial computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing them on demand to DRAM. Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. first, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with up to 4x and 20x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.

关键词： Flash memory

来源：评论

学校读者我要写书评

暂无评论

Named Entities based on the BERT-BILSTM-ACRF Model Recognition Research 23

Named Entities Based on the BERT-BILSTM-ACRF Model Recogniti...

引用

proceedings of the 2023 7th International Conference on natural language processing and Information Retrieval

作者： Jingdong Wang Yongjia Guo School of Computer Science Northeast Electric Power University China

ISBN: (纸本)9798400709227

As a crucial first step in the process of constructing knowledge graph, the accuracy of named entity recognition determines the construction effect of the final graph. However, at present, Chinese named entity recognition methods still have many problems, such as long training time and lower accuracy. Hence, we come up with a BERT-BILSTM-ACRF entity recognition method that combines the “self-attention” mechanism. To begin with, Bert model is selected as the embedding layer, the text is vectorized, and the character position in-formation is obtained through the bidirectional Long Short-Term Memory network. Secondly, the internal relationship of the character sequence is further searched through the self-attention mechanism, and finally the final optimal sequence is decoded by the conditional random field model. To check the effectiveness of the BERT-BILSTM-ACRF model, the model is applied to the data set of the university course textbook” Da-ta Structure”, and the result reaches 98.97%F1 value and 98.14%accuracy, which has good experimental results and certain practical value.

关键词： Knowledge graph Named entity recognition natural language processing Self-attention mechanism

来源：评论

学校读者我要写书评

暂无评论

Efficient Pre-training of Masked language Model via Concept-based Curriculum Masking

Efficient Pre-training of Masked Language Model via Concept-...

引用

2022 Conference on Empirical methods in natural language processing, EMNLP 2022

作者： Lee, Mingyu Park, Jun-Hyung Kim, Junho Kim, Kang-Min Lee, SangKeun Department of Artificial Intelligence Korea University Seoul Korea Republic of Department of Computer Science and Engineering Korea University Seoul Korea Republic of Department of Data Science The Catholic University of Korea Bucheon Korea Republic of

Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations, but incurs substantial training costs. In this paper, we propose a novel concept-based curriculum masking (CCM) method to efficiently pre-train a language model. CCM has two key differences from existing curriculum learning approaches to effectively reflect the nature of MLM. first, we introduce a carefully-designed linguistic difficulty criterion that evaluates the MLM difficulty of each token. Second, we construct a curriculum that gradually masks words related to the previously masked words by retrieving a knowledge graph. Experimental results show that CCM significantly improves pre-training efficiency. Specifically, the model trained with CCM shows comparative performance with the original BERT on the General language Understanding Evaluation benchmark at half of the training cost. Code is available at https://***/KoreaMGLEE/Concept-based-curriculum-masking. © 2022 Association for Computational Linguistics.

关键词： Curricula

来源：评论

学校读者我要写书评

暂无评论

On the Use of Predictive Deep Learning Approaches in the Frequency and Uniqueness-based Representation of Sequential Browsing Events 24th

On the Use of Predictive Deep Learning Approaches in the Fre...

引用

24th International Conference on Computational Science and Its Applications (ICCSA)

作者： Hakvar, Hakan Cavuldak, Cansu Soyler, Ogulcan Subasi, Yusuf Karadayi, Yildiz Safak, Ilgin Tasgetiren, Nail Aktas, Mehmet S. Fibabanka R&D Ctr Istanbul Turkiye Hepsiburada R&D Ctr Istanbul Turkiye Yildiz Tech Univ Comp Engn Dept Istanbul Turkiye

ISBN: (纸本)9783031646072;9783031646089

To gain a more comprehensive understanding of how users navigate, this study investigates the potential of predictive deep learning techniques. To properly describe user's browsing events such page visits, this paper first provides an overview of two embedding methodologies based on natural language processing and graphical representations. Secondly, this study presents two aggregation methods, one based on frequency and the other on uniqueness, that can be used to depict the possible sequence of events that may occur during a user's surfing session. With an emphasis on online banking, this research summarizes data representation approaches to model customers' digital footprints and learn about their banking product purchase intents. When we analyze a customer's digital footprint in the digital banking domain during the early stages of the application's lifecycle, we find that there are insufficient digital user footprints to model and comprehend users' behavior. Consequently, new approaches are needed to increase the quantity of session recordings kept by customers' digital footprints. We consider the possibility of increasing the quantity of digital footprints of bank customers by employing predictive deep learning techniques. To enhance the quantity of user navigational behaviors, we present a predictive technique based on long short-term memories. According to the findings, the newly generated customer sequences facilitate testing the suggested method, and accurately represent the initial client session data.

关键词： Customer Embedding Customer Behavior Digital Banking graph based Embedding Representation Learning Word Embedding

来源：评论

学校读者我要写书评

暂无评论

Coarse-to-Fine Entity Representations for Document-Level Relation Extraction 12th

Coarse-to-Fine Entity Representations for Document-Level Re...

引用

12th National CCF Conference on natural language processing and Chinese Computing, NLPCC 2023

作者： Dai, Damai Ren, Jing Zeng, Shuang Chang, Baobao Sui, Zhifang MOE Key Lab of Computational Linguistics Peking University Beijing China

ISBN: (纸本)9783031446955

Document-level Relation Extraction (RE) requires extracting relations expressed within and across sentences. Recent works show that graph-based methods, usually constructing a document-level graph that captures document-aware interactions, can obtain useful entity representations thus helping tackle document-level RE. These methods either focus more on the entire graph, or pay more attention to a part of the graph, e.g., paths between the target entity pair. However, we find that document-level RE may benefit from focusing on both of them simultaneously. Therefore, to obtain more comprehensive entity representations, we propose the Coarse-to-Fine Entity Representation model (CFER) that adopts a coarse-to-fine strategy involving two phases. first, CFER uses graph neural networks to integrate global information in the entire graph at a coarse level. Next, CFER utilizes the global information as a guidance to selectively aggregate path information between the target entity pair at a fine level. In classification, we combine the entity representations from both two levels into more comprehensive representations for relation extraction. Experimental results on two document-level RE datasets, DocRED and CDR, show that CFER outperforms baseline models and is robust to the uneven label distribution. Our code is available at https://***/Hunter-DDM/cfer-document-level-RE. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.

关键词： graph neural networks

来源：评论

学校读者我要写书评

暂无评论

Improving Unsupervised Extractive Summarization by Jointly Modeling Facet and Redundancy

引用

IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND language processing 2022年 30卷 1546-1557页

作者： Liang, Xinnian Li, Jing Wu, Shuangzhi Li, Mu Li, Zhoujun Beihang Univ Sch Comp Sci & Engn SCSE State Key Lab Software Dev Environm Beijing 100089 Peoples R China Renmin Univ China Sch Informat Beijing 100089 Peoples R China Dept Tencent Cloud Xiaowei Beijing 100089 Peoples R China

Unsupervised extractive summarization aims to extract salient sentences from documents without labeled corpus. Existing methods are mostly graph-based by computing sentence centrality. These methods have two main problems: facet bias and redundant problems. Facet bias problem leads summarization models tend to select sentences within the same facet, which often leads to the ignoring of other vital facets, especially on long-document and multi-documents. first, to address the facet bias problem, we proposed a novel Facet-Aware centrality-based Ranking model (FAR). We let the model pay more attention to different facets by introducing a sentence-document weight. The weight is added to the sentence centrality score. FAR can alleviate redundancy to some extent. Then, to further reduce redundancy, we proposed a novel Redundancy- and Facet-Aware Ranking model (RFAR) which jointly models facet and redundancy by incorporating Determinantal Point Process (DPP) into the previous proposed FAR. We evaluate our FAR and RFAR on a wide range of summarization tasks that include 8 representative benchmark datasets. Experimental results show that FAR and RFAR consistently outperforms strong baselines, especially in long- and multi-document scenarios, and even perform comparably to some supervised models. Besides, we find that our methods can alleviate the position bias problem.

关键词： Redundancy Mathematical models Computational modeling Speech processing Task analysis Visualization Training data Determinantal point process facet-bias problem redundancy unsupervised extractive summarization

来源：评论

学校读者我要写书评

暂无评论

ReproHum #0927-3: Reproducing The Human Evaluation Of The DExperts Controlled Text Generation Method 4

ReproHum #0927-3: Reproducing The Human Evaluation Of The DE...

引用

4th workshop on Human Evaluation of NLP Systems, HumEval 2024

作者： González-Corbelle, Javier Vivel-Couso, A. Alonso-Moral, J.M. Bugarín-Diz, A. Universidade de Santiago de Compostela Spain

ISBN: (纸本)9782493814418

This paper presents a reproduction study aimed at reproducing and validating a human NLP evaluation performed for the DExperts text generation method. The original study introduces DExperts, a controlled text generation method, evaluated using non-toxic prompts from the RealToxicityPrompts dataset. Our reproduction study aims to reproduce the human evaluation of the continuations generated by DExperts in comparison with four baseline methods, in terms of toxicity, topicality, and fluency. We first describe the agreed approach for reproduction within the ReproHum project and detail the configuration of the original evaluation, including necessary adaptations for reproduction. Then, we make a comparison of our reproduction results with those reported in the reproduced paper. Interestingly, we observe how the human evaluators in our experiment appreciate higher quality in the texts generated by DExperts in terms of less toxicity and better fluency. All in all, new scores are higher, also for the baseline methods. This study contributes to ongoing efforts in ensuring the reproducibility and reliability of findings in NLP evaluation and emphasizes the critical role of robust methodologies in advancing the field. © 2024 European language Resources Association (ELRA).

关键词： natural language processing systems

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：