In this work, we show that contemporary language models have a previously unknown skill - the capacity for electronic circuit design from high-level textual descriptions, akin to code generation. We introduce two benc...
详细信息
ISBN:
(纸本)9798891760615
In this work, we show that contemporary language models have a previously unknown skill - the capacity for electronic circuit design from high-level textual descriptions, akin to code generation. We introduce two benchmarks: PINS100, assessing model knowledge of electrical components, and MICRO25, evaluating a model's capability to design common micro-controller circuits and code in the ARDUINO ecosystem that involve input, output, sensors, motors, protocols, and logic - with models such as GPT-4 and Claude-V1 achieving between 60% to 96% PASS@ 1 on generating full devices. We include six case studies of using language models as a design assistant for moderately complex devices, such as a radiation-powered random number generator, an emoji keyboard, a visible spectrometer, and several assistive devices, while offering a qualitative analysis performance, outlining evaluation challenges, and suggesting areas of development to improve complex circuit design and practical utility. With this work, we aim to spur research at the juncture of naturallanguageprocessing and electronic design.
The use of propagandistic techniques in online content has increased in recent years aiming to manipulate online audiences. Fine-grained propaganda detection and extraction of textual spans where propaganda techniques...
详细信息
With the prevalence of the Internet and various types of social media, our daily life is surrounded by a huge amount of text information, which can provide us with the convenience of accessing information and communic...
详细信息
ISBN:
(纸本)9798400709760
With the prevalence of the Internet and various types of social media, our daily life is surrounded by a huge amount of text information, which can provide us with the convenience of accessing information and communication, but also brings challenges in management and maintenance. In the face of the current practical needs, traditional machine learning methods, such as TF-IDF, SVM, Naive Bayes need to rely on manual completion of the feature design, and can not solve the problem of data sparsity and high-dimensional feature vectors, so that the accuracy of text classification is insufficient. In this regard, based on the deep learning algorithm, this paper will propose a set of Chinese text classification model based on Word2vec and TextCNN to realize the automatic classification function of a large number of texts. Practice has proved that the test precision of this classification model is 94.12%, the recall is 92.19%, and the F1 value is 93.11%. Compared with the traditional TF-IDF, SVM and Naive Bayes methods, all the evaluation indexes are obviously improved, and the actual application effect is better than that of the original TextCNN. The overall design is in line with expectations and has certain application and promotion value.
Recent language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters. Yet, LMs often generate the factually incorrect responses to the given queries, since ...
详细信息
ISBN:
(纸本)9798891760608
Recent language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters. Yet, LMs often generate the factually incorrect responses to the given queries, since their knowledge may be inaccurate, incomplete, and outdated. To address this problem, previous works propose to augment LMs with the knowledge retrieved from an external knowledge source. However, such approaches often show suboptimal text generation performance due to two reasons: 1) the model may fail to retrieve the knowledge relevant to the given query, or 2) the model may not faithfully reflect the retrieved knowledge in the generated text. To overcome these, we propose to verify the output and the knowledge of the knowledge-augmented LMs with a separate verifier, which is a small LM that is trained to detect those two types of errors through instruction-finetuning. Then, when the verifier recognizes an error, we can rectify it by either retrieving new knowledge or generating new text. Further, we use an ensemble of the outputs from different instructions with a single verifier to enhance the reliability of the verification processes. We validate the effectiveness of the proposed verification steps on multiple question answering benchmarks, whose results show that the proposed verifier effectively identifies retrieval and generation errors, allowing LMs to provide more factually correct outputs. Our code is available at https://***/JinheonBaek/KALMV.
In naturallanguageprocessing (NLP) tasks, detecting out-of-distribution (OOD) samples is essential to safely deploy a language model in real-world problems. Recently, several studies report that pre-trained language...
详细信息
ISBN:
(纸本)9798350307627
In naturallanguageprocessing (NLP) tasks, detecting out-of-distribution (OOD) samples is essential to safely deploy a language model in real-world problems. Recently, several studies report that pre-trained language models (PLMs) accurately detect OOD data compared to LSTM, but we empirically find that PLMs show sub-par OOD detection performance when (1) OOD samples have similar semantic representation to in-distribution (IND) samples and (2) PLMs are finetuned under data scarcity settings. To alleviate above issues, state-of-the-art uncertainty quantification (UQ) methods can be used, but the comprehensive analysis of UQ methods with PLMs has received little consideration. In this work, we investigate seven UQ methods with PLMs and show their effectiveness in the text classification task.
As a promising paradigm to collaboratively train models with decentralized data, Federated Learning (FL) can be exploited to fine-tune Large language Models (LLMs). While LLMs correspond to huge size, the scale of the...
详细信息
Most multilingual vision-and-language (V&L) research aims to accomplish multilingual and multimodal capabilities within one model. However, the scarcity of multilingual captions for images has hindered the develop...
详细信息
ISBN:
(纸本)9798891760615
Most multilingual vision-and-language (V&L) research aims to accomplish multilingual and multimodal capabilities within one model. However, the scarcity of multilingual captions for images has hindered the development. To overcome this obstacle, we propose ICU, Image Caption Understanding, which divides a V&L task into two stages: a V&L model performs image captioning in English, and a multilingual language model (mLM), in turn, takes the caption as the alt text and performs cross-lingual language understanding. The burden of multilingual processing is lifted off V&L model and placed on mLM. Since the multilingual text data is relatively of higher abundance and quality, ICU can facilitate the conquering of language barriers for V&L models. In experiments on two tasks across 9 languages in the IGLUE benchmark, we show that ICU can achieve new state-of-the-art results for five languages, and comparable results for the rest.
Classification is a core NLP task architecture with many potential applications. While large language models (LLMs) have brought substantial advancements in text generation, their potential for enhancing classificatio...
详细信息
In interactions between users and language model agents, user utterances frequently exhibit ellipsis (omission of words or phrases) or imprecision (lack of exactness) to prioritize efficiency. This can lead to varying...
详细信息
ISBN:
(纸本)9798891761643
In interactions between users and language model agents, user utterances frequently exhibit ellipsis (omission of words or phrases) or imprecision (lack of exactness) to prioritize efficiency. This can lead to varying interpretations of the same input based on different assumptions or background knowledge. It is thus crucial for agents to adeptly handle the inherent ambiguity in queries to ensure reliability. However, even state-of-the-art large language models (LLMs) still face challenges in such scenarios, primarily due to the following hurdles: (1) LLMs are not explicitly trained to deal with ambiguous utterances;(2) the degree of ambiguity perceived by the LLMs may vary depending on the possessed knowledge. To address these issues, we propose Alignment with Perceived Ambiguity (APA), a novel pipeline that aligns LLMs to manage ambiguous queries by leveraging their own assessment of ambiguity (i.e., perceived ambiguity). Experimental results on question-answering datasets demonstrate that APA empowers LLMs to explicitly detect and manage ambiguous queries while retaining the ability to answer clear questions. Furthermore, our finding proves that APA excels beyond training with gold-standard labels, especially in out-of-distribution scenarios. The data and code are available at https://***/heyjoonkim/APA.
This work focuses on the task of query-based meeting summarization, in which the summary of a context (meeting transcript) is generated in response to a specific query. When using Large language Models (LLMs) for this...
详细信息
暂无评论