检索结果-内蒙古大学图书馆

On Dataset Transferability in Active Learning for Transformers 61

学校读者我要写书评

暂无评论

On Dataset Transferability in Active Learning for Transforme...

Findings of the Association for Computational Linguistics, ACL 2023

作者： Jelenić, Fran Jukić, Josip Drobac, Nina Šnajder, Jan University of Zagreb Faculty of Electrical Engineering and Computing Text Analysis and Knowledge Engineering Lab Croatia

ISBN: (纸本)9781959429623

Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning. While the effectiveness of AL for fine-tuning transformer-based pre-trained language models (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others. We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM. We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used. Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model. © 2023 Association for Computational Linguistics.

关键词： Classification (of information)

ALANNO: An Active Learning Annotation System for Mortals 17

学校读者我要写书评

暂无评论

ALANNO: An Active Learning Annotation System for Mortals

17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023

作者： Jukić, Josip Jelenić, Fran Bićanić, Miroslav Šnajder, Jan University of Zagreb Faculty of Electrical Engineering and Computing Text Analysis and Knowledge Engineering Lab Croatia

ISBN: (纸本)9781959429456

Supervised machine learning has become the cornerstone of today’s data-driven society, increasing the need for labeled data. However, the process of acquiring labels is often expensive and tedious. One possible remedy is to use active learning (AL) – a special family of machine learning algorithms designed to reduce labeling costs. Although AL has been successful in practice, a number of practical challenges hinder its effectiveness and are often overlooked in existing AL annotation tools. To address these challenges, we developed ALANNO, an open-source annotation system for NLP tasks equipped with features to make AL effective in real-world annotation projects. ALANNO facilitates annotation management in a multi-annotator setup and supports a variety of AL methods and underlying models, which are easily configurable and extensible. © 2023 Association for Computational Linguistics.

关键词： Learning algorithms

Claim Check-Worthiness Detection: How Well do LLMs Grasp Annotation Guidelines?

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Majer, Laura Šnajder, Jan Text Analysis and Knowledge Engineering Lab University of Zagreb Faculty of Electrical Engineering and Computing Croatia

The rising threat of disinformation underscores the need to fully or partially automate the factchecking process. Identifying text segments requiring fact-checking is known as claim detection (CD) and claim check-worthiness detection (CW), the latter incorporating complex domain-specific criteria of worthiness and often framed as a ranking task. Zero- and few-shot LLM prompting is an attractive option for both tasks, as it bypasses the need for labeled datasets and allows verbalized claim and worthiness criteria to be directly used for prompting. We evaluate the LLMs’ predictive accuracy on five CD/CW datasets from diverse domains, using corresponding annotation guidelines in prompts. We examine two key aspects: (1) how to best distill factuality and worthiness criteria into a prompt, and (2) how much context to provide for each claim. To this end, we experiment with different levels of prompt verbosity and varying amounts of contextual information given to the model. We additionally evaluate the top-performing models with ranking metrics, resembling prioritization done by fact-checkers. Our results show that optimal prompt verbosity varies, meta-data alone adds more performance boost than co-text, and confidence scores can be directly used to produce reliable check-worthiness rankings. © 2024, CC BY.

关键词： labeled data

You Are What You Talk About: Inducing Evaluative Topics for Personality Analysis

学校读者我要写书评

暂无评论

You Are What You Talk About: Inducing Evaluative Topics for ...

2022 Findings of the Association for Computational Linguistics: EMNLP 2022

作者： Jukić, Josip Vukojević, Iva Šnajder, Jan University of Zagreb Faculty of Electrical Engineering and Computing Text Analysis and Knowledge Engineering Lab Unska 3 Zagreb10000 Croatia

Expressing attitude or stance toward entities and concepts is an integral part of human behavior and personality. Recently, evaluative language data has become more accessible with social media's rapid growth, enabling large-scale opinion analysis. However, surprisingly little research examines the relationship between personality and evaluative language. To bridge this gap, we introduce the notion of evaluative topics, obtained by applying topic models to pre-filtered evaluative text from social media. We then link evaluative topics to individual text authors to build their evaluative profiles. We apply evaluative profiling to Reddit comments labeled with personality scores and conduct an exploratory study on the relationship between evaluative topics and Big Five personality facets, aiming for a more interpretable, facet-level analysis. Finally, we validate our approach by observing correlations consistent with prior research in personality psychology. © 2022 Association for Computational Linguistics.

关键词： Behavioral research

Large-scale Evaluation of Transformer-based Article Encoders on the Task of Citation Recommendation 3

学校读者我要写书评

暂无评论

Large-scale Evaluation of Transformer-based Article Encoders...

3rd Workshop on Scholarly Document Processing, SDP 2022 at 29th International Conference on Computational Linguistics, COLING 2022

作者： Medić, Zoran Šnajder, Jan Text Analysis and Knowledge Engineering Lab Faculty of Electrical Engineering and Computing University of Zagreb Unska 3 Zagreb10000 Croatia

Recently introduced transformer-based article encoders (TAEs) designed to produce similar vector representations for mutually related scientific articles have demonstrated strong performance on benchmark datasets for scientific article recommendation. However, the existing benchmark datasets are predominantly focused on single domains and, in some cases, contain easy negatives in small candidate pools. Evaluating representations on such benchmarks might obscure the realistic performance of TAEs in setups with thousands of articles in candidate pools. In this work, we evaluate TAEs on large benchmarks with more challenging candidate pools. We compare the performance of TAEs with a lexical retrieval baseline model BM25 on the task of citation recommendation, where the model produces a list of recommendations for citing in a given input article. We find out that BM25 is still very competitive with the state-of-the-art neural retrievers, a finding which is surprising given the strong performance of TAEs on small benchmarks. As a remedy for the limitations of the existing benchmarks, we propose a new benchmark dataset for evaluating scientific article representations: Multi-Domain Citation Recommendation dataset (MDCR), which covers different scientific fields and contains challenging candidate pools. © 2022 COLING. All Rights Reserved.

关键词： Benchmarking

NLPOP: a Dataset for Popularity Prediction of Promoted NLP Research on Twitter 12

学校读者我要写书评

暂无评论

NLPOP: a Dataset for Popularity Prediction of Promoted NLP R...

12th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA 2022

作者： Obadić, Leo Tutek, Martin Šnajder, Jan Text Analysis and Knowledge Engineering Lab. Faculty of Electrical Engineering and Computing University of Zagreb Unska 3 Zagreb10000 Croatia

ISBN: (纸本)9781955917520

Twitter has slowly but surely established itself as a forum for disseminating, analysing and promoting NLP research. The trend of researchers promoting work not yet peer-reviewed (preprints) by posting concise summaries presented itself as an opportunity to collect and combine multiple modalities of data. In scope of this paper, we (1) construct a dataset of Twitter threads in which researchers promote NLP preprints and (2) evaluate whether it is possible to predict the popularity of a thread based on the content of the Twitter thread, paper content and user metadata. We experimentally show that it is possible to predict popularity of threads promoting research based on their content, and that predictive performance depends on modelling textual input, indicating that the dataset could present value for related areas of NLP research such as citation recommendation and abstractive summarization. © 2022 Association for Computational Linguistics.

关键词： Natural language processing systems

Paragraph-level Citation Recommendation based on Topic Sentences as Queries

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Medić, Zoran Šnajder, Jan University of Zagreb Faculty of Electrical Engineering and Computing Text Analysis and Knowledge Engineering Lab Croatia

Citation recommendation (CR) models may help authors find relevant articles at various stages of the paper writing process. Most research has dealt with either global CR, which produces general recommendations suitable for the initial writing stage, or local CR, which produces specific recommendations more fitting for the final writing stages. We propose the task of paragraph-level CR as a middle ground between the two approaches, where the paragraph’s topic sentence is taken as input and recommendations for citing within the paragraph are produced at the output. We propose a model for this task, fine-tune it using the quadruplet loss on the dataset of ACL papers, and show improvements over the baselines. © 2023, CC BY.

关键词： Computational linguistics

On Dataset Transferability in Active Learning for Transformers

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Jelenić, Fran Jukić, Josip Drobac, Nina Šnajder, Jan University of Zagreb Faculty of Electrical Engineering and Computing Croatia Text Analysis and Knowledge Engineering Lab Croatia

关键词： text processing

You Are What You Talk About: Inducing Evaluative Topics for Personality Analysis

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Jukić, Josip Vukojević, Iva Šnajder, Jan University of Zagreb Faculty of Electrical Engineering and Computing Text Analysis and Knowledge Engineering Lab Unska 3 Zagreb10000 Croatia

关键词： Social networking (online)