检索结果-内蒙古大学图书馆

DP-Rewrite: Towards Reproducibility and Transparency in Differentially Private Text Rewriting 29

学校读者我要写书评

暂无评论

DP-Rewrite: Towards Reproducibility and Transparency in Diff...

29th International Conference on Computational Linguistics, COLING 2022

作者： Igamberdiev, Timour Arnold, Thomas Habernal, Ivan Trustworthy Human Language Technologies Ubiquitous Knowledge Processing Lab Department of Computer Science Technical University of Darmstadt Germany

Text rewriting with differential privacy (DP) provides concrete theoretical guarantees for protecting the privacy of individuals in textual documents. In practice, existing systems may lack the means to validate their privacy-preserving claims, leading to problems of transparency and reproducibility. We introduce DP-Rewrite, an open-source framework for differentially private text rewriting which aims to solve these problems by being modular, extensible, and highly customizable. Our system incorporates a variety of downstream datasets, models, pre-training procedures, and evaluation metrics to provide a flexible way to lead and validate private text rewriting research. To demonstrate our software in practice, we provide a set of experiments as a case study on the ADePT DP text rewriting system, detecting a privacy leak in its pre-training approach. Our system is publicly available, and we hope that it will help the community to make DP text rewriting research more accessible and transparent. © 2022 Proceedings - International Conference on Computational Linguistics, COLING. All rights reserved.

关键词： Transparency

The Inherent Limits of Pretrained LLMs: The Unexpected Convergence of Instruction Tuning and In-Context Learning Capabilities

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Bigoulaeva, Irina Madabushi, Harish Tayyar Gurevych, Iryna Ubiquitous Knowledge Processing Lab Technical University of Darmstadt Germany Department of Computer Science The University of Bath United Kingdom

Large Language Models (LLMs), trained on extensive web-scale corpora, have demonstrated remarkable abilities across diverse tasks, especially as they are scaled up. Nevertheless, even state-of-the-art models struggle in certain cases, sometimes failing at problems solvable by young children, indicating that traditional notions of task complexity are insufficient for explaining LLM capabilities. However, exploring LLM capabilities is complicated by the fact that most widely-used models are also 'instruction-tuned' to respond appropriately to prompts. With the goal of disentangling the factors influencing LLM performance, we investigate whether instruction-tuned models possess fundamentally different capabilities from base models that are prompted using in-context examples. Through extensive experiments across various model families, scales and task types, which included instruction tuning 90 different LLMs, we demonstrate that the performance of instruction-tuned models is significantly correlated with the in-context performance of their base counterparts. By clarifying what instruction-tuning contributes, we extend prior research into in-context learning, which suggests that base models use priors from pretraining data to solve tasks. Specifically, we extend this understanding to instruction-tuned models, suggesting that their pretraining data similarly sets a limiting boundary on the tasks they can solve, with the added influence of the instruction-tuning dataset. © 2025, CC BY.

关键词： Contrastive Learning

Constrained C-Test Generation via Mixed-Integer Programming

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Lee, Ji-Ung Pfetsch, Marc E. Gurevych, Iryna Ubiquitous Knowledge Processing Lab Department of Computer Science Germany Research Group Optimization Department of Mathematics Technical University of Darmstadt Germany Hessian AI

This work proposes a novel method to generate C-Tests;a deviated form of cloze tests (a gap filling exercise) where only the last part of a word is turned into a gap. In contrast to previous works that only consider varying the gap size or gap placement to achieve locally optimal solutions, we propose a mixed-integer programming (MIP) approach. This allows us to consider gap size and placement simultaneously, achieving globally optimal solutions, and to directly integrate state-of-the-art models for gap difficulty prediction into the optimization problem. A user study with 40 participants across four C-Test generation strategies (including GPT-4) shows that our approach (MIP) significantly outperforms two of the baseline strategies (based on gap placement and GPT-4);and performs on-par with the third (based on gap size). Our analysis shows that GPT-4 still struggles to fulfill explicit constraints during generation and that MIP produces C-Tests that correlate best with the perceived difficulty. We publish our code, model, and collected data consisting of 32 English C-Tests with 20 gaps each (totaling 3,200 individual gap responses) under an open source license.1 © 2024, CC BY.

关键词： Integer programming

Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model

学校读者我要写书评

暂无评论

Controllable Factuality in Document-Grounded Dialog Systems ...

2022 Findings of the Association for Computational Linguistics: EMNLP 2022

作者： Daheim, Nico Thulke, David Dugast, Christian Ney, Hermann Ubiquitous Knowledge Processing Lab Department of Computer Science Technical University of Darmstadt Germany Human Language Technology and Pattern Recognition RWTH Aachen University Germany AppTek GmbH

In this work, we present a model for document-grounded response generation in dialog that is decomposed into two components according to Bayes' theorem. One component is a traditional ungrounded response generation model and the other component models the reconstruction of the grounding document based on the dialog context and generated response. We propose different approximate decoding schemes and evaluate our approach on multiple open-domain and task-oriented document-grounded dialog datasets. Our experiments show that the model is more factual in terms of automatic factuality metrics than the baseline model. Furthermore, we outline how introducing scaling factors between the components allows for controlling the tradeoff between factuality and fluency in the model output. Finally, we compare our approach to a recently proposed method to control factuality in grounded dialog, CTRL (Rashkin et al., 2021), and show that both approaches can be combined to achieve additional improvements. © 2022 Association for Computational Linguistics.

关键词： Computational linguistics

Are Emergent Abilities in Large Language Models just In-Context Learning?

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Lu, Sheng Bigoulaeva, Irina Sachdeva, Rachneet Madabushi, Harish Tayyar Gurevych, Iryna Ubiquitous Knowledge Processing Lab Technical University of Darmstadt Germany Department of Computer Science The University of Bath United Kingdom

Large language models, comprising billions of parameters and pre-trained on extensive web-scale corpora, have been claimed to acquire certain capabilities without having been specifically trained on them. These capabilities, referred to as "emergent abilities," have been a driving force in discussions regarding the potentials and risks of language models. A key challenge in evaluating emergent abilities is that they are confounded by model competencies that arise through alternative prompting techniques, including in-context learning, which is the ability of models to complete a task based on a few examples. We present a novel theory that explains emergent abilities, taking into account their potential confounding factors, and rigorously substantiate this theory through over 1000 experiments. Our findings suggest that purported emergent abilities are not truly emergent, but result from a combination of in-context learning, model memory, and linguistic knowledge. Our work is a foundational step in explaining language model performance, providing a template for their efficient use and clarifying the paradox of their ability to excel in some instances while faltering in others. Thus, we demonstrate that their capabilities should not be overestimated. 1 © 2023, CC BY-SA.

关键词： Contrastive Learning

ArgumenText: Argument Classification and Clustering in a Generalized Search Scenario

学校读者我要写书评

暂无评论

Datenbank-Spektrum 2020年第2期20卷 115-121页

作者： Daxenberger, Johannes Schiller, Benjamin Stahlhut, Chris Kaiser, Erik Gurevych, Iryna Ubiquitous Knowledge Processing Lab Department of Computer Science Technische Universität Darmstadt Darmstadt Germany

The ArgumenText project creates argument mining technology for big and heterogeneous data and aims to evaluate its use in real-world applications. The technology mines and clusters arguments from a variety of textual sources for a large range of topics and in multiple languages. Its main strength is its generalization to very different textual sources including web crawls, news data, or customer reviews. We validated the technology with a focus on supporting decisions in innovation management as well as customer feedback analysis. Along with its public argument search engine and API, ArgumenText has released multiple datasets for argument classification and clustering. This contribution outlines the major technology-related challenges and proposed solutions for the tasks of argument extraction from heterogeneous sources and argument clustering. It also lays out exemplary industry applications and remaining challenges. © 2020, The Author(s).

关键词： Search engines

DP-Rewrite: Towards Reproducibility and Transparency in Differentially Private Text Rewriting

学校读者我要写书评

暂无评论

arXiv 2022年

关键词： Transparency

UKP-SQUARE: An Online Platform for Question Answering Research

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Baumgärtner, Tim Wang, Kexin Sachdeva, Rachneet Eichler, Max Geigle, Gregor Poth, Clifton Sterz, Hannah Puerto, Haritz Ribeiro, Leonardo F.R. Pfeiffer, Jonas Reimers, Nils Şahin, Gözde Gül Gurevych, Iryna Ubiquitous Knowledge Processing Lab Department of Computer Science Technical University of Darmstadt Germany Hugging Face Koç University Computer Science and Engineering Department Turkey

Recent advances in NLP and information retrieval have given rise to a diverse set of question answering tasks that are of different formats (e.g., extractive, abstractive), require different model architectures (e.g., generative, discriminative), and setups (e.g., with or without retrieval). Despite having a large number of powerful, specialized QA pipelines (which we refer to as Skills) that consider a single domain, model or setup, there exists no framework where users can easily explore and compare such pipelines and can extend them according to their needs. To address this issue, we present UKP-SQUARE, an extensible online QA platform for researchers which allows users to query and analyze a large collection of modern Skills via a user-friendly web interface and integrated behavioural tests. In addition, QA researchers can develop, manage, and share their custom Skills using our microservices that support a wide range of models (Transformers, Adapters, ONNX), data-stores and retrieval techniques (e.g., sparse and dense). UKP-SQUARE is available on https://***.1 Copyright © 2022, The Authors. All rights reserved.

关键词： Pipelines

Ranking Creative Language Characteristics in Small Data Scenarios 13

学校读者我要写书评

暂无评论

Ranking Creative Language Characteristics in Small Data Scen...

13th International Conference on Computational Creativity, ICCC 2022

作者： Siekiera, Julia Köppel, Marius Simpson, Edwin Stowe, Kevin Gurevych, Iryna Kramer, Stefan Dept. of Computer Science Johannes Gutenberg-Universität Mainz Germany Institute for Nuclear Physics Johannes Gutenberg-Universität Mainz Germany Dept. of Computer Science University of Bristol United Kingdom Ubiquitous Knowledge Processing Lab. Technische Universität Darmstadt Germany

ISBN: (纸本)9789895416042

The ability to rank creative natural language provides an important general tool for downstream language understanding and generation. However, current deep ranking models require substantial amounts of labeled data that are difficult and expensive to obtain for new domains, languages and creative characteristics. A recent neural approach, DirectRanker, reduces the amount of training data needed but has not previously been used to rank creative text. We therefore adapt DirectRanker to provide a new deep model for ranking creative language with small numbers of training instances, and compare it with a Bayesian approach, Gaussian process preference learning (GPPL), which was previously shown to work well with sparse data. Our experiments with short creative language texts show the effectiveness of DirectRanker even with small training datasets. Combining DirectRanker with GPPL outperforms the previous state of the art on humor and metaphor novelty tasks, increasing Spearman’s ρ by 25% and 29% on average. Furthermore, we provide a possible application to validate jokes in the process of creativity generation. © 2022 Proceedings of the 13th International Conference on Computational Creativity, ICCC 2022. All rights reserved.

关键词： Bayesian networks