检索结果-内蒙古大学图书馆

LISTENING TO THE WISE FEW: SELECT-AND-COPY ATTENTION HEADS FOR MULTIPLE-CHOICE QA

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Tulchinskii, Eduard Kushnareva, Laida Kuznetsov, Kristian Voznyuk, Anastasia Andriiainen, Andrei Piontkovskaya, Irina Burnaev, Evgeny Barannikov, Serguei Skolkovo Institute of Science and Technology Russia AI Foundation and Algorithm Lab Moscow Institute of Physics and Technology Russia CNRS Université Paris Cité France

A standard way to evaluate the abilities of LLM involves presenting a multiple-choice question and selecting the option with the highest logit as the model’s predicted answer. However, such a format for evaluating LLMs has limitations, since even if the model knows the correct answer, it may struggle to select the corresponding letter simply due to difficulties in following this rigid format. To address this, we introduce new scores that better capture and reveal model’s underlying knowledge: the Query-Key Score (QK-score), derived from the interaction between query and key representations in attention heads, and the Attention Score, based on attention weights. These scores are extracted from specific select- and-copy heads, which show consistent performance across popular Multi-Choice Question Answering (MCQA) datasets. Based on these scores, our method improves knowledge extraction, yielding up to 16% gain for LLaMA2-7B and up to 10% for larger models on popular MCQA benchmarks. At the same time, the accuracy on a simple synthetic dataset, where the model explicitly knows the right answer, increases by almost 60%, achieving nearly perfect accuracy, therefore demonstrating the method’s efficiency in mitigating MCQA format limitations. To support our claims, we conduct experiments on models ranging from 7 billion to 70 billion parameters in both zero- and few-shot setups. © 2024, CC BY.

关键词： Question answering

Scalar Function Topology Divergence: Comparing Topology of 3D Objects

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Trofimov, Ilya Voronkova, Daria Tulchinskii, Eduard Burnaev, Evgeny Barannikov, Serguei Skolkovo Institute of Science and Technology Moscow Russia AIRI Moscow Russia CNRS IMJ Paris Cité University France AI Foundation and Algorithm Lab Moscow Russia

We propose a new topological tool for computer vision - Scalar Function Topology Divergence (SFTD), which measures the dissimilarity of multi-scale topology between sublevel sets of two functions having a common domain. Functions can be defined on an undirected graph or Euclidean space of any dimensionality. Most of the existing methods for comparing topology are based on Wasserstein distance between persistence barcodes and they don't take into account the localization of topological features. The minimization of SFTD ensures that the corresponding topological features of scalar functions are located in the same places. The proposed tool provides useful visualizations depicting areas where functions have topological dissimilarities. We provide applications of the proposed method to 3D computer vision. In particular, experiments demonstrate that SFTD as an additional loss improves the reconstruction of cellular 3D shapes from 2D fluorescence microscopy images, and helps to identify topological errors in 3D segmentation. Additionally, we show that SFTD outperforms Betti matching loss in 2D segmentation problems. The code is publicly available: https://***/IlyaTrofimov/SFTD. © 2024, CC BY.

关键词： Fluorescence microscopy

Intrinsic Dimension Estimation for Robust Detection of ai-Generated Texts

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Tulchinskii, Eduard Kuznetsov, Kristian Kushnareva, Laida Cherniavskii, Daniil Nikolenko, Sergey Burnaev, Evgeny Barannikov, Serguei Piontkovskaya, Irina Skolkovo Institute of Science and Technology Russia AI Foundation and Algorithm Lab Russia Russia CNRS Université Paris Cité France St. Petersburg Department the Steklov Institute of Mathematics Russia

Rapidly increasing quality of ai-generated content makes it difficult to distinguish between human and ai-generated texts, which may lead to undesirable consequences for society. Therefore, it becomes increasingly important to study the properties of human texts that are invariant over different text domains and varying proficiency of human writers, can be easily calculated for any language, and can robustly separate natural and ai-generated texts regardless of the generation model and sampling method. In this work, we propose such an invariant for human-written texts, namely the intrinsic dimensionality of the manifold underlying the set of embeddings for a given text sample. We show that the average intrinsic dimensionality of fluent texts in a natural language is hovering around the value 9 for several alphabet-based languages and around 7 for Chinese, while the average intrinsic dimensionality of ai-generated texts for each language is ≈ 1.5 lower, with a clear statistical separation between human-generated and ai-generated distributions. This property allows us to build a score-based artificial text detector. The proposed detector’s accuracy is stable over text domains, generator models, and human writer proficiency levels, outperforming SOTA detectors in model-agnostic and cross-domain scenarios by a significant margin. We release code and dataMSC Codes 68T50 © 2023, CC BY.

关键词：

Robust ai-Generated Text Detection by Restricted Embeddings

学校读者我要写书评

暂无评论

arXiv 2024年

关键词： Embeddings

ai-generated text boundary detection with RoFT

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Kushnareva, Laida Gaintseva, Tatiana Magai, German Barannikov, Serguei Abulkhanov, Dmitry Kuznetsov, Kristian Tulchinskii, Eduard Piontkovskaya, Irina Nikolenko, Sergey AI Foundation and Algorithm Lab Russia Digital Environment Research Institute Queen Mary University of London United Kingdom HSE University Russia Noeon Research Japan Skolkovo Institute of Science and Technology Russia CNRS Université Paris Cité France St. Petersburg Department The Steklov Institute of Mathematics Russia

Due to the rapid development of large language models, people increasingly often encounter texts that may start as written by a human but continue as machine-generated. Detecting the boundary between human-written and machine-generated parts of such texts is a challenging problem that has not received much attention in literature. We attempt to bridge this gap and examine several ways to adapt state of the art artificial text detection classifiers to the boundary detection setting. We push all detectors to their limits, using the Real or Fake text benchmark that contains short texts on several topics and includes generations of various language models. We use this diversity to deeply examine the robustness of all detectors in cross-domain and cross-model settings to provide baselines and insights for future research. In particular, we find that perplexity-based approaches to boundary detection tend to be more robust to peculiarities of domain-specific data than supervised fine-tuning of the RoBERTa model;we also find which features of the text confuse boundary detection algorithms and negatively influence their performance in cross-domain settings. © 2023, CC BY.

关键词：

Intrinsic dimension estimation for robust detection of ai-generated texts 23

学校读者我要写书评

暂无评论

Intrinsic dimension estimation for robust detection of AI-ge...

Proceedings of the 37th International Conference on Neural Information Processing Systems

作者： Eduard Tulchinskii Kristian Kuznetsov Laida Kushnareva Daniil Cherniavskii Sergey Nikolenko Evgeny Burnaev Serguei Barannikov Irina Piontkovskaya Skolkovo Institute of Science and Technology Russia AI Foundation and Algorithm Lab Russia Artificial Intelligence Research Institute (AIRI) Russia St. Petersburg Department of the Steklov Institute of Mathematics Russia Skolkovo Institute of Science and Technology Russia and Artificial Intelligence Research Institute (AIRI) Russia Skolkovo Institute of Science and Technology Russia and CNRS Université Paris Cité France

Rapidly increasing quality of ai-generated content makes it difficult to distinguish between human and ai-generated texts, which may lead to undesirable consequences for society. Therefore, it becomes increasingly important to study the properties of human texts that are invariant over different text domains and varying proficiency of human writers, can be easily calculated for any language, and can robustly separate natural and ai-generated texts regardless of the generation model and sampling method. In this work, we propose such an invariant for human-written texts, namely the intrinsic dimensionality of the manifold underlying the set of embeddings for a given text sample. We show that the average intrinsic dimensionality of fluent texts in a natural language is hovering around the value 9 for several alphabet-based languages and around 7 for Chinese, while the average intrinsic dimensionality of ai-generated texts for each language is ≈ 1.5 lower, with a clear statistical separation between human-generated and ai-generated distributions. This property allows us to build a score-based artificial text detector. The proposed detector's accuracy is stable over text domains, generator models, and human writer proficiency levels, outperforming SOTA detectors in model-agnostic and cross-domain scenarios by a significant margin. We release code and data ***/ArGintum/GPTID.

关键词：

Improving Interpretability and Robustness for the Detection of ai-Generated Images

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gaintseva, Tatiana Kushnareva, Laida Magai, German Piontkovskaya, Irina Nikolenko, Sergey Benning, Marting Barannikov, Serguei Slabaugh, Gregory Digital Environment Research Institute Queen Mary University of London United Kingdom AI Foundation and Algorithm Lab Russia HSE University Russia Noeon Research Japan Skolkovo Institute of Science and Technology Russia CNRS Universite Paris Cite France St. Petersburg Department Steklov Institute of Mathematics Russia Department of Computer Science University College London United Kingdom

With growing abilities of generative models, artificial content detection becomes an increasingly important and difficult task. However, all popular approaches to this prcoblem suffer from poor generalization across domains and generative models. In this work, we focus on the robustness of ai-generated image (aiGI) detectors. We analyze existing state-of-the-art aiGI detection methods based on frozen CLIP embeddings and show how to interpret them, shedding light on how images produced by various ai generators differ from real ones. Next we propose two ways to improve robustness: based on removing harmful components of the embedding vector and based on selecting the best performing attention heads in the image encoder model. Our methods increase the mean out-of-distribution (OOD) classification score by up to 6% for cross-model transfer. We also propose a new dataset for aiGI detection and use it in our evaluation;we believe this dataset will help boost further research. The dataset and code are provided as a supplement. © 2024, CC BY.

关键词： Embeddings