检索结果-内蒙古大学图书馆

Quantifying Logical Consistency in Transformers via Query-Key Alignment

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Tulchinskii, Eduard Voznyuk, Anastasia Kushnareva, Laida Andriiainen, Andrei Piontkovskaya, Irina Burnaev, Evgeny Barannikov, Serguei Skolkovo Institute of Science and Technology Russia AI Foundation Algorithm Lab Moscow Institute of Physics and Technology Russia CNRS Université Paris Cité France

Large language models (LLMs) have demonstrated impressive performance in various natural language processing tasks, yet their ability to perform multi-step logical reasoning remains an open challenge. Although Chain-of-Thought prompting has improved logical reasoning by enabling models to generate intermediate steps, it lacks mechanisms to assess the coherence of these logical transitions. In this paper, we propose a novel, lightweight evaluation strategy for logical reasoning that uses query–key alignments inside transformer attention heads. By computing a single forward pass and extracting a "QK-score" from carefully chosen heads, our method reveals latent representations that reliably separate valid from invalid inferences, offering a scalable alternative to traditional ablation-based techniques. We also provide an empirical validation on multiple logical reasoning benchmarks, demonstrating improved robustness of our evaluation method against distractors and increased reasoning depth. The experiments were conducted on a diverse set of models, ranging from 1.5B to 70B parameters. © 2025, CC BY.

关键词： Structured Query Language

Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Kuznetsov, Kristian Kushnareva, Laida Druzhinina, Polina Razzhigaev, Anton Voznyuk, Anastasia Piontkovskaya, Irina Burnaev, Evgeny Barannikov, Serguei Skolkovo Institute of Science and Technology Russia AI Foundation and Algorithm Lab United States Moscow Institute of Physics and Technology Russia CNRS Université Paris Cité France Russia

Artificial Text Detection (ATD) is becoming increasingly important with the rise of advanced Large Language Models (LLMs). Despite numerous efforts, no single algorithm performs consistently well across different types of unseen text or guarantees effective generalization to new LLMs. Interpretability plays a crucial role in achieving this goal. In this study, we enhance ATD interpretability by using Sparse Autoencoders (SAE) to extract features from Gemma-2-2b’s residual stream. We identify both interpretable and efficient features, analyzing their semantics and relevance through domain- and model-specific statistics, a steering approach, and manual or LLM-based interpretation. Our methods offer valuable insights into how texts from various models differ from human-written content. We show that modern LLMs have a distinct writing style, especially in information-dense domains, even though they can produce human-like outputs with personalized prompts. © 2025, CC BY.

关键词： Semantics

One-Step Residual Shifting Diffusion for Image Super-Resolution via Distillation

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Selikhanovych, Daniil Li, David Leonov, Aleksei Gushchin, Nikita Kushneriuk, Sergei Filippov, Alexander Burnaev, Evgeny Koshelev, Iaroslav Korotin, Alexander HSE University Russia Yandex Research AI Foundation Algorithm Lab Skolkovo Institute of Science and Technology Russia Moscow Institute of Physics and Technology Russia Artificial Intelligence Research Institute

Diffusion models for super-resolution (SR) produce high-quality visual results but require expensive computational costs. Despite the development of several methods to accelerate diffusion-based SR models, some (e.g., SinSR) fail to produce realistic perceptual details, while others (e.g., OSEDiff) may hallucinate non-existent structures. To overcome these issues, we present RSD, a new distillation method for ResShift, one of the top diffusion-based SR models. Our method is based on training the student network to produce such images that a new fake ResShift model trained on them will coincide with the teacher model. RSD achieves single-step restoration and outperforms the teacher by a large margin. We show that our distillation method can surpass the other distillation-based method for ResShift - SinSR - making it on par with state-of-the-art diffusion-based SR distillation methods. Compared to SR methods based on pre-trained text-to-image models, RSD produces competitive perceptual quality, provides images with better alignment to degraded input images, and requires fewer parameters and GPU memory. We provide experimental results on various real-world and synthetic datasets, including RealSR, RealSet65, DRealSR, ImageNet, and DIV2K. Copyright © 2025, The Authors. All rights reserved.

关键词：

A Modular Conditional Diffusion Framework for Image Reconstruction 38

学校读者我要写书评

暂无评论

A Modular Conditional Diffusion Framework for Image Reconstr...

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Zhussip, Magauiya Koshelev, Iaroslav Lefkimmiatis, Stamatis MTS AI United States AI Foundation and Algorithm Lab United States

Diffusion Probabilistic Models (DPMs) have been recently utilized to deal with various blind image restoration (IR) tasks, where they have demonstrated outstanding performance in terms of perceptual quality. However, the task-specific nature of existing solutions and the excessive computational costs related to their training, make such models impractical and challenging to use for different IR tasks than those that were initially trained for. This hinders their wider adoption, especially by those who lack access to powerful computational resources and vast amount of training data. In this work we aim to address the above issues and enable the successful adoption of DPMs in practical IR-related applications. Towards this goal, we propose a modular diffusion probabilistic IR framework (DP-IR), which allows us to combine the performance benefits of existing pre-trained state-of-the-art IR networks and generative DPMs, while it requires only the additional training of a relatively small module (0.7M params) related to the particular IR task of interest. Moreover, the architecture of the proposed framework allows for a sampling strategy that leads to at least four times reduction of neural function evaluations without suffering any performance loss, while it can also be combined with existing acceleration techniques such as DDIM. We evaluate our model on four benchmarks for the tasks of burst JDD-SR, dynamic scene deblurring, and superresolution. Our method outperforms existing approaches in terms of perceptual quality while it retains a competitive performance with respect to fidelity metrics. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

A Modular Conditional Diffusion Framework for Image Reconstruction

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhussip, Magauiya Koshelev, Iaroslav Lefkimmiatis, Stamatis MTS AI AI Foundation and Algorithm Lab

Diffusion Probabilistic Models (DPMs) have been recently utilized to deal with various blind image restoration (IR) tasks, where they have demonstrated outstanding performance in terms of perceptual quality. However, the task-specific nature of existing solutions and the excessive computational costs related to their training, make such models impractical and challenging to use for different IR tasks than those that were initially trained for. This hinders their wider adoption, especially by those who lack access to powerful computational resources and vast amount of training data. In this work we aim to address the above issues and enable the successful adoption of DPMs in practical IR-related applications. Towards this goal, we propose a modular diffusion probabilistic IR framework (DP-IR), which allows us to combine the performance benefits of existing pre-trained state-of-the-art IR networks and generative DPMs, while it requires only the additional training of a relatively small module (0.7M params) related to the particular IR task of interest. Moreover, the architecture of the proposed framework allows for a sampling strategy that leads to at least four times reduction of neural function evaluations without suffering any performance loss, while it can also be combined with existing acceleration techniques such as DDIM. We evaluate our model on four benchmarks for the tasks of burst JDD-SR, dynamic scene deblurring, and superresolution. Our method outperforms existing approaches in terms of perceptual quality while it retains a competitive performance with respect to fidelity metrics. Copyright © 2024, The Authors. All rights reserved.

关键词： Benchmarking

Robust Two-View Geometry Estimation with Implicit Differentiation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Pyatov, Vladislav Koshelev, Iaroslav Lefkimmiatis, Stamatios Center for AI Technology Russia AI Foundation and Algorithm Lab Russia MTS AI Group Russia

We present a novel two-view geometry estimation framework which is based on a differentiable robust loss function fitting. We propose to treat the robust fundamental matrix estimation as an implicit layer, which allows us to avoid backpropagation through time and significantly improves the numerical stability. To take full advantage of the information from the feature matching stage we incorporate learnable weights that depend on the matching confidences. In this way our solution brings together feature extraction, matching and two-view geometry estimation in a unified end-to-end trainable pipeline. We evaluate our approach on the camera pose estimation task in both outdoor and indoor scenarios. The experiments on several datasets show that the proposed method outperforms both classic and learning-based state-of-the-art methods by a large margin. The project webpage is available at: https://***/VladPyatov/ihls © 2024, CC BY-NC-ND.

关键词： Geometry

A3D: DOES DIFFUSION DREAM ABOUT 3D ALIGNMENT?

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ignatyev, Savva Konovalova, Nina Selikhanovych, Daniil Voynov, Oleg Patakin, Nikolay Olkov, Ilya Senushkin, Dmitry Artemov, Alexey Konushin, Anton Filippov, Alexander Wonka, Peter Burnaev, Evgeny Skoltech Russia AIRI Russia Medida AI Israel AI Foundation and Algorithm Lab Russia KAUST Saudi Arabia

We tackle the problem of text-driven 3D generation from a geometry alignment perspective. Given a set of text prompts, we aim to generate a collection of objects with semantically corresponding parts aligned across them. Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models to high-quality representations of the 3D objects. These methods handle multiple text queries separately, and therefore the resulting objects have a high variability in object pose and structure. However, in some applications, such as 3D asset design, it may be desirable to obtain a set of objects aligned with each other. In order to achieve the alignment of the corresponding parts of the generated objects, we propose to embed these objects into a common latent space and optimize the continuous transitions between these objects. We enforce two kinds of properties of these transitions: smoothness of the transition and plausibility of the intermediate objects along the transition. We demonstrate that both of these properties are essential for good alignment. We provide several practical scenarios that benefit from alignment between the objects, including 3D editing and object hybridization, and experimentally demonstrate the effectiveness of our method. ***/a3d © 2024, CC BY-NC-SA.

关键词： Semantics

Robust Two-View Geometry Estimation with Implicit Differentiation

学校读者我要写书评

暂无评论

Robust Two-View Geometry Estimation with Implicit Differenti...

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

作者： Vladislav Pyatov Iaroslav Koshelev Stamatios Lefkimmiatis Skolkovo Institute of Science and Technology (Skoltech) Center for AI Technology Russia AI Foundation and Algorithm Lab Russia MTS AI Group Russia

ISBN: (数字)9798350377705

ISBN: (纸本)9798350377712

We present a novel two-view geometry estimation framework which is based on a differentiable robust loss function fitting. We propose to treat the robust fundamental matrix estimation as an implicit layer, which allows us to avoid backpropagation through time and significantly improves the numerical stability. To take full advantage of the information from the feature matching stage we incorporate learnable weights that depend on the matching confidences. In this way our solution brings together feature extraction, matching and two-view geometry estimation in a unified end-to-end trainable pipeline. We evaluate our approach on the camera pose estimation task in both outdoor and indoor scenarios. The experiments on several datasets show that the proposed method outperforms both classic and learning-based state-of- the-art methods by a large margin. The project webpage is available at: https://***/VladPyatov/ihls

关键词： Geometry Backpropagation Pose estimation Robot vision systems Pipelines Fitting Cameras Feature extraction Numerical stability Intelligent robots

NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ignatyev, Savva Selikhanovych, Daniil Voynov, Oleg Wang, Yiqun Wonka, Peter Lefkimmiatis, Stamatios Burnaev, Evgeny Skoltech Russia AIRI Russia Chongqing University China KAUST Saudi Arabia AI Foundation and Algorithm Lab Russia

We present a novel method for 3D surface reconstruction from multiple images where only a part of the object of interest is captured. Our approach builds on two recent developments: surface reconstruction using neural radiance fields for the reconstruction of the visible parts of the surface, and guidance of pre-trained 2D diffusion models in the form of Score Distillation Sampling (SDS) to complete the shape in unobserved regions in a plausible manner. We introduce three components. First, we suggest employing normal maps as a pure geometric representation for SDS instead of color renderings which are entangled with the appearance information. Second, we introduce the freezing of the SDS noise during training which results in more coherent gradients and better convergence. Third, we propose Multi-View SDS as a way to condition the generation of the non-observable part of the surface without fine-tuning or making changes to the underlying 2D Stable Diffusion model. We evaluate our approach on the BlendedMVS dataset demonstrating significant qualitative and quantitative improvements over competing methods. Copyright © 2023, The Authors. All rights reserved.

关键词： Distillation