检索结果-内蒙古大学图书馆

PRESERVING PRE-TRAINED FEATURES HELPS CALIBRATE FINE-TUNED LANGUAGE MODELS 11

学校读者我要写书评

暂无评论

PRESERVING PRE-TRAINED FEATURES HELPS CALIBRATE FINE-TUNED L...

11th International Conference on Learning Representations, ICLR 2023

作者： He, Guande Chen, Jianfei Zhu, Jun Dept.of Comp.Sci. & Tech. Institute for AI Tsinghua-Bosch Joint Center for ML BNRist Center State Key Lab for Intell.Tech.& Sys. Tsinghua University Beijing China Pazhou Lab Guangzhou510330 China

Large pre-trained language models (PLMs) have demonstrated strong performance on natural language understanding (NLU) tasks through ***, fine-tuned models still suffer from overconfident predictions, especially in out-of-domain *** this paper, we tackle the problem of calibrating fine-tuned language *** demonstrate that the PLMs are well-calibrated on the masked language modeling task with robust predictive confidence under domain shift, yet the fine-tuned models fail to retain such property due to catastrophic forgetting, which impacts the calibration on the downstream classification *** light of these observations, we evaluate the calibration of several methods that preserve pre-trained features and show that preserving pre-trained features can improve the calibration of fine-tuned language *** these methods, our proposed method that encourages the fine-tuned model to learn generative representations with auxiliary language modeling objective achieves competitive accuracy and the lowest expected calibration error compared to several strong baselines under both in-domain and out-of-domain settings on three downstream NLU tasks. © 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.

关键词： Calibration

TAILORING LANGUAGE GENERATION MODELS UNDER TOTAL VARIATION DISTANCE

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ji, Haozhe Ke, Pei Hu, Zhipeng Zhang, Rongsheng Huang, Minlie Dept. of Comp. Sci. & Tech. State Key Lab of Intelligent Tech. & Sys China BNRist Center Tsinghua University Beijing100084 China Fuxi AI Lab NetEase Inc. China

The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. From a distributional view, MLE in fact minimizes the Kullback-Leibler divergence (KLD) between the distribution of the real data and that of the model. However, this approach forces the model to distribute non-zero (sometimes large) probability mass to all training samples regardless of their quality. Moreover, in the attempt to cover the low-probability regions in the data distribution, the model systematically overestimates the probability of corrupted text sequences, which we conjecture is one of the main reasons for text degeneration during autoregressive decoding. To remedy this problem, we leverage the total variation distance (TVD) with its robustness to outliers, and develop practical bounds to apply it to language generation. Then, we introduce the TaiLr1 objective that balances the tradeoff of estimating TVD. Intuitively, TaiLr downweights real data samples that have low model probabilities with tunable penalization intensity. Experimental results show that our method alleviates the overestimation of degenerated sequences without sacrificing diversity and improves generation quality on a wide range of text generation tasks.2 © 2023, CC0.

关键词： Maximum likelihood estimation

EQUIVARIANT ENERGY-GUIDED SDE FOR INVERSE MOLECULAR DESIGN 11

学校读者我要写书评

暂无评论

EQUIVARIANT ENERGY-GUIDED SDE FOR INVERSE MOLECULAR DESIGN

11th International Conference on Learning Representations, ICLR 2023

作者： Bao, Fan Zhao, Min Hao, Zhongkai Li, Peiyao Li, Chongxuan Zhu, Jun Dept. of Comp. Sci. & Tech. Institute for AI Tsinghua-Huawei Joint Center for AI BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China Gaoling School of Artificial Intelligence Renmin University of China Beijing China Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing China

Inverse molecular design is critical in material science and drug discovery, where the generated molecules should satisfy certain desirable properties. In this paper, we propose equivariant energy-guided stochastic differential equations (EEGSDE), a flexible framework for controllable 3D molecule generation under the guidance of an energy function in diffusion models. Formally, we show that EEGSDE naturally exploits the geometric symmetry in 3D molecular conformation, as long as the energy function is invariant to orthogonal transformations. Empirically, under the guidance of designed energy functions, EEGSDE significantly improves the baseline on QM9, in inverse molecular design targeted to quantum properties and molecular structures. Furthermore, EEGSDE is able to generate molecules with multiple target properties by combining the corresponding energy functions linearly. © 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.

关键词： Molecules

Dense Contrastive Loss for Instance Segmentation 33

学校读者我要写书评

暂无评论

Dense Contrastive Loss for Instance Segmentation

33rd British Machine Vision Conference Proceedings, BMVC 2022

作者： Chen, Hang Tang, Chufeng Hu, Xiaolin Depart. of Comp. Sci. & Tech. State Key Lab of Intell. Tech. & Sys. THU-Bosch JCML Center BNRist Institute for AI THBI IDG/McGovern Institute for Brain Research Tsinghua University Beijing China Beijing China

Instance segmentation, which requires instance-level mask prediction, is a fundamental task in computer vision. Many methods have been proposed in this field. However, the existing methods still do not perform well in complex scenarios such as occlusion. In this work, we analyzed the segmentation errors of some typical instance segmentation models. We found that false negatives (i.e. misclassification of foreground pixels as background) accounted for the majority of errors. It can be attributed to the inconsistent features of the same instance under complex scenarios. To address this problem, we proposed a dense contrastive loss to encourage the segmentation network to learn more consistent feature representations. Specifically, features on the same instance are pulled closer, while features on different instances and features between instances and the background are pushed farther apart. Without introducing any extra inference cost, the proposed method mitigated false-negative errors and achieved significant improvements on the Cityscapes and MS-COCO datasets. Code will be available at https://***/tinyalpha/DCL. © 2022. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.

关键词： Computer vision

DAB-DETR: DYNAMIC ANCHOR BOXES ARE BETTER QUERIES FOR DETR 10

学校读者我要写书评

暂无评论

DAB-DETR: DYNAMIC ANCHOR BOXES ARE BETTER QUERIES FOR DETR

10th International Conference on Learning Representations, ICLR 2022

作者： Liu, Shilong Li, Feng Zhang, Hao Yang, Xiao Qi, Xianbiao Su, Hang Zhu, Jun Zhang, Lei Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. & Sys. Institute for AI Tsinghua-Bosch Joint Center for ML Tsinghua University China China Hong Kong University of Science and Technology Hong Kong Peng Cheng Laboratory Guangdong Shenzhen China

We present in this paper a novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer a deeper understanding of the role of queries in DETR. This new formulation directly uses box coordinates as queries in Transformer decoders and dynamically updates them layer by layer. Using box coordinates not only helps using explicit positional priors to improve the query-to-feature similarity and eliminate the slow training convergence issue in DETR, but also allows us to modulate the positional attention map using the box width and height information. Such a design makes it clear that queries in DETR can be implemented as performing soft ROI pooling layer by layer in a cascade manner. As a result, it leads to the best performance on MS-COCO benchmark among the DETR-like detection models under the same setting, e.g., AP 45.7% using ResNet50-DC5 as backbone trained in 50 epochs. We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our methods. Code is available at https://***/IDEA-opensource/DAB-DETR. © 2022 ICLR 2022 - 10th International Conference on Learning Representationss. All rights reserved.

关键词：

ANALYTIC-DPM: AN ANALYTIC ESTIMATE OF THE OPTIMAL REVERSE VARIANCE IN DIFFUSION PROBABILISTIC MODELS 10

学校读者我要写书评

暂无评论

ANALYTIC-DPM: AN ANALYTIC ESTIMATE OF THE OPTIMAL REVERSE VA...

10th International Conference on Learning Representations, ICLR 2022

作者： Bao, Fan Li, Chongxuan Zhu, Jun Zhang, Bo Dept. of Comp. Sci. & Tech. Institute for AI Tsinghua-Huawei Joint Center for AI BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China Gaoling School of Artificial Intelligence Renmin University of China Beijing China Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing China

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20× to 80× speed up. © 2022 ICLR 2022 - 10th International Conference on Learning Representationss. All rights reserved.

关键词： Monte Carlo methods

PRESERVING PRE-TRAINED FEATURES HELPS CALIBRATE FINE-TUNED LANGUAGE MODELS

学校读者我要写书评

暂无评论

arXiv 2023年

作者： He, Guande Chen, Jianfei Zhu, Jun Dept. of Comp. Sci. & Tech. Institute for AI Tsinghua-Bosch Joint Center for ML BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China Pazhou Lab Guangzhou510330 China

Large pre-trained language models (PLMs) have demonstrated strong performance on natural language understanding (NLU) tasks through fine-tuning. However, fine-tuned models still suffer from overconfident predictions, especially in out-of-domain settings. In this paper, we tackle the problem of calibrating fine-tuned language models. We demonstrate that the PLMs are well-calibrated on the masked language modeling task with robust predictive confidence under domain shift, yet the fine-tuned models fail to retain such property due to catastrophic forgetting, which impacts the calibration on the downstream classification task. In light of these observations, we evaluate the calibration of several methods that preserve pre-trained features and show that preserving pre-trained features can improve the calibration of fine-tuned language models. Among these methods, our proposed method that encourages the fine-tuned model to learn generative representations with auxiliary language modeling objective achieves competitive accuracy and the lowest expected calibration error compared to several strong baselines under both in-domain and out-of-domain settings on three downstream NLU tasks. Copyright © 2023, The Authors. All rights reserved.

关键词： Calibration

INVESTIGATING UNCERTAINTY CALIBRATION OF ALIGNED LANGUAGE MODELS UNDER THE MULTIPLE-CHOICE SETTING

学校读者我要写书评

暂无评论

arXiv 2023年

作者： He, Guande Cui, Peng Chen, Jianfei Hu, Wenbo Zhu, Jun Dept. of Comp. Sci. & Tech. Institute for AI Tsinghua-Bosch Joint Center for ML BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China Hefei University of Technology China

Despite the significant progress made in practical applications of aligned language models (LMs), they tend to be overconfident in output answers compared to the corresponding pre-trained LMs. In this work, we systematically evaluate the impact of the alignment process on logit-based uncertainty calibration of LMs under the multiple-choice setting. We first conduct a thoughtful empirical study on how aligned LMs differ in calibration from their pre-trained counterparts. Experimental results reveal that there are two distinct uncertainties in LMs under the multiple-choice setting, which are responsible for the answer decision and the format preference of the LMs, respectively. Then, we investigate the role of these two uncertainties on aligned LM’s calibration through fine-tuning in simple synthetic alignment schemes and conclude that one reason for aligned LMs’ overconfidence is the conflation of these two types of uncertainty. Furthermore, we examine the utility of common post-hoc calibration methods for aligned LMs and propose an easy-to-implement and sample-efficient method to calibrate aligned LMs. We hope our findings could provide insights into the design of more reliable alignment processes for LMs. © 2023, CC BY.

关键词： Calibration

Visual Recognition by Request

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Tang, Chufeng Xie, Lingxi Zhang, Xiaopeng Hu, Xiaolin Tian, Qi Dept. of Comp. Sci. & Tech. State Key Lab for Intell. Tech. & Sys. Institute for AI BNRist Tsinghua University China Huawei Inc. China

Humans have the ability of recognizing visual semantics in an unlimited granularity, but existing visual recognition algorithms cannot achieve this goal. In this paper, we establish a new paradigm named visual recognition by request (ViRReq1) to bridge the gap. The key lies in decomposing visual recognition into atomic tasks named requests and leveraging a knowledge base, a hierarchical and text-based dictionary, to assist task definition. ViRReq allows for (i) learning complicated whole-part hierarchies from highly incomplete annotations and (ii) inserting new concepts with minimal efforts. We also establish a solid baseline by integrating language-driven recognition into recent semantic and instance segmentation methods, and demonstrate its flexible recognition ability on CPP and ADE20K, two datasets with hierarchical whole-part annotations. Copyright © 2022, The Authors. All rights reserved.

关键词： Semantics