检索结果-内蒙古大学图书馆

Stability and generalization of bilevel programming in hyperparameter optimization 21

学校读者我要写书评

暂无评论

Stability and generalization of bilevel programming in hyper...

Proceedings of the 35th International Conference on Neural Information Processing systems

作者： Fan Bao Guoqiang Wu Chongxuan Li Jun Zhu Bo Zhang Dept. of Comp. Sci. & Tech. Institute for AI Tsinghua-Huawei Joint Center for AI BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China

ISBN: (纸本)9781713845393

The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w.r.t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the classical cross-validation algorithm. Our results suggest that gradient-based algorithms can be better than cross-validation under certain conditions in a theoretical perspective. Furthermore, we prove that regularization terms in both the outer and inner levels can relieve the overfitting problem in gradient-based algorithms. In experiments on feature learning and data reweighting for noisy labels, we corroborate our theoretical findings.

关键词：

Rethinking and reweighting the univariate losses for multi-label ranking: consistency and generalization 21

学校读者我要写书评

暂无评论

Rethinking and reweighting the univariate losses for multi-l...

Proceedings of the 35th International Conference on Neural Information Processing systems

作者： Guoqiang Wu Chongxuan Li Kun Xu Jun Zhu Dept. of Comp. Sci. & Tech. Institute for AI State Key Lab for Intell. Tech. & Sys. Tsinghua-Bosch Joint Center for ML BNRist Center Tsinghua University Beijing China

ISBN: (纸本)9781713845393

The (partial) ranking loss is a commonly used evaluation measure for multi-label classification, which is usually optimized with convex surrogates for computational efficiency. Prior theoretical efforts on multi-label ranking mainly focus on (Fisher) consistency analyses. However, there is a gap between existing theory and practice — some inconsistent pairwise losses can lead to promising performance, while some consistent univariate losses usually have no clear superiority in practice. To take a step towards filling up this gap, this paper presents a systematic study from two complementary perspectives of consistency and generalization error bounds of learning algorithms. We theoretically find two key factors of the distribution (or dataset) that affect the learning guarantees of algorithms: the instance-wise class imbalance and the label size c. Specifically, in an extremely imbalanced case, the algorithm with the consistent univariate loss has an error bound of O(c), while the one with the inconsistent pairwise loss depends on O(√c) as shown in prior work. This may shed light on the superior performance of pairwise methods in practice, where real datasets are usually highly imbalanced. Moreover, we present an inconsistent reweighted univariate loss-based algorithm that enjoys an error bound of O(√c) for promising performance as well as the computational efficiency of univariate losses. Finally, experimental results confirm our theoretical findings.

关键词：

ANALYTIC-DPM: AN ANALYTIC ESTIMATE OF THE OPTIMAL REVERSE VARIANCE IN DIFFUSION PROBABILISTIC MODELS

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Bao, Fan Li, Chongxuan Zhu, Jun Zhang, Bo Dept. of Comp. Sci. & Tech. Institute for AI Tsinghua-Huawei Joint Center for AI BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China Gaoling School of Artificial Intelligence Renmin University of China Beijing China Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing China Huawei Noah's Ark Lab China

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20× to 80× speed up. Copyright © 2022, The Authors. All rights reserved.

关键词： Monte Carlo methods

On the convergence of prior-guided zeroth-order optimization algorithms 21

学校读者我要写书评

暂无评论

On the convergence of prior-guided zeroth-order optimization...

Proceedings of the 35th International Conference on Neural Information Processing systems

作者： Shuyu Cheng Guoqiang Wu Jun Zhu Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. & Sys. Institute for AI Tsinghua-Bosch Joint Center for ML Tsinghua University Beijing China and Pazhou Lab Guangzhou China

ISBN: (纸本)9781713845393

Zeroth-order (ZO) optimization is widely used to handle challenging tasks, such as query-based black-box adversarial attacks and reinforcement learning. Various attempts have been made to integrate prior information into the gradient estimation procedure based on finite differences, with promising empirical results. However, their convergence properties are not well understood. This paper makes an attempt to fill up this gap by analyzing the convergence of prior-guided ZO algorithms under a greedy descent framework with various gradient estimators. We provide a convergence guarantee for the prior-guided random gradient-free (PRGF) algorithms. Moreover, to further accelerate over greedy descent methods, we present a new accelerated random search (ARS) algorithm that incorporates prior information, together with a convergence analysis. Finally, our theoretical results are confirmed by experiments on several numerical benchmarks as well as adversarial attacks.

关键词：

Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Shao, Huiyang Xu, Qianqian Yang, Zhiyong Bao, Shilong Huang, Qingming Key Lab of Intell. Info. Process. Inst. of Comput. Tech. CAS China School of Computer Science and Tech. University of Chinese Academy of Sciences China State Key Lab of Info. Security Inst. of Info. Engineering CAS China School of Cyber Security University of Chinese Academy of Sciences China BDKM University of Chinese Academy of Sciences China Peng Cheng Laboratory China

The Partial Area Under the ROC Curve (PAUC), typically including One-way Partial AUC (OPAUC) and Two-way Partial AUC (TPAUC), measures the average performance of a binary classifier within a specific false positive rate and/or true positive rate interval, which is a widely adopted measure when decision constraints must be considered. Consequently, PAUC optimization has naturally attracted increasing attention in the machine learning community within the last few years. Nonetheless, most of the existing methods could only optimize PAUC approximately, leading to inevitable biases that are not controllable. Fortunately, a recent work presents an unbiased formulation of the PAUC optimization problem via distributional robust optimization. However, it is based on the pair-wise formulation of AUC, which suffers from the limited scalability w.r.t. sample size and a slow convergence rate, especially for TPAUC. To address this issue, we present a simpler reformulation of the problem in an asymptotically unbiased and instance-wise manner. For both OPAUC and TPAUC, we come to a nonconvex strongly concave minimax regularized problem of instance-wise functions. On top of this, we employ an efficient solver enjoys a linear per-iteration computational complexity w.r.t. the sample size and a time-complexity of O(Ε−1/3) to reach a Ε stationary point. Furthermore, we find that the minimax reformulation also facilitates the theoretical analysis of generalization error as a byproduct. Compared with the existing results, we present new error bounds that are much easier to prove and could deal with hypotheses with real-valued outputs. Finally, extensive experiments on several benchmark datasets demonstrate the effectiveness of our method. Copyright © 2022, The Authors. All rights reserved.

关键词： Optimization

DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Liu, Shilong Liang, Yaoyuan Li, Feng Huang, Shijia Zhang, Hao Su, Hang Zhu, Jun Zhang, Lei Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. & Sys. Institute for AI Tsinghua-Bosch Joint Center for ML Tsinghua University China China Tsinghua-Berkeley Shenzhen Institute Tsinghua University China The Hong Kong University of Science and Technology Hong Kong The Chinese University of Hong Kong Hong Kong

In this paper, we study the problem of visual grounding by considering both phrase extraction and grounding (PEG). In contrast to the previous phrase-known-at-test setting, PEG requires a model to extract phrases from text and locate objects from image simultaneously, which is a more practical setting in real applications. As phrase extraction can be regarded as a 1D text segmentation problem, we formulate PEG as a dual detection problem and propose a novel DQ-DETR model, which introduces dual queries to probe different features from image and text for object prediction and phrase mask prediction. Each pair of dual queries is designed to have shared positional parts but different content parts. Such a design effectively alleviates the difficulty of modality alignment between image and text (in contrast to a single query design) and empowers Transformer decoder to leverage phrase mask-guided attention to improve the performance. To evaluate the performance of PEG, we also propose a new metric CMAP (cross-modal average precision), analogous to the AP metric in object detection. The new metric overcomes the ambiguity of Recall@1 in many-box-to-one-phrase cases in phrase grounding. As a result, our PEG pre-trained DQ-DETR establishes new state-of-the-art results on all visual grounding benchmarks with a ResNet-101 backbone. For example, it achieves 91.04% and 83.51% in terms of recall rate on RefCOCO testA and testB with a ResNet-101 backbone. Code will be availabl at https://***/IDEA-Research/DQ-DETR. Copyright © 2022, The Authors. All rights reserved.

关键词： Image enhancement

Scalable quasi-Bayesian inference for instrumental variable regression 21

学校读者我要写书评

暂无评论

Scalable quasi-Bayesian inference for instrumental variable ...

Proceedings of the 35th International Conference on Neural Information Processing systems

作者： Ziyu Wang Yuhao Zhou Tongzheng Ren Jun Zhu Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. & Sys. Institute for AI Tsinghua-Bosch Joint Center for ML Tsinghua University Department of Computer Science UT Austin

ISBN: (纸本)9781713845393

Recent years have witnessed an upsurge of interest in employing flexible machine learning models for instrumental variable (IV) regression, but the development of uncertainty quantification methodology is still lacking. In this work we present a scalable quasi-Bayesian procedure for IV regression, building upon the recently developed kernelized IV models. Contrary to Bayesian modeling for IV, our approach does not require additional assumptions on the data generating process, and leads to a scalable approximate inference algorithm with time cost comparable to the corresponding point estimation methods. Our algorithm can be further extended to work with neural network models. We analyze the theoretical properties of the proposed quasi-posterior, and demonstrate through empirical evaluation the competitive performance of our method.

关键词：

Relaxed conditional image transfer for semi-supervised domain adaptation

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Luo, Qijun Liu, Zhili Hong, Lanqing Li, Chongxuan Yang, Kuo Wang, Liyuan Zhou, Fengwei Li, Guilin Li, Zhenguo Zhu, Jun Chinese University of Hong Kong Hong Kong The Hong Kong University of Science and Technology China Huawei Noah’s Ark Lab China Dept. of Comp. Sci. & Tech. Institute for AI THBI Lab BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China

Semi-supervised domain adaptation (SSDA), which aims to learn models in a partially labeled target domain with the assistance of the fully labeled source domain, attracts increasing attention in recent years. To explicitly leverage the labeled data in both domains, we naturally introduce a conditional GAN framework to transfer images without changing the semantics in SSDA. However, we identify a label-domination problem in such an approach. In fact, the generator tends to overlook the input source image and only memorizes prototypes of each class, which results in unsatisfactory adaptation performance. To this end, we propose a simple yet effective Relaxed conditional GAN (Relaxed cGAN) framework. Specifically, we feed the image without its label to our generator. In this way, the generator has to infer the semantic information of input data. We formally prove that its equilibrium is desirable and empirically validate its practical convergence and effectiveness in image transfer. Additionally, we propose several tech.iques to make use of unlabeled data in the target domain, enhancing the model in SSDA settings. We validate our method on the well-adopted datasets: Digits, DomainNet, and Office-Home. We achieve state-of-the-art performance on DomainNet, Office-Home and most digit benchmarks in low-resource and high-resource settings. Copyright © 2021, The Authors. All rights reserved.

关键词： Semantics

Stability and generalization of bilevel programming in hyperparameter optimization

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Bao, Fan Wu, Guoqiang Li, Chongxuan Zhu, Jun Zhang, Bo Dept. of Comp. Sci. & Tech. Institute for AI Tsinghua-Huawei Joint Center for AI BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China School of Software Shandong University Gaoling School of AI Renmin University of China

关键词： Machine learning