检索结果-内蒙古大学图书馆

Bi-level score matching for learning energy-based latent variable models 20

学校读者我要写书评

暂无评论

Bi-level score matching for learning energy-based latent var...

Proceedings of the 34th International Conference on Neural Information Processing systems

作者： Fan Bao Chongxuan Li Kun Xu Hang Su Jun Zhu Bo Zhang Dept. of Comp. Sci. & Tech. Institute for AI THBI Lab BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China

ISBN: (纸本)9781713829546

Score matching (SM) [24] provides a compelling approach to learn energy-based models (EBMs) by avoiding the calculation of partition function. However, it remains largely open to learn energy-based latent variable models (EBLVMs), except some special cases. This paper presents a bi-level score matching (BiSM) method to learn EBLVMs with general structures by reformulating SM as a bi-level optimization problem. The higher level introduces a variational posterior of the latent variables and optimizes a modified SM objective, and the lower level optimizes the variational posterior to fit the true posterior. To solve BiSM efficiently, we develop a stochastic optimization algorithm with gradient unrolling. Theoretically, we analyze the consistency of BiSM and the convergence of the stochastic algorithm. Empirically, we show the promise of BiSM in Gaussian restricted Boltzmann machines and highly nonstructural EBLVMs parameterized by deep convolutional neural networks. BiSM is comparable to the widely adopted contrastive divergence and SM methods when they are applicable; and can learn complex EBLVMs with intractable posteriors to generate natural images.

关键词：

Function space particle optimization for Bayesian neural networks 7

学校读者我要写书评

暂无评论

Function space particle optimization for Bayesian neural net...

7th International Conference on Learning Representations, ICLR 2019

作者： Wang, Ziyu Ren, Tongzheng Zhu, Jun Zhang, Bo Department of Computer Science and Technology Institute for Artificial Intelligence State Key Lab for Intell. Tech. and Sys. BNRist Center THBI Lab Tsinghua University China

While Bayesian neural networks (BNNs) have drawn increasing attention, their posterior inference remains challenging, due to the high-dimensional and over-parameterized nature. Recently, several highly flexible and scalable variational inference procedures based on the idea of particle optimization have been proposed. These methods directly optimize a set of particles to approximate the target posterior. However, their application to BNNs often yields sub-optimal performance, as they have a particular failure mode on over-parameterized models. In this paper, we propose to solve this issue by performing particle optimization directly in the space of regression functions. We demonstrate through extensive experiments that our method successfully overcomes this issue, and outperforms strong baselines in a variety of tasks including prediction, defense against adversarial examples, and reinforcement learning. © 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved.

关键词： Reinforcement learning

Composite binary decomposition networks 33

学校读者我要写书评

暂无评论

Composite binary decomposition networks

33rd AAAI Conference on Artificial intell.gence, AAAI 2019, 31st Annual Conference on Innovative Applications of Artificial intell.gence, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial intell.gence, EAAI 2019

作者： Qiaoben, You Wang, Zheng Li, Jianguo Dong, Yinpeng Jiang, Yu-Gang Zhu, Jun Dept. of Comp. Sci. and Tech. State Key Lab for Intell. Tech. and Sys. Institute for AI Tsinghua University China School of Computer Science Fudan University China Intel Labs China China

ISBN: (纸本)9781577358091

Binary neural networks have great resource and computing efficiency, while suffer from long training procedure and non-negligible accuracy drops, when comparing to the full-precision counterparts. In this paper, we propose the composite binary decomposition networks (CBDNet), which first compose real-valued tensor of each layer with a limited number of binary tensors, and then decompose some conditioned binary tensors into two low-rank binary tensors, so that the number of parameters and operations are greatly reduced comparing to the original ones. Experiments demonstrate the effectiveness of the proposed method, as CBDNet can approximate image classification network ResNet-18 using 5.25 bits, VGG-16 using 5.47 bits, DenseNet-121 using 5.72 bits, object detection networks SSD300 using 4.38 bits, and semantic segmentation networks SegNet using 5.18 bits, all with minor accuracy drops. 1,. © 2019, Association for the Advancement of Artificial intell.gence (***).

关键词： Tensors

Scalable training of hierarchical topic models 44th

学校读者我要写书评

暂无评论

Scalable training of hierarchical topic models

44th International Conference on Very Large Data Bases, VLDB 2018

作者： Chen, Jianfei Zhu, Jun Lu, Jie Liu, Shixia Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. and Sys. Tsinghua University Beijing100084 China School of Software BNRist Center State Key Lab for Intell. Tech. and Sys. Tsinghua University Beijing100084 China

Large-scale topic models serve as basic tools for feature extraction and dimensionality reduction in many practical applications. As a natural extension of at topic models, hierarchical topic models (HTMs) are able to learn topics of different levels of abstraction, which lead to deeper un- derstanding and better generalization than their at coun- terparts. However, existing scalable systems for at topic models cannot handle HTMs, due to their complicated data structures such as trees and concurrent dynamically growing matrices, as well as their susceptibility to local optima. In this paper, we study the hierarchical latent Dirichlet al- location (hLDA) model which is a powerful nonparametric Bayesian HTM. We propose an efficient partially collapsed Gibbs sampling algorithm for hLDA, as well as an initializa- tion strategy to deal with local optima introduced by tree- structured models. We also identify new system challenges in building scalable systems for HTMs, and propose efficient data layout for vectorizing HTM as well as distributed data structures including dynamic matrices and trees. Empiri- cal studies show that our system is 87 times more efficient than the previous open-source implementation for hLDA, and can scale to thousands of CPU cores. We demonstrate our scalability on a 131-million-document corpus with 28 billion tokens, which is 4-5 orders of magnitude larger than previously used corpus. Our distributed implementation can extract 1,722 topics from the corpus with 50 machines in just 7 hours. © 2018 VLDB Endowment 21508097/18/4.

关键词： Matrix algebra

Reward shaping via meta-learning

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Zou, Haosheng Ren, Tongzheng Yan, Dong Su, Hang Zhu, Jun Dept. of Comp. Sci. & Tech. State Key Lab of Intell. Tech. & Sys. TNList Lab CBICR Center Tsinghua University Beijing China

Reward shaping is one of the most effective methods to tackle the crucial yet challenging problem of credit assignment in Reinforcement Learning (RL). However, designing shaping functions usually requires much expert knowledge and hand-engineering, and the difficulties are further exacerbated given multiple similar tasks to solve. In this paper, we consider reward shaping on a distribution of tasks, and propose a general meta-learning framework to automatically learn the efficient reward shaping on newly sampled tasks, assuming only shared state space but not necessarily action space. We first derive the theoretically optimal reward shaping in terms of credit assignment in model-free RL. We then propose a value-based meta-learning algorithm to extract an effective prior over the optimal reward shaping. The prior can be applied directly to new tasks, or provably adapted to the task-posterior while solving the task within few gradient updates. We demonstrate the effectiveness of our shaping through significantly improved learning efficiency and interpretable visualizations across various settings, including notably a successful transfer from DQN to DDPG. Copyright © 2019, The Authors. All rights reserved.

关键词： Learning algorithms

Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks

学校读者我要写书评

暂无评论

Evading Defenses to Transferable Adversarial Examples by Tra...

IEEE/CVF Conference on Computer Vision and Pattern Recognition

作者： Yinpeng Dong Tianyu Pang Hang Su Jun Zhu Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. & Sys. Institute for AI THBI Lab Tsinghua University

ISBN: (纸本)9781728132945

Deep neural networks are vulnerable to adversarial examples, which can mislead classifiers by adding imperceptible perturbations. An intriguing property of adversarial examples is their good transferability, making black-box attacks feasible in real-world applications. Due to the threat of adversarial attacks, many methods have been proposed to improve the robustness. Several state-of-the-art defenses are shown to be robust against transferable adversarial examples. In this paper, we propose a translation-invariant attack method to generate more transferable adversarial examples against the defense models. By optimizing a perturbation over an ensemble of translated images, the generated adversarial example is less sensitive to the white-box model being attacked and has better transferability. To improve the efficiency of attacks, we further show that our method-can be implemented by convolving the gradient at the untranslated image with a pre-defined kernel. Our method is generally applicable to any gradient-based attack method. Extensive experiments on the ImageNet dataset validate the effectiveness of the proposed method. Our best attack fools eight state-of-the-art defenses at an 82% success rate on average based only on the transferability, demonstrating the insecurity of the current defense tech.iques.

关键词： Portability defenses BOX MODELS Unsafe Classifiers Vulnerable perturbation

Multi-objects generation with amortized structural regularization

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Xu, Kun Li, Chongxuan Zhu, Jun Zhang, Bo Dept. of Comp. Sci. & Tech. Institute for AI THBI Lab BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China

Deep generative models (DGMs) have shown promise in image generation. However, most of the existing work learn the model by simply optimizing a divergence between the marginal distributions of the model and the data, and often fail to capture the rich structures and relations in multi-object images. Human knowledge is a critical element to the success of DGMs to infer these structures. In this paper, we propose the amortized structural regularization (ASR) framework, which adopts the posterior regularization (PR) to embed human knowledge into DGMs via a set of structural constraints. We derive a lower bound of the regularized log-likelihood, which can be jointly optimized with respect to the generative model and recognition model efficiently. Empirical results show that ASR significantly outperforms the DGM baselines in terms of inference accuracy and sample quality. Copyright © 2019, The Authors. All rights reserved.

关键词： Machine learning

Generative Well-intentioned Networks

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Cosentino, Justin Zhu, Jun Dept. of Comp. Sci. & Tech. Institute for AI THBI Lab BNRist Center State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China

We propose Generative Well-intentioned Networks (GWINs), a novel framework for increasing the accuracy of certainty-based, closed-world classifiers. A conditional generative network recovers the distribution of observations that the classifier labels correctly with high certainty. We introduce a reject option to the classifier during inference, allowing the classifier to reject an observation instance rather than predict an uncertain label. These rejected observations are translated by the generative network to high-certainty representations, which are then relabeled by the classifier. This architecture allows for any certainty-based classifier or rejection function and is not limited to multilayer perceptrons. The capability of this framework is assessed using benchmark classification datasets and shows that GWINs significantly improve the accuracy of uncertain observations. Copyright © 2019, The Authors. All rights reserved.

关键词： Classification (of information)

Evading defenses to transferable adversarial examples by translation-invariant attacks

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Dong, Yinpeng Pang, Tianyu Su, Hang Zhu, Jun Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. and Sys. Institute for AI THBI Lab Tsinghua University Beijing100084 China

Deep neural networks are vulnerable to adversarial examples, which can mislead classifiers by adding imperceptible perturbations. An intriguing property of adversarial examples is their good transferability, making black-box attacks feasible in real-world applications. Due to the threat of adversarial attacks, many methods have been proposed to improve the robustness. Several state-of-the-art defenses are shown to be robust against transferable adversarial examples. In this paper, we propose a translation-invariant attack method to generate more transferable adversarial examples against the defense models. By optimizing a perturbation over an ensemble of translated images, the generated adversarial example is less sensitive to the white-box model being attacked and has better transferability. To improve the efficiency of attacks, we further show that our method can be implemented by convolving the gradient at the untranslated image with a pre-defined kernel. Our method is generally applicable to any gradient-based attack method. Extensive experiments on the ImageNet dataset validate the effectiveness of the proposed method. Our best attack fools eight state-of-the-art defenses at an 82% success rate on average based only on the transferability, demonstrating the insecurity of the current defense tech.iques. Copyright © 2019, The Authors. All rights reserved.

关键词： Deep neural networks