检索结果-内蒙古大学图书馆

Stochastic expectation maximization with variance reduction 18

学校读者我要写书评

暂无评论

Stochastic expectation maximization with variance reduction

Proceedings of the 32nd International Conference on Neural Information Processing systems

作者： Jianfei Chen Jun Zhu Yee Whye Teh Tong Zhang Dept. of Comp. Sci. & Tech. BNRist Center State Key Lab for Intell. Tech. & Sys. Institute for AI THBI Lab Tsinghua University Beijing China Department of Statistics University of Oxford Tencent AI Lab

Expectation-Maximization (EM) is a popular tool for learning latent variable models, but the vanilla batch EM does not scale to large data sets because the whole data set is needed at every E-step. Stochastic Expectation Maximization (sEM) reduces the cost of E-step by stochastic approximation. However, sEM has a slower asymptotic convergence rate than batch EM, and requires a decreasing sequence of step sizes, which is difficult to tune. In this paper, we propose a variance reduced stochastic EM (sEM-vr) algorithm inspired by variance reduced stochastic gradient descent algorithms. We show that sEM-vr has the same exponential asymptotic convergence rate as batch EM. Moreover, sEM-vr only requires a constant step size to achieve this rate, which alleviates the burden of parameter tuning. We compare sEM-vr with batch EM, sEM and other algorithms on Gaussian mixture models and probabilistic latent semantic analysis, and sEM-vr converges significantly faster than these baselines.

关键词：

Graphical generative adversarial networks 18

学校读者我要写书评

暂无评论

Graphical generative adversarial networks

Proceedings of the 32nd International Conference on Neural Information Processing systems

作者： Chongxuan Li Max Welling Jun Zhu Bo Zhang Department of Computer Science & Technology Institute for Artificial Intelligence BNRist Center THBI Lab State Key Lab for Intell. Tech. & Sys. Tsinghua University University of Amsterdam and the Canadian Institute for Advanced Research (CIFAR)

We propose Graphical Generative Adversarial Networks (Graphical-GAN) to model structured data. Graphical-GAN conjoins the power of Bayesian networks on compactly representing the dependency structures among random variables and that of generative adversarial networks on learning expressive dependency functions. We introduce a structured recognition model to infer the posterior distribution of latent variables given observations. We generalize the Expectation Propagation (EP) algorithm to learn the generative model and recognition model jointly. Finally, we present two important instances of Graphical-GAN, i.e. Gaussian Mixture GAN (GMGAN) and state Space GAN (SSGAN), which can successfully learn the discrete and temporal structures on visual datasets, respectively.

关键词：

Scalable inference for nested chinese restaurant process topic models

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Chen, Jianfei Zhu, Jun Lu, Jie Liu, Shixia Dept. of Comp. Sci. and Tech. TNList Lab State Key Lab for Intell. Tech. and Sys. Cbicr Center Tsinghua University Beijing100084 China School of Software TNList Lab State Key Lab for Intell. Tech. and Sys. Tsinghua University Beijing100084 China

Nested Chinese Restaurant Process (nCRP) topic models are powerful nonparametric Bayesian methods to extract a topic hierarchy from a given text corpus, where the hierarchical structure is automatically determined by the data. Hierarchical Latent Dirichlet Allocation (hLDA) is a popular instance of nCRP topic models. However, hLDA has only been evaluated at small scale, because the existing collapsed Gibbs sampling and instantiated weight variational inference algorithms either are not scalable or sacrifice inference quality with mean-field assumptions. Moreover, an efficient distributed implementation of the data structures, such as dynamically growing count matrices and trees, is challenging. In this paper, we propose a novel partially collapsed Gibbs sampling (PCGS) algorithm, which combines the advantages of collapsed and instantiated weight algorithms to achieve good scalability as well as high model quality. An initialization strategy is presented to further improve the model quality. Finally, we propose an efficient distributed implementation of PCGS through vectorization, pre-processing, and a careful design of the concurrent data structures and communication strategy. Empirical studies show that our algorithm is 111 times more efficient than the previous open-source implementation for hLDA, with comparable or even *** model quality. Our distributed implementation can extract 1,722 topics from a 131-million-document corpus with 28 billion tokens, which is 4-5 orders of magnitude larger than the previous largest corpus, with 50 machines in 7 hours. Copyright © 2017, The Authors. All rights reserved.

关键词： Statistics

Riemannian Stein variational gradient descent for Bayesian inference

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Liu, Chang Zhu, Jun Dept. of Comp. Sci. & Tech. TNList Lab Center for Bio-Inspired Computing Research State Key Lab for Intell. Tech. & Systems Tsinghua University Beijing China

We develop Riemannian Stein Variational Gradient Descent (RSVGD), a Bayesian inference method that generalizes Stein Variational Gradient Descent (SVGD) to Riemann manifold. The benefits are two-folds: (i) for inference tasks in Euclidean spaces, RSVGD has the advantage over SVGD of utilizing information geometry, and (ii) for inference tasks on Riemann manifolds, RSVGD brings the unique advantages of SVGD to the Riemannian world. To appropriately transfer to Riemann manifolds, we conceive novel and non-trivial tech.iques for RSVGD, which are required by the intrinsically different characteristics of general Riemann manifolds from Euclidean spaces. We also discover Riemannian Stein's Identity and Riemannian Kernelized Stein Discrepancy. Experimental results show the advantages over SVGD of exploring distribution geometry and the advantages of particle-efficiency, iteration-effectiveness and approximation flexibility over other inference methods on Riemann manifolds. Copyright © 2017, The Authors. All rights reserved.

关键词： Gradient methods

Population matching discrepancy and applications in deep learning 17

学校读者我要写书评

暂无评论

Population matching discrepancy and applications in deep lea...

Proceedings of the 31st International Conference on Neural Information Processing systems

作者： Jianfei Chen Chongxuan Li Yizhong Ru Jun Zhu Dept. of Comp. Sci. & Tech. TNList Lab State Key Lab for Intell. Tech. & Sys. Tsinghua University Beijing China

ISBN: (纸本)9781510860964

A differentiable estimation of the distance between two distributions based on samples is important for many deep learning tasks. One such estimation is maximum mean discrepancy (MMD). However, MMD suffers from its sensitive kernel bandwidth hyper-parameter, weak gradients, and large mini-batch size when used as a training objective. In this paper, we propose population matching discrepancy (PMD) for estimating the distribution distance based on samples, as well as an algorithm to learn the parameters of the distributions using PMD as an objective. PMD is defined as the minimum weight matching of sample populations from each distribution, and we prove that PMD is a strongly consistent estimator of the first Wasserstein metric. We apply PMD to two deep learning tasks, domain adaptation and generative modeling. Empirical results demonstrate that PMD overcomes the aforementioned drawbacks of MMD, and outperforms MMD on both tasks in terms of the performance as well as the convergence speed.

关键词：

Message passing stein variational gradient descent

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Zhuo, Jingwei Liu, Chang Shi, Jiaxin Zhu, Jun Chen, Ning Zhang, Bo Dept. of Comp. Sci. and Tech. BNRist Center State Key Lab for Intell. Tech. and Sys. Thbi Lab Tsinghua University Beijing100084 China

Stein variational gradient descent (SVGD) is a re-cently proposed particle-based Bayesian inference method, which has attracted a lot of interest due to its remarkable approximation ability and particle efficiency compared to traditional variational in-ference and Markov Chain Monte Carlo methods. However, we observed that particles of SVGD tend to collapse to modes of the target distribution, and this particle degeneracy phenomenon be-comes more severe with higher dimensions. Our theoretical analysis finds out that there exists a negative correlation between the dimensionality and the repulsive force of SVGD which should be blamed for this phenomenon. We propose Message Passing SVGD (MP-SVGD) to solve this problem. By leveraging the conditional indepen-dence structure of probabilistic graphical models (PGMs), MP-SVGD converts the original high-dimensional global inference problem into a set of local ones over the Markov blanket with lower dimensions. Experimental results show its advan-tages of preventing vanishing repulsive force in high-dimensional space over SVGD, and its parti-cle efficiency and approximation flexibility over other inference methods on graphical models. Copyright © 2017, The Authors. All rights reserved.

关键词： Gradient methods

Smooth neighbors on teacher graphs for semi-supervised learning

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Luo, Yucen Zhu, Jun Li, Mengxi Ren, Yong Zhang, Bo Dept. of Comp. Sci. & Tech. State Key Lab for Intell. Tech. & Sys. BNRist Lab Tsinghua University Department of Electronical Engineering Tsinghua University

The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data samples. In this paper, we propose a novel method, called Smooth Neighbors on Teacher Graphs (SNTG). In SNTG, a graph is constructed based on the predictions of the teacher model, i.e., the implicit self-ensemble of models. Then the graph serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold. We achieve state-of-the-art results on semi-supervised learning benchmarks. The error rates are 9.89%, 3.99% for CIFAR-10 with 4000 labels, SVHN with 500 labels, respectively. In particular, the improvements are significant when the labels are fewer. For the non-augmented MNIST with only 20 labels, the error rate is reduced from previous 4.81% to 1.36%. Our method also shows robustness to noisy labels. Copyright © 2017, The Authors. All rights reserved.

关键词： Supervised learning

Triple generative adversarial nets 17

学校读者我要写书评

暂无评论

Triple generative adversarial nets

Proceedings of the 31st International Conference on Neural Information Processing systems

作者： Chongxuan Li Kun Xu Jun Zhu Bo Zhang Dept. of Comp. Sci. & Tech. TNList Lab State Key Lab of Intell. Tech. & Sys. Center for Bio-Inspired Computing Research Tsinghua University Beijing China

ISBN: (纸本)9781510860964

Generative Adversarial Nets (GANs) have shown promise in image generation and semi-supervised learning (SSL). However, existing GANs in SSL have two problems: (1) the generator and the discriminator (i.e. the classifier) may not be optimal at the same time; and (2) the generator cannot control the semantics of the generated samples. The problems essentially arise from the two-player formulation, where a single discriminator shares incompatible roles of identifying fake samples and predicting labels and it only estimates the data without considering the labels. To address the problems, we present triple generative adversarial net (Triple-GAN), which consists of three players—a generator, a discriminator and a classifier. The generator and the classifier characterize the conditional distributions between images and labels, and the discriminator solely focuses on identifying fake image-label pairs. We design compatible utilities to ensure that the distributions characterized by the classifier and the generator both converge to the data distribution. Our results on various datasets demonstrate that Triple-GAN as a unified model can simultaneously (1) achieve the state-of-the-art classification results among deep generative models, and (2) disentangle the classes and styles of the input and transfer smoothly in the data space via interpolation in the latent space class-conditionally.

关键词：

Towards robust detection of adversarial examples

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Pang, Tianyu Du, Chao Dong, Yinpeng Zhu, Jun Departmeng of Computer Science and Technology Institute for Artificial Intelligence BNRist Center State Key Lab for Intell. Tech. and Sys. THBI Lab Tsinghua University Beijing China

Although the recent progress is substantial, deep learning methods can be vulnerable to the maliciously generated adversarial examples. In this paper, we present a novel training procedure and a thresholding test strategy, towards robust detection of adversarial examples. In training, we propose to minimize the reverse crossentropy (RCE), which encourages a deep network to learn latent representations that better distinguish adversarial examples from normal ones. In testing, we propose to use a thresholding strategy as the detector to filter out adversarial examples for reliable predictions. Our method is simple to implement using standard algorithms, with little extra training cost compared to the common cross-entropy minimization. We apply our method to defend various attacking methods on the widely used MNIST and CIFAR-10 datasets, and achieve significant improvements on robust predictions under all the threat models in the adversarial setting. Copyright © 2017, The Authors. All rights reserved.

关键词： Entropy