检索结果-内蒙古大学图书馆

Multidimensional scaling, Sammon mapping, and Isomap: Tutorial and survey

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Ghojogh, Benyamin Ghodsi, Ali Karray, Fakhri Crowley, Mark Department of Electrical and Computer Engineering Machine Learning Laboratory University of Waterloo WaterlooON Canada Department of Statistics and Actuarial Science David R. Cheriton School of Computer Science Data Analytics Laboratory University of Waterloo WaterlooON Canada Department of Electrical and Computer Engineering Centre for Pattern Analysis and Machine Intelligence University of Waterloo WaterlooON Canada

Multidimensional Scaling (MDS) is one of the first fundamental manifold learning methods. It can be categorized into several methods, i.e., classical MDS, kernel classical MDS, metric MDS, and non-metric MDS. Sammon mapping and Isomap can be considered as special cases of metric MDS and kernel classical MDS, respectively. In this tutorial and survey paper, we review the theory of MDS, Sammon mapping, and Isomap in detail. We explain all the mentioned categories of MDS. Then, Sammon mapping, Isomap, and kernel Isomap are explained. Out-of-sample embedding for MDS and Isomap using eigenfunctions and kernel mapping are introduced. Then, Nystrom approximation and its use in landmark MDS and landmark Isomap are introduced for big data embedding. We also provide some simulations for illustrating the embedding by these methods. Copyright © 2020, The Authors. All rights reserved.

关键词： Embeddings

Stochastic Neighbor Embedding with Gaussian and Student-t Distributions: Tutorial and Survey

学校读者我要写书评

暂无评论

arXiv 2020年

Stochastic Neighbor Embedding (SNE) is a manifold learning and dimensionality reduction method with a probabilistic approach. In SNE, every point is consider to be the neighbor of all other points with some probability and this probability is tried to be preserved in the embedding space. SNE considers Gaussian distribution for the probability in both the input and embedding spaces. However, t-SNE uses the Student-t and Gaussian distributions in these spaces, respectively. In this tutorial and survey paper, we explain SNE, symmetric SNE, t-SNE (or Cauchy-SNE), and t-SNE with general degrees of freedom. We also cover the out-of-sample extension and acceleration for these methods. Copyright © 2020, The Authors. All rights reserved.

关键词： Gaussian distribution

Locally linear embedding and its variants: Tutorial and survey

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Ghojogh, Benyamin Ghodsi, Ali Karray, Fakhri Crowley, Mark Department of Electrical and Computer Engineering Machine Learning Laboratory University of Waterloo WaterlooON Canada Department of Statistics and Actuarial Science & David R. Cheriton School of Computer Science Data Analytics Laboratory University of Waterloo WaterlooON Canada Department of Electrical and Computer Engineering Centre for Pattern Analysis and Machine Intelligence University of Waterloo WaterlooON Canada

This is a tutorial and survey paper for Locally Linear Embedding (LLE) and its variants. The idea of LLE is fitting the local structure of manifold in the embedding space. In this paper, we first cover LLE, kernel LLE, inverse LLE, and feature fusion with LLE. Then, we cover out-of-sample embedding using linear reconstruction, eigenfunctions, and kernel mapping. Incremental LLE is explained for embedding streaming data. Landmark LLE methods using the Nystrom approximation and locally linear landmarks are explained for big data embedding. We introduce the methods for parameter selection of number of neighbors using residual variance, Procrustes statistics, preservation neighborhood error, and local neighborhood selection. Afterwards, Supervised LLE (SLLE), enhanced SLLE, SLLE projection, probabilistic SLLE, supervised guided LLE (using Hilbert-Schmidt independence criterion), and semi-supervised LLE are explained for supervised and semi-supervised embedding. Robust LLE methods using least squares problem and penalty functions are also introduced for embedding in the presence of outliers and noise. Then, we introduce fusion of LLE with other manifold learning methods including Isomap (i.e., ISOLLE), principal component analysis, Fisher discriminant analysis, discriminant LLE, and Isotop. Finally, we explain weighted LLE in which the distances, reconstruction weights, or the embeddings are adjusted for better embedding;we cover weighted LLE for deformed distributed data, weighted LLE using probability of occurrence, SLLE by adjusting weights, modified LLE, and iterative LLE. Copyright © 2020, The Authors. All rights reserved.

关键词： Iterative methods

Geometric interpretation of running Nyström-based kernel machines and error analysis

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Li, Weida Liu, Mingxia Zhang, Daoqiang The College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing211106 China The Department of Radiology and BRIC University of North Carolina at Chapel Hill Chapel HillNC27599 United States

Recently, Nyström method has proved its prominence empirically and theoretically in speeding up the training of kernel machines while retaining satisfactory performances and accuracy. So far, there are several different approaches proposed to exploit Nyström method in scaling up kernel machines. However, there is no comparative study over these approaches, and they were individually analyzed for specific types of kernel machines. Therefore, it remains a question that the philosophy of which approach is more promising when it extends to other kernel machines. In this work, motivated by the column inclusion property of Gram matrices, we develop a new approach with a clear geometric interpretation for running Nyström-based kernel machines. We show that the other two well-studied approaches can be equivalently transformed to be our proposed one. Consequently, analysis established for the proposed approach also works for these two. Particularly, our proposed approach makes it possible to develop approximation errors in a general setting. Besides, our analysis also manifests the relations among the aforementioned two approaches and another naive one. First, the analytical forms of the corresponding approximate solutions are only at odds with one term. Second, the naive approach can be implemented efficiently by sharing the same training procedure with others. These analytical results lead to the conjecture that the naive approach can provide more accurate approximate solutions than the other two sophisticated approaches. Since our analysis also offers ways for computing the accuracy of these approximate solutions, we run experiments with classification tasks to confirm our conjecture. Copyright © 2020, The Authors. All rights reserved.

关键词： machine learning

Faster stochastic alternating direction method of multipliers for nonconvex optimization

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Huang, Feihu Chen, Songcan Huang, Heng Department of Electrical & Computer Engineering University of Pittsburgh PA15261 United States College of Computer Science & Technology Nanjing University of Aeronautics and Astronautics Nanjing211106 China MIIT Key Laboratory of Pattern Analysis & Machine Intelligence JD Finance America Corporation United States

In this paper, we propose a faster stochastic alternating direction method of multipliers (ADMM) for nonconvex optimization by using a new stochastic path-integrated differential estimator (SPIDER), called as SPIDER-ADMM. Moreover, we prove that the SPIDER-ADMM achieves a record-breaking incremental first-order oracle (IFO) complexity of O(n + n1/2∊−1) for finding an ∊-approximate stationary point, which improves the deterministic ADMM by a factor O(n1/2), where n denotes the sample size. As one of major contribution of this paper, we provide a new theoretical analysis framework for nonconvex stochastic ADMM methods with providing the optimal IFO complexity. Based on this new analysis framework, we study the unsolved optimal IFO complexity of the existing nonconvex SVRG-ADMM and SAGA-ADMM methods, and prove they have the optimal IFO complexity of O(n + n2/3∊−1). Thus, the SPIDER-ADMM improves the existing stochastic ADMM methods by a factor of O(n1/6). Moreover, we extend SPIDER-ADMM to the online setting, and propose a faster online SPIDER-ADMM. Our theoretical analysis shows that the online SPIDER-ADMM has the IFO complexity of O(∊−3 2 ), which improves the existing best results by a factor of O(∊−2 1 ). Finally, the experimental results on benchmark datasets validate that the proposed algorithms have faster convergence rate than the existing ADMM algorithms for nonconvex optimization. Copyright © 2020, The Authors. All rights reserved.

关键词： Stochastic systems

Fisher and Kernel Fisher Discriminant analysis: Tutorial

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Ghojogh, Benyamin Karray, Fakhri Crowley, Mark Department of Electrical and Computer Engineering Canada Machine Learning Laboratory University of Waterloo WaterlooON Canada Centre for Pattern Analysis and Machine Intelligence University of Waterloo WaterlooON Canada

This is a detailed tutorial paper which explains the Fisher discriminant analysis (FDA) and kernel FDA. We start with projection and reconstruction. Then, one- and multi-dimensional FDA subspaces are covered. Scatters in two- and then multi-classes are explained in FDA. Then, we discuss on the rank of the scatters and the dimensionality of the subspace. A real-life example is also provided for interpreting FDA. Then, possible singularity of the scatter is discussed to introduce robust FDA. PCA and FDA directions are also compared. We also prove that FDA and linear discriminant analysis are equivalent. Fisher forest is also introduced as an ensemble of fisher subspaces useful for handling data with different features and dimensionality. After-wards, kernel FDA is explained for both one- and multi-dimensional subspaces with both two- and multi-classes. Finally, some simulations are performed on AT&T face dataset to illustrate FDA and compare it with PCA. Copyright © 2019, The Authors. All rights reserved.

关键词： Discriminant analysis

ALiPy: Active learning in python

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Tang, Ying-Peng Li, Guo-Xiang Huang, Sheng-Jun College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing211106 China

Supervised machine learning methods usually require a large set of labeled examples for model training. However, in many real applications, there are plentiful unlabeled data but limited labeled data;and the acquisition of labels is costly. Active learning (AL) reduces the labeling cost by iteratively selecting the most valuable data to query their labels from the annotator. This article introduces a Python toobox ALiPy1 for active learning. ALiPy provides a module based implementation of active learning framework, which allows users to conveniently evaluate, compare and analyze the performance of active learning methods. In the toolbox, multiple options are available for each component of the learning framework, including data process, active selection, label query, results visualization, etc. In addition to the implementations of more than 20 state-of-the-art active learning algorithms, ALiPy also supports users to easily configure and implement their own approaches under different active learning settings, such as AL for multi-label data, AL with noisy annotators, AL with different costs and so on. The toolbox is well-documented and open-source on Github2, and can be easily installed through PyPI. Copyright © 2019, The Authors. All rights reserved.

关键词： Python

Global expanding, local shrinking: Discriminant multi-label learning with missing labels

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Ma, Zhongchen Chen, Songcan School of Computer Science & communications Engineering Jiangsu University Zhenjiang212013 China College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics Nanjing211106 China MIIT Key Laboratory of Pattern Analysis & Machine Intelligence College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics Nanjing211106 China

In multi-label learning, the issue of missing labels brings a major challenge. Many methods attempt to recovery missing labels by exploiting low-rank structure of label matrix. However, these methods just utilize global low-rank label structure, ignore both local low-rank label structures and label discriminant information to some extent, leaving room for further performance improvement. In this paper, we develop a simple yet effective discriminant multi-label learning (DM2L) method for multi-label learning with missing labels. Specifically, we impose the low-rank structures on all the predictions of instances from the same labels (local shrinking of rank), and a maximally separated structure (high-rank structure) on the predictions of instances from different labels (global expanding of rank). In this way, these imposed low-rank structures can help modeling both local and global low-rank label structures, while the imposed high-rank structure can help providing more underlying discriminability. Our subsequent theoretical analysis also supports these intuitions. In addition, we provide a nonlinear extension via using kernel trick to enhance DM2L and establish a concave-convex objective to learn these models. Compared to the other methods, our method involves the fewest assumptions and only one hyper-parameter. Even so, extensive experiments show that our method still outperforms the state-of-the-art methods. Copyright © 2020, The Authors. All rights reserved.

关键词： machine learning

Addressing the mystery of population decline of the rose-crested blue pipit in a nature preserve using data visualization

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Ghojogh, Benyamin Crowley, Mark Karray, Fakhri Department of Electrical and Computer Engineering Machine Learning Laboratory University of Waterloo WaterlooON Canada Department of Electrical and Computer Engineering Centre for Pattern Analysis and Machine Intelligence University of Waterloo WaterlooON Canada

Two main methods for exploring patterns in data are data visualization and machine learning. The former relies on humans for investigating the patterns while the latter relies on machine learning algorithms. This paper tries to find the patterns using only data visualization. It addresses the mystery of population decline of a bird, named Rose-Crested Blue Pipit, in a hypothetical nature preserve. Different visualization techniques are used and the reasons of the problem are found and categorized. Finally, the solutions for preventing the future similar problems are suggested. This paper can be useful for getting introduced to some data visualization tools and techniques. Copyright © 2019, The Authors. All rights reserved.

关键词： Data visualization