Data uncertainty is a challenging problem in machine learning. Distributionally robust optimization (DRO) can be used to model the data uncertainty. Based on DRO, a new support vector machines with double regularizati...
详细信息
Data uncertainty is a challenging problem in machine learning. Distributionally robust optimization (DRO) can be used to model the data uncertainty. Based on DRO, a new support vector machines with double regularization terms and double margins can be derived. The proposed model can capture the data uncertainty in a probabilistic way and perform automatic feature selection for high dimensional data. We prove that the optimal solutions of this model change piecewise linearly with respect to the hyperparameters. Based on this property, we can derive the entire solutionpath by computing solutions only at the breakpoints. A solution path algorithm is proposed to efficiently identify the optimal solutions, thereby accelerating the hyperparameter tuning process. In computational efficiency experiments, the proposed solution path algorithm demonstrates superior performance compared to the CVXPY method and the Sequential Minimal Optimization (SMO) algorithm. Numerical experiments further confirm that the proposed model achieves robust performance even under noisy data conditions.
In this paper, we propose a general distributionally robust regression model based on distributionally robust optimization theory. The proposed model has a piecewise linear loss function and elastic net penalty term, ...
详细信息
In this paper, we propose a general distributionally robust regression model based on distributionally robust optimization theory. The proposed model has a piecewise linear loss function and elastic net penalty term, and it generalizes many other regression models. We prove the piecewise linear property of the optimal solutions to this model, which enables us to develop a solution path algorithm for the hyperparameter tuning. A Doubly regularized Least Absolute Deviations (DrLAD) regression model is proposed based on this framework, and a solution path algorithm is developed to speed up the tuning of two hyperparameters in this model. Numerical experiments are implemented to validate the performance of this model and the computational efficiency of the solution path algorithm.
The twin support vector machine and its extensions have made great achievements in dealing with binary classification problems. However, it suffers from difficulties in effective solution of multi-classification and f...
详细信息
The twin support vector machine and its extensions have made great achievements in dealing with binary classification problems. However, it suffers from difficulties in effective solution of multi-classification and fast model selection. This work devotes to the fast regularization parameter tuning algorithm for the twin multi-class support vector machine. Specifically, a novel sample data set partition strategy is first adopted, which is the basis for the model construction. Then, combining the linear equations and block matrix theory, the Lagrangian multipliers are proved to be piecewise linear w.r.t. the regularization parameters, so that the regularization parameters are continuously updated by only solving the break points. Next, Lagrangian multipliers are proved to be 1 as the regularization parameter approaches infinity, thus, a simple yet effective initialization algorithm is devised. Finally, eight kinds of events are defined to seek for the starting event for the next iteration. Extensive experimental results on nine UCI data sets show that the proposed method can achieve comparable classification performance without solving any quadratic programming problem.
Variable selection is a challenging problem in high-dimensional linear regression problems with a large number of predictors. Thus, sparsity-inducing and clustering-inducing regularization methods are widely used to i...
详细信息
ISBN:
(纸本)9781728146034
Variable selection is a challenging problem in high-dimensional linear regression problems with a large number of predictors. Thus, sparsity-inducing and clustering-inducing regularization methods are widely used to identify highly correlated covariates. Ordered Weight L-1 (OWL) family of regularizers for linear regression perform well to identify precise clusters of correlated covariates and interpret the effect of each variable. solution path algorithms are helpful to select hyperparameters to tune the OWL model. Due to over-complex representation of the penalty, so far the OWL model has no solution path algorithms for hyperparameter selection. To address this challenge, in this paper, we propose an efficient approximate solution path algorithm (OWLAGpath) to solve the OWL model with accuracy guarantee. For a given accuracy bound epsilon, OWLAGpath can find the corresponding solutions for the OWL model with numerous hyperparameters while keeping the sparsity and precise features grouping properties. Theoretically, we prove that all the solutions produced by OWLAGpath can strictly satisfy the given accuracy bound epsilon. The experimental results on three benchmark datasets not only confirm the effectiveness and efficiency of our OWLAGpathalgorithm, but also show the advantages of OWLAGpath for model selection than the existing algorithms.
A fuzzy weighted doubly regularized support vector machine for binary classification is proposed in this paper. Fuzzy weights are presented by using the distance information within each class. Then the fuzzy weighted ...
详细信息
ISBN:
(纸本)9781479970162
A fuzzy weighted doubly regularized support vector machine for binary classification is proposed in this paper. Fuzzy weights are presented by using the distance information within each class. Then the fuzzy weighted doubly regularized support vector machine is proposed by combing the weighted hinge loss and the adaptive elastic net penalty. A reasonable correlation between two model parameters is also given and the solution path algorithm to compute the solutionpaths of the proposed support vector machine is developed. The simulation results on two data sets demonstrate the effectiveness of the proposed method.
A fuzzy weighted doubly regularized support vector machine for binary classification is proposed in this *** weights are presented by using the distance information within each *** the fuzzy weighted doubly regularize...
详细信息
ISBN:
(纸本)9781479970186
A fuzzy weighted doubly regularized support vector machine for binary classification is proposed in this *** weights are presented by using the distance information within each *** the fuzzy weighted doubly regularized support vector machine is proposed by combing the weighted hinge loss and the adaptive elastic net penalty.A reasonable correlation between two model parameters is also given and the solution path algorithm to compute the solutionpaths of the proposed support vector machine is *** simulation results on two data sets demonstrate the effectiveness of the proposed method.
In high dimensional regression, clustering features according to their effects on outcomes is often as important as feature selection. For example, insurance premiums are set for each rate class pertaining to risk fac...
详细信息
In high dimensional regression, clustering features according to their effects on outcomes is often as important as feature selection. For example, insurance premiums are set for each rate class pertaining to risk factors related to claim risk. To calculate reliable insurance premiums, it is often necessary to group the numerous rate classes into fewer classes. However, the combinations of ways to consolidate rate classes are vast, and it is computationally challenging to consider each combination individually. Under such circumstances, sparse regularization techniques for feature clustering are extremely useful as methods that automatically consolidate rate classes with no significant differences in risk levels during the estimation process. For this purpose, clustered Lasso and octagonal shrinkage and clustering algorithm for regression (OSCAR) can be used to generate feature groups automatically using pairwise L1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_1$$\end{document} norm and pairwise L infinity\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_\infty $$\end{document} norm, respectively. This paper proposes efficient pathalgorithms for clustered Lasso and OSCAR to construct solutionpaths with respect to their regularization parameters. Despite the excessive terms in exhaustive pairwise regularization, their computational costs are reduced by using the symmetry of those terms. Simple equivalent conditions to check subgradient equations in each feature group are derived using the graph theory. The proposed algorithms are shown to be more efficient than existing algorithms via numerical experimen
Fair classification which enforces a fairness constraint on the original learning problem is an emerging topic in machine learning. Due to its non-convexity and non-discontinuity, the original (true) fairness constrai...
详细信息
ISBN:
(纸本)9781450392365
Fair classification which enforces a fairness constraint on the original learning problem is an emerging topic in machine learning. Due to its non-convexity and non-discontinuity, the original (true) fairness constraint is normally relaxed to a convex and smooth surrogate which could lead to slightly deviated solutions and could violate the original fairness constraint. To re-calibrate with the original constraint, existing methods usually hand-tunes a hyper-parameter of the convex surrogate. Such a method is obviously time consuming, besides it cannot guarantee to find the fairer classifier (i.e., original fairness constraint is less than a smaller threshold). To address this challenging problem, we propose a novel true fairness score pathalgorithm which guarantees to find fairer classifiers efficiently. Specifically, we first give a new formulation of fair classification which treats the surrogate fairness constraint as an additional regularization term, with a fairness hyper-parameter controlling the degree of surrogate fairness. Then, we propose a solution path algorithm which tracks the solutions of fair classification regarding to the fairness hyper-parameter. Based on the solutionpath, we further propose a true fairness score pathalgorithm which derives the curve of fairness score with respect to the fairness hyper-parameter and allows us to find the fairer classifiers. Finally, extensive experimental results not only verify the effectiveness of our algorithm, but also show that we can find the fairer classifiers efficiently.
Self-paced learning (SPL) is an emerging research topic in recent machine learning research which is often formulated as a bi-convex problem. The choice of the age parameter in SPL can control the learning pace and is...
详细信息
ISBN:
(纸本)9781665423984
Self-paced learning (SPL) is an emerging research topic in recent machine learning research which is often formulated as a bi-convex problem. The choice of the age parameter in SPL can control the learning pace and is crucial to achieve optimal performance. Traditionally, the age parameter is programmed to increase in a fixed rate while solving the SPL problem using the alternative optimization strategy (AOS). However, this simple heuristic is likely to miss the optimal age parameter especially when efficiency is a major concern. To address this problem, we propose a solutionpath method, APSPL, which can track the optimal solutions of SPL with respect to the change of age parameter (age path). Specifically, we use the difference of convex (DC) formulation to replace the original biconvex problem, which enables us to derive the path-following algorithm. For better efficiency, our algorithm uses a decremental and incremental training strategy to avoid retraining several times at different age values. We theoretically prove that the solutions produced by APSPL are the same as those generated by traditional SPL solvers. We also provide the finite time convergence proof of APSPL. To demonstrate the applicability of APSPL, we provide an extension of APSPL for semi-supervised classification. To the best of our knowledge, APSPL is the first solution path algorithm for self-paced learning. Experimental results on a variety of benchmark datasets not only verify the effectiveness and efficiency of APSPL over traditional SPL, but also show the advantage of using the optimal age parameter.
Penalized regression can improve prediction accuracy and reduce dimension. The generalized lasso problem is used in many applications in various fields. The generalized lasso penalizes a linear transformation of the c...
详细信息
Penalized regression can improve prediction accuracy and reduce dimension. The generalized lasso problem is used in many applications in various fields. The generalized lasso penalizes a linear transformation of the coefficients rather than the coefficients themselves. The proposed algorithm solves the generalized lasso problem and provides the full solutionpath. A confidence set can then be constructed on the generalized lasso parameters based on the modified residual bootstrap lasso. The approach is demonstrated using spatially varying coefficients regression, and it is shown to be both accurate and efficient compared to previous work. (C) 2019 Elsevier B.V. All rights reserved.
暂无评论