A Third-Order Majorization Algorithm for Logistic Regression With Convergence Rate Guarantees

作     者：Zhao, Licheng Pu, Wenqiang Zhou, Rui Shi, Qingjiang 

作者机构：Shenzhen Res Inst Big Data Shenzhen 518172 Peoples R China Tongji Univ Sch Software Engn Shanghai 201804 Peoples R China Shenzhen Res Inst Big Data Shenzhen 518172 Peoples R China 

出 版 物：《IEEE SIGNAL PROCESSING LETTERS》 (IEEE Signal Process Lett)

年 卷 期：2024年第31卷

页      面：1700-1704页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 

基　　金：National Nature Science Foundation of China (NSFC) [62206182, 62101350, 62201362] Guangdong Basic and Applied Basic Research Foundation [2024A1515010154] Shenzhen Science and Technology Program [RCBS20221008093126071] 

主　　题：Signal processing algorithms Minimization Convergence Optimization Taylor series Machine learning algorithms Logistic regression Convergence rate logistic regression majorization-minimization third-order 

摘      要：In this paper, we study the classical Logistic Regression (LR) problem in machine learning. Traditionally, the solving algorithms are based on either the first- or second-order approximation of the objective. For instance, the Fixed-Hessian Newton (FHN) method approximates the true Hessian with a constant estimate. In contrast, our design additionally exploits the third-order information. Applying the majorization-minimization (MM) framework, we construct a novel majorizing function based on the third-order Taylor expansion and the minimization solution is in closed-form with perseverance of the true gradient and Hessian structures. In analysis, we prove the convergence rate of the proposed algorithm. The enhanced numerical performance can be verified through simulation results.