A unified framework for correlation mining in ultra-high dimension

作     者：Wei, Yun Rajaratnam, Bala Hero, Alfred O. 

作者机构：Department of Statistical Science Duke University DurhamNC27707 United States Department of Mathematics University of Michigan Ann ArborMI48109 United States Department of Statistics University of California DavisCA95616 United States Department of Electrical Engineering and Computer Science University of Michigan Ann ArborMI48109 United States 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2021年

核心收录：

主　　题：Covariance matrix 

摘      要：Many applications benefit from theory relevant to the identification of variables having large correlations or partial correlations in high dimension. Recently there has been progress in the ultra-high dimensional setting when the sample size n is fixed and the dimension p tends to infinity. Despite these advances, the correlation screening framework suffers from practical, methodological and theoretical deficiencies. For instance, previous correlation screening theory requires that the population covariance matrix be sparse and block diagonal. This block sparsity assumption is however restrictive in practical applications. As a second example, correlation and partial correlation screening requires the estimation of dependence measures, which can be computationally prohibitive. In this paper, we propose a unifying approach to correlation and partial correlation mining that is not restricted to block diagonal correlation structure, thus yielding a methodology that is suitable for modern applications. By making connections to random geometric graphs, the number of highly correlated or partial correlated variables are shown to have compound Poisson finite-sample characterizations, which hold for both the finite p case and when p tends to infinity. The unifying framework also demonstrates a duality between correlation and partial correlation screening with theoretical and practical *** Codes 62H20, 62E17, 62H15 © 2021, CC BY-NC-ND.