检索结果-内蒙古大学图书馆

Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Yunshuai Qiao, Junbo Liao, Jincheng Li, Wei Li, Simiao Xie, Jiao Shen, Yunhang Hu, Jie Lin, Shaohui East China Normal University Shanghai China Huawei Noah’s Ark Lab China Xiamen University China Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE China

Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student’s learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones. Our codes are available at https://***/super-SSS/DCKD. Copyright © 2024, The Authors. All rights reserved.

关键词： Contrastive Learning

Semi-supervised inference for block-wise missing data without imputation

学校读者我要写书评

暂无评论

The Journal of Machine Learning Research

The Journal of Machine Learning Research 2024年第1期25卷 4902-4937页

作者： Shanshan Song Yuanyuan Lin Yong Zhou School of Mathematical Sciences and School of Economics and Management Tongji University Shanghai China Department of Statistics The Chinese University of Hong Kong Hong Kong China Key Laboratory of Advanced Theory and Application in Statistics and Data Science MOE Academy of Statistics and Interdisciplinary Sciences and School of Statistics East China Normal University Shanghai China

We consider statistical inference for single or low-dimensional parameters in a high-dimensional linear model under a semi-supervised setting, wherein the data are a combination of a labelled block-wise missing data set of a relatively small size and a large unlabelled data set. The proposed method utilises both labelled and unlabelled data without any imputation or removal of the missing observations. The asymptotic properties of the estimator are established under regularity conditions. Hypothesis testing for low-dimensional coefficients are also studied. Extensive simulations are conducted to examine the theoretical results. The method is evaluated on the Alzheimer's Disease Neuroimaging Initiative data.

关键词： block-missing data confidence intervals hypothesis testing semi-supervised inference

Distributed algorithms for u-statistics-based empirical risk minimization

学校读者我要写书评

暂无评论

The Journal of Machine Learning Research

The Journal of Machine Learning Research 2023年第1期24卷 12301-12343页

作者： Lanjue Chen Alan T.K. Wan Shuyi Zhang Yong Zhou Key Laboratory of Advanced Theory and Application in Statistics and Data Science Ministry of Education Academy of Statistics and Interdisciplinary Sciences and School of Statistics East China Normal University Shanghai China Department of Management Sciences School of Data Science and Department of Biostatistics City University of Hong Kong Kowloon Hong Kong

Empirical risk minimization, where the underlying loss function depends on a pair of data points, covers a wide range of application areas in statistics including pairwise ranking and survival analysis. The common empirical risk estimator obtained by averaging values of a loss function over all possible pairs of observations is essentially a U-statistic. One well-known problem with minimizing U-statistic type empirical risks, is that the computational complexity of U-statistics increases quadratically with the sample size. When faced with big data, this poses computational challenges as the colossal number of observation pairs virtually prohibits centralized computing to be performed on a single machine. This paper addresses this problem by developing two computationally and statistically efficient methods based on the divide-and-conquer strategy on a decentralized computing system, whereby the data are distributed among machines to perform the tasks. One of these methods is based on a surrogate of the empirical risk, while the other method extends the one-step updating scheme in classical M-estimation to the case of pairwise loss. We show that the proposed estimators are as asymptotically efficient as the benchmark global U-estimator obtained under centralized computing. As well, we introduce two distributed iterative algorithms to facilitate the implementation of the proposed methods, and conduct extensive numerical experiments to demonstrate their merit.

关键词：

A data-driven line search rule for support recovery in high-dimensional data analysis

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Li, Peili Jiao, Yuling Lu, Xiliang Kang, Lican School of Statistics Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE East China Normal University Shanghai200062 China School of Mathematics and Statistics Hubei Key Laboratory of Computational Science Wuhan University Wuhan430072 China School of Mathematics and Statistics Wuhan University Wuhan430072 China

In this work, we consider the algorithm to the (nonlinear) regression problems with 0penalty. The existing algorithms for 0based optimization problem are often carried out with a fixed step size, and the selection of an appropriate step size depends on the restricted strong convexity and smoothness for the loss function, hence it is difficult to compute in practical calculation. In sprite of the ideas of support detection and root finding Huang et al. (2021), we proposes a novel and efficient data-driven line search rule to adaptively determine the appropriate step size. We prove the 2error bound to the proposed algorithm without much restrictions for the cost functional. A large number of numerical comparisons with state-of-the-art algorithms in linear and logistic regression problems show the stability, effectiveness and superiority of the proposed algorithms. Copyright © 2021, The Authors. All rights reserved.

关键词： Machine learning

ID-Unet: Iterative soft and hard deformation for view synthesis

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Yin, Mingyu Sun, Li Li, Qingli Shanghai Key Laboratory of Multidimensional Information Processing Shanghi China Key Laboratory of Advanced Theory and Application in Statistics & Data Science East China Normal University Shanghai200241 China

View synthesis is usually done by an autoencoder, in which the encoder maps a source view image into a latent content code, and the decoder transforms it into a target view image according to the condition. However, the source contents are often not well kept in this setting, which leads to unnecessary changes during the view translation. Although adding skipped connections, like Unet, alleviates the problem, but it often causes the failure on the view conformity. This paper proposes a new architecture by performing the source-to-target deformation in an iterative way. Instead of simply incorporating the features from multiple layers of the encoder, we design soft and hard deformation modules, which warp the encoder features to the target view at different resolutions, and give results to the decoder to complement the details. Particularly, the current warping flow is not only used to align the feature of the same resolution, but also as an approximation to coarsely deform the high resolution feature. Then the residual flow is estimated and applied in the high resolution, so that the deformation is built up in the coarse-to-fine fashion. To better constrain the model, we synthesize a rough target view image based on the intermediate flows and their warped features. The extensive ablation studies and the final results on two different data sets show the effectiveness of the proposed model. https://***/MingyuY/Iterative-view-synthesis © 2021, CC BY.

关键词： Signal encoding

Two applications of Fourier Transform for Lévy Measures On Rp

学校读者我要写书评

暂无评论

SSRN

SSRN 2023年

作者： Qiu, Rui Li, Lu Yin, Xiangrong Yu, Zhou School of Statistics East China Normal University Shanghai200062 China Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE Shanghai200062 China Department of Statistics University of Kentucky LexingtonKY40536-0082 United States

For univariate variable x ∈ R, the Lévy measures for the two real-valued continuous negative definite functions ln and lncosh(x) are shown in Böttcher et al. (2018). In this note, by utilizing the Fourier transform, we present the corresponding Lévy measures for ln and lncosh(|x|) with x ∈ Rp. These Lévy measures have significant potential applications across various domains. © 2023, The Authors. All rights reserved.

关键词： Fourier transforms

Rethinking Centered Kernel Alignment in Knowledge Distillation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Zikai Shen, Yunhang Shao, Shitong Gong, Linrui Lin, Shaohui School of Computer Science and Technology East China Normal University Shanghai China Key Laboratory of Advanced Theory and Application in Statistics and Data Science MOE China Youtu Lab Tencent Shanghai China Hunan University China

Knowledge distillation has emerged as a highly effective method for bridging the representation discrepancy between large-scale models and lightweight models. Prevalent approaches involve leveraging appropriate metrics to minimize the divergence or distance between the knowledge extracted from the teacher model and the knowledge learned by the student model. Centered Kernel Alignment (CKA) is widely used to measure representation similarity and has been applied in several knowledge distillation methods. However, these methods are complex and fail to uncover the essence of CKA, thus not answering the question of how to use CKA to achieve simple and effective distillation properly. This paper first provides a theoretical perspective to illustrate the effectiveness of CKA, which decouples CKA to the upper bound of Maximum Mean Discrepancy (MMD) and a constant term. Drawing from this, we propose a novel Relation-Centered Kernel Alignment (RCKA) framework, which practically establishes a connection between CKA and MMD. Furthermore, we dynamically customize the application of CKA based on the characteristics of each task, with less computational source yet comparable performance than the previous methods. The extensive experiments on the CIFAR-100, ImageNet-1k, and MS-COCO demonstrate that our method achieves state-of-the-art performance on almost all teacher-student pairs for image classification and object detection, validating the effectiveness of our approaches. Our code is available in https://***/Klayand/PCKA. Copyright © 2024, The Authors. All rights reserved.

关键词： Distillation

Studentized permutation method for comparing restricted mean survival times with small sample from randomized trials

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Ditzhaus, Marc Yu, Menggang Xu, Jin Department of Statistics TU Dortmund University Germany Department of Biostatistics and Medical Informatics University of Wisconsin MadisonWI United States Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE and School of Statistics East China Normal University China

Recent observations, especially in cancer immunotherapy clinical trials with time-to-event outcomes, show that the commonly used proportional hazard assumption is often not justifiable, hampering an appropriate analyse of the data by hazard ratios. An attractive alternative advocated is given by the restricted mean survival time (RMST), which does not rely on any model assumption and can always be interpreted intuitively. As pointed out recently by Horiguchi and Uno (2020a), methods for the RMST based on asymptotic theory suffer from inflated type-I error under small sample sizes. To overcome this problem, they suggested a permutation strategy leading to more convincing results in simulations. However, their proposal requires an exchangeable data set-up between comparison groups which may be limiting in practice. In addition, it is not possible to invert their testing procedure to obtain valid confidence intervals, which can provide more in-depth information. In this paper, we address these limitations by proposing a studentized permutation test as well as the corresponding permutation-based confidence intervals. In our extensive simulation study, we demonstrate the advantage of our new method, especially in situations with relative small sample sizes and unbalanced groups. Finally we illustrate the application of the proposed method by re-analysing data from a recent lung cancer clinical trial. Copyright © 2021, The Authors. All rights reserved.

关键词： Diseases

Diagnostic Tests Before Modeling Longitudinal Actuarial data

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Li, Yinhuan Fung, Tsz Chai Peng, Liang Qian, Linyi Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE School of Statistics East China Normal University Shanghai200241 China Department of Risk Management and Insurance Georgia State University AtlantaGA30303 United States

In non-life insurance, it is essential to understand the serial dynamics and dependence structure of the longitudinal insurance data before using them. Existing actuarial literature primarily focuses on modeling, which typically assumes a lack of serial dynamics and a pre-specified dependence structure of claims across multiple years. To fill in the research gap, we develop two diagnostic tests, namely the serial dynamic test and correlation test, to assess the appropriateness of these assumptions and provide justifiable modeling directions. The tests involve the following ingredients: i) computing the change of the cross-sectional estimated parameters under a logistic regression model and the empirical residual correlations of the claim occurrence indicators across time, which serve as the indications to detect serial dynamics;ii) quantifying estimation uncertainty using the randomly weighted bootstrap approach;iii) developing asymptotic theories to construct proper test statistics. The proposed tests are examined by simulated data and applied to two non-life insurance datasets, revealing that the two datasets behave differently. Copyright © 2023, The Authors. All rights reserved.

关键词： Dynamics