检索结果-内蒙古大学图书馆

OD-DETR: Online Distillation for Stabilizing Training of Detection Transformer

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wu, Shengjian Sun, Li Li, Qingli Shanghai Key Laboratory of Multidimensional Information Processing East China Normal University China FinVolution Group Iran Key Laboratory of Advanced Theory and Application in Statistics and Data Science East China Normal University China

DEtection TRansformer (DETR) becomes a dominant paradigm, mainly due to its common architecture with high accuracy and no post-processing. However, DETR suffers from unstable training dynamics. It consumes more data and epochs to converge compared with CNN-based detectors. This paper aims to stabilize DETR training through the online distillation. It utilizes a teacher model, accumulated by Exponential Moving Average (EMA), and distills its knowledge into the online model in following three aspects. First, the matching relation between object queries and ground truth (GT) boxes in the teacher is employed to guide the student, so queries within the student are not only assigned labels based on their own predictions, but also refer to the matching results from the teacher. Second, the teacher’s initial query is given to the online student, and its prediction is directly constrained by the corresponding output from the teacher. Finally, the object queries from teacher’s different decoder stages are used to build the auxiliary group to accelerate the convergence. For each GT, two queries with the least matching costs are selected into this extra group, and they predict the GT box and participate the optimization. Extensive experiments show that the proposed OD-DETR successfully stabilizes the training, and significantly increases the performance without bringing in more parameters. Copyright © 2024, The Authors. All rights reserved.

关键词： Students

Unleashing the potential of prompt engineering for large language models

学校读者我要写书评

暂无评论

Patterns

Patterns 2025年

作者： Chen, Banghao Zhang, Zhaofeng Langrené, Nicolas Zhu, Shengxin Advanced Institute of Natural Science Beijing Normal University No.18 Jingfeng Road Guangdong Zhuhai519087 China Guangdong Provincial/Zhuhai Key Laboratory of Interdisciplinary Research and Application for Data Science Beijing Normal-Hong Kong Baptist University Guangdong Zhuhai519087 China Department of Statistics and Data Science Beijing Normal-Hong Kong Baptist University Guangdong Zhuhai519087 China Department of Mathematical Science Beijing Normal-Hong Kong Baptist University Guangdong Zhuhai519087 China Department of Biomedical Informatics Yong Loo Lin School of Medicine National University of Singapore Singapore119228 Singapore Department of Mathematics University of Michigan Ann ArborMI48109 United States

This review explores the role of prompt engineering in unleashing the capabilities of large language models (LLMs). Prompt engineering is the process of structuring inputs, and it has emerged as a crucial technique for maximizing the utility and accuracy of these models. Both foundational and advanced prompt engineering methodologies—including techniques such as self-consistency, chain of thought, and generated knowledge, which can significantly enhance the performance of models—are explored in this paper. Additionally, the prompt methods for vision language models (VLMs) are examined in detail. Prompt methods are evaluated with subjective and objective metrics, ensuring a robust analysis of their efficacy. Critical to this discussion is the role of prompt engineering in artificial intelligence (AI) security, particularly in terms of defending against adversarial attacks that exploit vulnerabilities in LLMs. Strategies for minimizing these risks and improving the robustness of models are thoroughly reviewed. Finally, we provide a perspective for future research and applications. © 2025 The Author(s)

关键词： Visualization

A hybrid deep learning method for controlled stochastic Kolmogorov systems with regime-switching

学校读者我要写书评

暂无评论

A hybrid deep learning method for controlled stochastic Kolm...

International Conference on Control, Decision and Information Technologies (CoDIT)

作者： Yu Zhang Zhuo Jin Jiaqin Wei School of Statistics Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE East China Normal University Shanghai China Department of Actuarial Studies and Business Analytics Macquarie University NSW Australia

ISBN: (数字)9798350373974

ISBN: (纸本)9798350373981

In this paper, we employ numerical methods based on deep learning algorithms for solving controlled stochastic Kolmogorov systems with regime-switching. Different from classical control problems, each component of the state in controlled Kolmogorov systems is nonnegative. Due to the nonlinearity and complexity of the controlled stochastic Kolmogorov systems, we develop a hybrid deep learning method to numerically solve the optimal controls under this system. Subsequently, we apply the hybrid deep learning method to solve a specific case of a controlled stochastic Kolmogorov system, specifically controlled SIS (susceptible-infected-susceptible) systems. Finally, the effectiveness of the proposed hybrid deep learning method is verified through numerical results.

关键词： Deep learning Closed-form solutions Neural networks Optimal control Aerospace electronics Control systems Complexity theory Approximation methods Information technology Method of moments

Semi-supervised inference for block-wise missing data without imputation

学校读者我要写书评

暂无评论

The Journal of Machine Learning Research

The Journal of Machine Learning Research 2024年第1期25卷 4902-4937页

作者： Shanshan Song Yuanyuan Lin Yong Zhou School of Mathematical Sciences and School of Economics and Management Tongji University Shanghai China Department of Statistics The Chinese University of Hong Kong Hong Kong China Key Laboratory of Advanced Theory and Application in Statistics and Data Science MOE Academy of Statistics and Interdisciplinary Sciences and School of Statistics East China Normal University Shanghai China

We consider statistical inference for single or low-dimensional parameters in a high-dimensional linear model under a semi-supervised setting, wherein the data are a combination of a labelled block-wise missing data set of a relatively small size and a large unlabelled data set. The proposed method utilises both labelled and unlabelled data without any imputation or removal of the missing observations. The asymptotic properties of the estimator are established under regularity conditions. Hypothesis testing for low-dimensional coefficients are also studied. Extensive simulations are conducted to examine the theoretical results. The method is evaluated on the Alzheimer's Disease Neuroimaging Initiative data.

关键词： block-missing data confidence intervals hypothesis testing semi-supervised inference

DYNAMIC-LLAVA: EFFICIENT MULTIMODAL LARGE LANGUAGE MODELS VIA DYNAMIC VISION-LANGUAGE CONTEXT SPARSIFICATION

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Huang, Wenxuan Zhai, Zijie Shen, Yunhang Cao, Shaosheng Zhao, Fei Xu, Xiangfeng Ye, Zheyu Hu, Yao Lin, Shaohui East China Normal University China Xiamen University China Xiaohongshu Inc China Nanjing University China Key Laboratory of Advanced Theory and Application in Statistics and Data Science MOE China

Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding, directly affecting the efficacy of MLLMs. Existing methods attempt to reduce the vision context redundancy to achieve efficient MLLMs. Unfortunately, the efficiency benefits of the vision context reduction in the prefill stage gradually diminish during the decoding stage. To address this problem, we proposed a dynamic vision-language context sparsification framework Dynamic-LLaVA, which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLaVA designs a tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice, Dynamic-LLaVA can reduce computation consumption by ∼75% in the prefill stage. Meanwhile, throughout the entire generation process of MLLMs, Dynamic-LLaVA reduces the ∼50% computation consumption under decoding without KV cache, while saving ∼50% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaVA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation or even performance gains compared to the full-context inference baselines. Code is available at https://***/Osilly/dynamic_llava . Copyright © 2024, The Authors. All rights reserved.

关键词： Decoding

A data-driven line search rule for support recovery in high-dimensional data analysis

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Li, Peili Jiao, Yuling Lu, Xiliang Kang, Lican School of Statistics Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE East China Normal University Shanghai200062 China School of Mathematics and Statistics Hubei Key Laboratory of Computational Science Wuhan University Wuhan430072 China School of Mathematics and Statistics Wuhan University Wuhan430072 China

In this work, we consider the algorithm to the (nonlinear) regression problems with 0penalty. The existing algorithms for 0based optimization problem are often carried out with a fixed step size, and the selection of an appropriate step size depends on the restricted strong convexity and smoothness for the loss function, hence it is difficult to compute in practical calculation. In sprite of the ideas of support detection and root finding Huang et al. (2021), we proposes a novel and efficient data-driven line search rule to adaptively determine the appropriate step size. We prove the 2error bound to the proposed algorithm without much restrictions for the cost functional. A large number of numerical comparisons with state-of-the-art algorithms in linear and logistic regression problems show the stability, effectiveness and superiority of the proposed algorithms. Copyright © 2021, The Authors. All rights reserved.

关键词： Machine learning

Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Yunshuai Qiao, Junbo Liao, Jincheng Li, Wei Li, Simiao Xie, Jiao Shen, Yunhang Hu, Jie Lin, Shaohui East China Normal University Shanghai China Huawei Noah’s Ark Lab China Xiamen University China Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE China

Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student’s learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones. Our codes are available at https://***/super-SSS/DCKD. Copyright © 2024, The Authors. All rights reserved.

关键词： Contrastive Learning

Two applications of Fourier Transform for Lévy Measures On Rp

学校读者我要写书评

暂无评论

SSRN

SSRN 2023年

作者： Qiu, Rui Li, Lu Yin, Xiangrong Yu, Zhou School of Statistics East China Normal University Shanghai200062 China Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE Shanghai200062 China Department of Statistics University of Kentucky LexingtonKY40536-0082 United States

For univariate variable x ∈ R, the Lévy measures for the two real-valued continuous negative definite functions ln and lncosh(x) are shown in Böttcher et al. (2018). In this note, by utilizing the Fourier transform, we present the corresponding Lévy measures for ln and lncosh(|x|) with x ∈ Rp. These Lévy measures have significant potential applications across various domains. © 2023, The Authors. All rights reserved.

关键词： Fourier transforms

Studentized permutation method for comparing restricted mean survival times with small sample from randomized trials

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Ditzhaus, Marc Yu, Menggang Xu, Jin Department of Statistics TU Dortmund University Germany Department of Biostatistics and Medical Informatics University of Wisconsin MadisonWI United States Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE and School of Statistics East China Normal University China

Recent observations, especially in cancer immunotherapy clinical trials with time-to-event outcomes, show that the commonly used proportional hazard assumption is often not justifiable, hampering an appropriate analyse of the data by hazard ratios. An attractive alternative advocated is given by the restricted mean survival time (RMST), which does not rely on any model assumption and can always be interpreted intuitively. As pointed out recently by Horiguchi and Uno (2020a), methods for the RMST based on asymptotic theory suffer from inflated type-I error under small sample sizes. To overcome this problem, they suggested a permutation strategy leading to more convincing results in simulations. However, their proposal requires an exchangeable data set-up between comparison groups which may be limiting in practice. In addition, it is not possible to invert their testing procedure to obtain valid confidence intervals, which can provide more in-depth information. In this paper, we address these limitations by proposing a studentized permutation test as well as the corresponding permutation-based confidence intervals. In our extensive simulation study, we demonstrate the advantage of our new method, especially in situations with relative small sample sizes and unbalanced groups. Finally we illustrate the application of the proposed method by re-analysing data from a recent lung cancer clinical trial. Copyright © 2021, The Authors. All rights reserved.

关键词： Diseases