检索结果-内蒙古大学图书馆

CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Li, Siyuan Sun, Li Li, Qingli Shanghai Key Laboratory of Multidimensional Information Processing China Key Laboratory of Advanced Theory and Application in Statistics and Data Science East China Normal University Shanghai China

Pre-trained vision-language models like CLIP have recently shown superior performances on various downstream tasks, including image classification and segmentation. However, in fine-grained image re-identification (ReID), the labels are indexes, lacking concrete text descriptions. Therefore, it remains to be determined how such models could be applied to these tasks. This paper first finds out that simply fine-tuning the visual model initialized by the image encoder in CLIP, has already obtained competitive performances in various ReID tasks. Then we propose a two-stage strategy to facilitate a better visual representation. The key idea is to fully exploit the cross-modal description ability in CLIP through a set of learnable text tokens for each ID and give them to the text encoder to form ambiguous descriptions. In the first training stage, image and text encoders from CLIP keep fixed, and only the text tokens are optimized from scratch by the contrastive loss computed within a batch. In the second stage, the ID-specific text tokens and their encoder become static, providing constraints for fine-tuning the image encoder. With the help of the designed loss in the downstream task, the image encoder is able to represent data as vectors in the feature embedding accurately. The effectiveness of the proposed strategy is validated on several datasets for the person or vehicle ReID tasks. Code is available at https://***/Syliz517/CLIP-ReID. Copyright © 2022, The Authors. All rights reserved.

关键词： Image segmentation

Disentangling the spatial structure and style in conditional vae

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Zhang, Ziye Sun, Li Zheng, Zhilin Li, Qingli Shanghai Key Laboratory of Multidimensional Information Processing Key Laboratory of Advanced Theory and Application in Statistics and Data Science East China Normal University Shanghai200241 China

This paper aims to disentangle the latent space in cVAE into the spatial structure and the style code, which are complementary to each other, with one of them zs being label relevant and the other zu irrelevant. The generator is built by a connected encoder-decoder and a label condition mapping network. Depending on whether the label is related with the spatial structure, the output zs from the condition mapping network is used either as a style code or a spatial structure code. The encoder provides the label irrelevant posterior from which zu is sampled. The decoder employs zs and zu in each layer by adaptive normalization like SPADE or AdaIN. Extensive experiments on two datasets with different types of labels show the effectiveness of our method. Copyright © 2019, The Authors. All rights reserved.

关键词： Mapping

DYNAMIC-LLAVA: EFFICIENT MULTIMODAL LARGE LANGUAGE MODELS VIA DYNAMIC VISION-LANGUAGE CONTEXT SPARSIFICATION

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Huang, Wenxuan Zhai, Zijie Shen, Yunhang Cao, Shaosheng Zhao, Fei Xu, Xiangfeng Ye, Zheyu Hu, Yao Lin, Shaohui East China Normal University China Xiamen University China Xiaohongshu Inc China Nanjing University China Key Laboratory of Advanced Theory and Application in Statistics and Data Science MOE China

Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding, directly affecting the efficacy of MLLMs. Existing methods attempt to reduce the vision context redundancy to achieve efficient MLLMs. Unfortunately, the efficiency benefits of the vision context reduction in the prefill stage gradually diminish during the decoding stage. To address this problem, we proposed a dynamic vision-language context sparsification framework Dynamic-LLaVA, which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLaVA designs a tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice, Dynamic-LLaVA can reduce computation consumption by ∼75% in the prefill stage. Meanwhile, throughout the entire generation process of MLLMs, Dynamic-LLaVA reduces the ∼50% computation consumption under decoding without KV cache, while saving ∼50% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaVA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation or even performance gains compared to the full-context inference baselines. Code is available at https://***/Osilly/dynamic_llava . Copyright © 2024, The Authors. All rights reserved.

关键词： Decoding

QS-Attn: Query-Selected Attention for Contrastive Learning in I2I Translation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Hu, Xueqi Zhou, Xinyue Huang, Qiusheng Shi, Zhengyi Sun, Li Li, Qingli Shanghai Key Laboratory of Multidimensional Information Processing Shanghai China Key Laboratory of Advanced Theory and Application in Statistics and Data Science East China Normal University Shanghai China

Unpaired image-to-image (I2I) translation often requires to maximize the mutual information between the source and the translated images across different domains, which is critical for the generator to keep the source content and prevent it from unnecessary modifications. The self-supervised contrastive learning has already been successfully applied in the I2I. By constraining features from the same location to be closer than those from different ones, it implicitly ensures the result to take content from the source. However, previous work uses the features from random locations to impose the constraint, which may not be appropriate since some locations contain less information of source domain. Moreover, the feature itself does not reflect the relation with others. This paper deals with these problems by intentionally selecting significant anchor points for contrastive learning. We design a query-selected attention (QS-Attn) module, which compares feature distances in the source domain, giving an attention matrix with a probability distribution in each row. Then we select queries according to their measurement of significance, computed from the distribution. The selected ones are regarded as anchors for contrastive loss. At the same time, the reduced attention matrix is employed to route features in both domains, so that source relations maintain in the synthesis. We validate our proposed method in three different I2I datasets, showing that it increases the image quality without adding learnable parameters. Codes are available at https://***/sapphire497/query-selected-attention. Copyright © 2022, The Authors. All rights reserved.

关键词： Location

Novel View Synthesis on Unpaired data by Conditional Deformable Variational Auto-Encoder

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Yin, Mingyu Sun, Li Li, Qingli Shanghai Key Laboratory of Multidimensional Information Processing China Key Laboratory of Advanced Theory and Application in Statistics & Data Science East China Normal University Shanghai200241 China

Novel view synthesis often needs the paired data from both the source and target views. This paper proposes a view translation model under cVAE-GAN framework without requiring the paired data. We design a conditional deformable module (CDM) which uses the view condition vectors as the filters to convolve the feature maps of the main branch in VAE. It generates several pairs of displacement maps to deform the features, like the 2D optical flows. The results are fed into the deformed feature based normalization module (DFNM), which scales and offsets the main branch feature, given its deformed one as the input from the side branch. Taking the advantage of the CDM and DFNM, the encoder outputs a view-irrelevant posterior, while the decoder takes the code drawn from it to synthesize the reconstructed and the view-translated images. To further ensure the disentanglement between the views and other factors, we add adversarial training on the code. The results and ablation studies on MultiPIE and 3D chair datasets validate the effectiveness of the framework in cVAE and the designed module. https://***/MingyuY/deformable-view-synthesis Copyright © 2020, The Authors. All rights reserved.

关键词： Signal encoding

Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Yunshuai Qiao, Junbo Liao, Jincheng Li, Wei Li, Simiao Xie, Jiao Shen, Yunhang Hu, Jie Lin, Shaohui East China Normal University Shanghai China Huawei Noah’s Ark Lab China Xiamen University China Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE China

Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student’s learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones. Our codes are available at https://***/super-SSS/DCKD. Copyright © 2024, The Authors. All rights reserved.

关键词： Contrastive Learning

Semi-supervised inference for block-wise missing data without imputation

学校读者我要写书评

暂无评论

The Journal of Machine Learning Research

The Journal of Machine Learning Research 2024年第1期25卷 4902-4937页

作者： Shanshan Song Yuanyuan Lin Yong Zhou School of Mathematical Sciences and School of Economics and Management Tongji University Shanghai China Department of Statistics The Chinese University of Hong Kong Hong Kong China Key Laboratory of Advanced Theory and Application in Statistics and Data Science MOE Academy of Statistics and Interdisciplinary Sciences and School of Statistics East China Normal University Shanghai China

We consider statistical inference for single or low-dimensional parameters in a high-dimensional linear model under a semi-supervised setting, wherein the data are a combination of a labelled block-wise missing data set of a relatively small size and a large unlabelled data set. The proposed method utilises both labelled and unlabelled data without any imputation or removal of the missing observations. The asymptotic properties of the estimator are established under regularity conditions. Hypothesis testing for low-dimensional coefficients are also studied. Extensive simulations are conducted to examine the theoretical results. The method is evaluated on the Alzheimer's Disease Neuroimaging Initiative data.

关键词： block-missing data confidence intervals hypothesis testing semi-supervised inference

Distributed algorithms for u-statistics-based empirical risk minimization

学校读者我要写书评

暂无评论

The Journal of Machine Learning Research

The Journal of Machine Learning Research 2023年第1期24卷 12301-12343页

作者： Lanjue Chen Alan T.K. Wan Shuyi Zhang Yong Zhou Key Laboratory of Advanced Theory and Application in Statistics and Data Science Ministry of Education Academy of Statistics and Interdisciplinary Sciences and School of Statistics East China Normal University Shanghai China Department of Management Sciences School of Data Science and Department of Biostatistics City University of Hong Kong Kowloon Hong Kong

Empirical risk minimization, where the underlying loss function depends on a pair of data points, covers a wide range of application areas in statistics including pairwise ranking and survival analysis. The common empirical risk estimator obtained by averaging values of a loss function over all possible pairs of observations is essentially a U-statistic. One well-known problem with minimizing U-statistic type empirical risks, is that the computational complexity of U-statistics increases quadratically with the sample size. When faced with big data, this poses computational challenges as the colossal number of observation pairs virtually prohibits centralized computing to be performed on a single machine. This paper addresses this problem by developing two computationally and statistically efficient methods based on the divide-and-conquer strategy on a decentralized computing system, whereby the data are distributed among machines to perform the tasks. One of these methods is based on a surrogate of the empirical risk, while the other method extends the one-step updating scheme in classical M-estimation to the case of pairwise loss. We show that the proposed estimators are as asymptotically efficient as the benchmark global U-estimator obtained under centralized computing. As well, we introduce two distributed iterative algorithms to facilitate the implementation of the proposed methods, and conduct extensive numerical experiments to demonstrate their merit.

关键词：

A data-driven line search rule for support recovery in high-dimensional data analysis

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Li, Peili Jiao, Yuling Lu, Xiliang Kang, Lican School of Statistics Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE East China Normal University Shanghai200062 China School of Mathematics and Statistics Hubei Key Laboratory of Computational Science Wuhan University Wuhan430072 China School of Mathematics and Statistics Wuhan University Wuhan430072 China

In this work, we consider the algorithm to the (nonlinear) regression problems with 0penalty. The existing algorithms for 0based optimization problem are often carried out with a fixed step size, and the selection of an appropriate step size depends on the restricted strong convexity and smoothness for the loss function, hence it is difficult to compute in practical calculation. In sprite of the ideas of support detection and root finding Huang et al. (2021), we proposes a novel and efficient data-driven line search rule to adaptively determine the appropriate step size. We prove the 2error bound to the proposed algorithm without much restrictions for the cost functional. A large number of numerical comparisons with state-of-the-art algorithms in linear and logistic regression problems show the stability, effectiveness and superiority of the proposed algorithms. Copyright © 2021, The Authors. All rights reserved.

关键词： Machine learning