检索结果-内蒙古大学图书馆

Learning from rich semantics and coarse locations for long-tailed object detection 23

学校读者我要写书评

暂无评论

Learning from rich semantics and coarse locations for long-t...

Proceedings of the 37th International Conference on Neural Information Processing Systems

作者： Lingchen Meng Xiyang Dai Jianwei Yang Dongdong Chen Yinpeng Chen Mengchen Liu Yi-Ling Chen Zuxuan Wu Lu Yuan Yu-Gang Jiang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University and Shanghai Collaborative Innovation Center of Intelligent Visual Computing Microsoft

Long-tailed object detection (LTOD) aims to handle the extreme data imbalance in real-world datasets, where many tail classes have scarce instances. One popular strategy is to explore extra data with image-level labels, yet it produces limited results due to (1) semantic ambiguity—an image-level label only captures a salient part of the image, ignoring the remaining rich semantics within the image; and (2) location sensitivity—the label highly depends on the locations and crops of the original image, which may change after data transformations like random cropping. To remedy this, we propose RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes. RichSem leverages rich semantics from images, which are then served as additional "soft supervision" for training detectors. Specifically, we add a semantic branch to our detector to learn these soft semantics and enhance feature representations for long-tailed object detection. The semantic branch is only used for training and is removed during inference. RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors. Our method achieves state-of-the-art performance without requiring complex training and testing procedures. Moreover, we show the effectiveness of our method on other long-tailed datasets with additional experiments. Code is available at https://***/MengLcool/RichSem.

关键词：

Single-Image Shadow Removal Using Deep Learning: A Comprehensive Survey

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Guo, Lanqing Wang, Chong Wang, Yufei Yu, Yi Huang, Siyu Yang, Wenhan Kot, Alex C. Wen, Bihan ROSE Lab Nanyang Technological University 50 Nanyang Ave 639798 Singapore Visual Computing Division School of Computing Clemson University ClemsonSC29631 United States Pengcheng Laboratory No. 2 Xingke 1st Street Shenzhen518066 China

Shadow removal aims at restoring the image content within shadow regions, pursuing a uniform distribution of illumination that is consistent between shadow and non-shadow regions. Comparing to other image restoration tasks, there are two unique challenges in shadow removal: 1) The patterns of shadows are arbitrary, varied, and often have highly complex trace structures, making "trace-less" image recovery difficult. 2) The degradation caused by shadows is spatially non-uniform, resulting in inconsistencies in illumination and color between shadow and non-shadow areas. Recent developments in this field are primarily driven by deep learning-based solutions, employing a variety of learning strategies, network architectures, loss functions, and training data. Nevertheless, a thorough and insightful review of deep learning-based shadow removal techniques is still lacking. In this paper, we are the first to provide a comprehensive survey to cover various aspects ranging from technical details to applications. We highlight the major advancements in deep learning-based single-image shadow removal methods, thoroughly review previous research across various categories, and provide insights into the historical progression of these developments. Additionally, we summarize performance comparisons both quantitatively and qualitatively. Beyond the technical aspects of shadow removal methods, we also explore potential future directions for this field. The related repository is released at: https://***/GuoLanqing/Awesome-Shadow-Removal. Copyright © 2024, The Authors. All rights reserved.

关键词： Deep learning

DeepStack: Deeply Stacking visual Tokens is Surprisingly Simple and Effective for LMMs

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Meng, Lingchen Yang, Jianwei Tian, Rui Dai, Xiyang Wu, Zuxuan Gao, Jianfeng Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China Microsoft Corporation United States

Most large multimodal models (LMMs) are implemented by feeding visual tokens as a sequence into the first layer of a large language model (LLM). The resulting architecture is simple but significantly increases computation and memory costs, as it has to handle a large number of additional tokens in its input layer. This paper presents a new architecture DeepStack for LMMs. Considering N layers in the language and vision transformer of LMMs, we stack the visual tokens into N groups and feed each group to its aligned transformer layer from bottom to top, as illustrated in Fig. 1. Surprisingly, this simple method greatly enhances the power of LMMs to model interactions among visual tokens across layers but with minimal additional cost. We apply DeepStack to both language and vision transformer in LMMs, and validate the effectiveness of DeepStack LMMs with extensive empirical results. Using the same context length, our DeepStack 7B and 13B parameters surpass their counterparts by 2.7 and 2.9 on average across 9 benchmarks, respectively. Using only one-fifth of the context length, DeepStack rivals closely to the counterparts Copyright © 2024, The Authors. All rights reserved.

关键词： visual languages

Unsigned Orthogonal Distance Fields: An Accurate Neural Implicit Representation for Diverse 3D Shapes

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Lu, Yujie Wan, Long Ding, Nayu Wang, Yulong Shen, Shuhan Cai, Shen Gao, Lin Visual and Geometric Perception Lab Donghua University China Institute of Automation Chinese Academy of Sciences China Institute of Computing Technology Chinese Academy of Sciences China University of Chinese Academy of Sciences China

Neural implicit representation of geometric shapes has witnessed considerable advancements in recent years. However, common distance field based implicit representations, specifically signed distance field (SDF) for watertight shapes or unsigned distance field (UDF) for arbitrary shapes, routinely suffer from degradation of reconstruction accuracy when converting to explicit surface points and meshes. In this paper, we introduce a novel neural implicit representation based on unsigned orthogonal distance fields (UODFs). In UODFs, the minimal unsigned distance from any spatial point to the shape surface is defined solely in one orthogonal direction, contrasting with the multi-directional determination made by SDF and UDF. Consequently, every point in the 3D UODFs can directly access its closest surface points along three orthogonal directions. This distinctive feature leverages the accurate reconstruction of surface points without interpolation errors. We verify the effectiveness of UODFs through a range of reconstruction examples, extending from simple watertight or non-watertight shapes to complex shapes that include hollows, internal or assembling structures. Copyright © 2024, The Authors. All rights reserved.

关键词：

Artwork Identification in a Museum Environment: A Quantitative Evaluation of Factors Affecting Identification Accuracy 8th

学校读者我要写书评

暂无评论

Artwork Identification in a Museum Environment: A Quantitati...

8th European-Mediterranean Conference, EuroMed 2020

作者： Lanitis, A. Theodosiou, Z. Partaourides, H. Visual Media Computing Lab Department of Multimedia and Graphic Arts Cyprus University of Technology Limassol Limassol Cyprus CYENS Centre of Excellence Nicosia Cyprus

ISBN: (纸本)9783030730420

The ability to identify the artworks that a museum visitor is looking at, using first-person images seamlessly captured by wearable cameras can be used as a means for invoking applications that provide information about the exhibits, and provide information about visitors’ activities. As part of our efforts to optimize the artwork recognition accuracy of an artwork identification system under development, an investigation aiming to determine the effect of different conditions on the artwork recognition accuracy in a gallery/exhibition environment is presented. Through the controlled introduction of different distractors in a virtual museum environment, it is feasible to assess the effect on the recognition performance of different conditions. The results of the experiment are important for improving the robustness of artwork recognition systems, and at the same time the conclusions of this work can provide specific guidelines to curators, museum professionals and visitors, that will enable the efficient identification of artworks, using images captured with wearable cameras in a museum environment. © 2021, Springer Nature Switzerland AG.

关键词： Computer vision

Pix2Cap-COCO: Advancing visual Comprehension via Pixel-Level Captioning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： You, Zuyao Wang, Junke Kong, Lingyu He, Bo Wu, Zuxuan Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China University of Maryland College Park United States

(Image Present) We present Pix2Cap-COCO, the first panoptic pixel-level caption dataset designed to advance fine-grained visual understanding. To achieve this, we carefully design an automated annotation pipeline that prompts GPT-4V to generate pixel-aligned, instance-specific captions for individual objects within images, enabling models to learn more granular relationships between objects and their contexts. This approach results in 167,254 detailed captions, with an average of 22.94 words per caption. Building on Pix2Cap-COCO, we introduce a novel task, panoptic segmentation-captioning, which challenges models to recognize instances in an image and provide detailed descriptions for each simultaneously. To benchmark this task, we design a robust baseline based on X-Decoder. The experimental results demonstrate that Pix2Cap-COCO is a particularly challenging dataset, as it requires models to excel in both fine-grained visual understanding and detailed language generation. Furthermore, we leverage Pix2Cap-COCO for Supervised Fine-Tuning (SFT) on large multimodal models (LMMs) to enhance their performance. For example, training with Pix2Cap-COCO significantly improves the performance of GPT4RoI, yielding gains in CIDEr (+1.4%), ROUGE (+0.4%), and SPICE (+0.5%) on visual Genome dataset, and strengthens its region understanding ability on the ViP-Bench, with an overall improvement of +5.1%, including notable increases in recognition accuracy (+11.2%) and language generation quality (+22.2%). Code is available at https://***/geshang777/pix2cap. © 2025, CC BY-NC-SA.

关键词： Pixels

Multi-prompt alignment for multi-source unsupervised domain adaptation 23

学校读者我要写书评

暂无评论

Multi-prompt alignment for multi-source unsupervised domain ...

Proceedings of the 37th International Conference on Neural Information Processing Systems

作者： Haoran Chen Xintong Han Zuxuan Wu Yu-Gang Jiang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University and Shanghai Collaborative Innovation Center of Intelligent Visual Computing Huya Inc

Most existing methods for unsupervised domain adaptation (UDA) rely on a shared network to extract domain-invariant features. However, when facing multiple source domains, optimizing such a network involves updating the parameters of the entire network, making it both computationally expensive and challenging, particularly when coupled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity models for downstream tasks in a computationally economic way, we introduce Multi-Prompt Alignment (MPA), a simple yet efficient framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss. Then, MPA denoises the learned prompts through an auto-encoding process and aligns them by maximizing the agreement of all the reconstructed prompts. Moreover, we show that the resulting subspace acquired from the auto-encoding process can easily generalize to a streamlined set of target domains, making our method more efficient for practical usage. Extensive experiments show that MPA achieves state-of-the-art results on three popular datasets with an impressive average accuracy of 54.1% on DomainNet.

关键词：

Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Yian Tian, Wentao Jiao, Yang Chen, Jingjing Zhao, Na Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center on Intelligent Visual Computing China Singapore University of Technology and Design Singapore

Recently, Multimodal Large Language Models (MLLMs) have achieved significant success across multiple disciplines due to their exceptional instruction-following capabilities and extensive world knowledge. However, whether these MLLMs possess human-like compositional reasoning abilities remains an open problem. To unveil their reasoning behaviors, we first curate a Multimodal Assumptive Reasoning Benchmark (MARS-Bench) in this paper. Interestingly, we find that most prevalent MLLMs can be easily fooled by the introduction of a presupposition into the question, whereas such presuppositions appear naive to human reasoning. Besides, we also propose a simple yet effective method, Active Deduction (AD), to encourage the model to actively perform composite deduction before reaching a final decision. Equipped with the proposed AD method, a MLLM demonstrates significant improvements in assumptive reasoning abilities without compromising its general-purpose question-answering performance. We also provide extensive evaluations of both open-source and private MLLMs on MARS-Bench, along with experimental analyses of the AD method. Copyright © 2024, The Authors. All rights reserved.

关键词： Adversarial machine learning

ADAPTIVE RETENTION & CORRECTION: TEST-TIME TRAINING FOR CONTINUAL LEARNING

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chen, Haoran Goldblum, Micah Wu, Zuxuan Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China New York University United States

Continual learning, also known as lifelong learning or incremental learning, refers to the process by which a model learns from a stream of incoming data over time. A common problem in continual learning is the classification layer’s bias towards the most recent task. Traditionally, methods have relied on incorporating data from past tasks during training to mitigate this problem. However, the recent shift in continual learning to memory-free environments has rendered these approaches infeasible. In this study, we propose a solution focused on the testing phase. We first introduce a simple Out-of-Task Detection method, OTD, designed to accurately identify samples from past tasks during testing. Using OTD, we then propose: (1) an Adaptive Retention mechanism for dynamically tuning the classifier layer on past task data;(2) an Adaptive Correction mechanism for revising predictions when the model classifies the data from previous tasks into classes of the current task. We name our approach Adaptive Retention & Correction (ARC). Although designed for memory-free environments, ARC also proves effective in memory-based settings. Extensive experiments show that our proposed method can be plugged in to virtually any existing continual learning approach without requiring any modifications to its training procedure. Specifically, when integrated with state-of-the-art approaches, ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets, respectively. Code is available at Github Link. © 2024, CC BY.

关键词： Contrastive Learning