检索结果-内蒙古大学图书馆

SegIC: Unleashing the Emergent Correspondence for In-Context Segmentation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Meng, Lingchen Lan, Shiyi Li, Hengduo Alvarez, Jose M. Wu, Zuxuan Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China NVIDIA United States University of Maryland United States

In-context segmentation aims at segmenting novel images using a few labeled example images, termed as "in-context examples", exploring content similarities between examples and the target. The resulting models can be generalized seamlessly to novel segmentation tasks, significantly reducing the labeling and training costs compared with conventional pipelines. However, in-context segmentation is more challenging than classic ones requiring the model to learn segmentation rules conditioned on a few samples. Unlike previous work with ad-hoc or non-end-to-end designs, we propose SegIC, an end-to-end segment-in-context framework built upon a single vision foundation model (VFM). In particular, SegIC leverages the emergent correspondence within VFM to capture dense relationships between target images and in-context samples. As such, information from in-context samples is then extracted into three types of instructions, i.e. geometric, visual, and meta instructions, serving as explicit conditions for the final mask prediction. SegIC is a straightforward yet effective approach that yields state-of-the-art performance on one-shot segmentation benchmarks. Notably, SegIC can be easily generalized to diverse tasks, including video object segmentation and open-vocabulary segmentation. Code will be available at https://***/MengLcool/SEGIC. Copyright © 2023, The Authors. All rights reserved.

关键词： Image segmentation

EventHallusion: Diagnosing Event Hallucinations in Video LLMs

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Jiacheng Jiao, Yang Chen, Shaoxiang Zhao, Na Tan, Zhiyu Li, Hao Chen, Jingjing Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center on Intelligent Visual Computing China Singapore University of Technology and Design Singapore Shanghai Academy of Artificial Intelligence for Science China Meituan China

Recently, Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field. Despite remarkable content reasoning and instruction following capabilities they demonstrated, the hallucination problem of these VideoLLMs is less explored compared with its counterpart in the image domain. To mitigate this gap, we propose EventHallusion, a novel benchmark that focuses on assessing the VideoLLMs’ hallucination toward event, the crux of video analysis. From a hallucination attribution perspective, our EventHallusion benchmark is curated to assess a VideoLLM’s susceptibility toward language priors and vision-language biases. On the other hand, we also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs. The proposed TCD method rectifies the model’s bias toward its priors during the decoding stage by comparing the original video with a modified version, in which temporal cues are ambiguited. Through comprehensive evaluation of eight open-source and three closed-source VideoLLMs on the proposed EventHallusion benchmark, we observe that the open-source models suffer significantly from hallucination problems, whereas the closed-source ones perform markedly better. By further equipping open-source VideoLLMs with the proposed TCD approach, evident performance improvements are achieved across most metrics in the EventHallusion benchmark. Copyright © 2024, The Authors. All rights reserved.

关键词： Decoding

OmniViD: A Generative Framework for Universal Video Understanding

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Junke Chen, Dongdong Luo, Chong He, Bo Yuan, Lu Wu, Zuxuan Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China Microsoft Cloud + AI Microsoft Research Asia China University of Maryland College Park United States

The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution. Despite sharing a common goal, different tasks often rely on distinct model architectures and annotation formats. In contrast, natural language processing benefits from a unified output space, i.e., text sequences, which simplifies the training of powerful foundational language models, such as GPT-3, with extensive training corpora. Inspired by this, we seek to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens. In this way, a variety of video tasks could be formulated as video-grounded token generation. This enables us to address various types of video tasks, including classification (such as action recognition), captioning (covering clip captioning, video question answering, and dense video captioning), and localization tasks (such as visual object tracking) within a fully shared encoder-decoder architecture, following a generative framework. Through comprehensive experiments, we demonstrate such a simple and straightforward idea is quite effective and can achieve state-of-the-art or competitive results on seven video benchmarks, providing a novel perspective for more universal video understanding. Code is available at https://***/wangjk666/OmniVid. Copyright © 2024, The Authors. All rights reserved.

关键词： Object detection

CUDA and Applications to Task-based Programming

学校读者我要写书评

暂无评论

CUDA and Applications to Task-based Programming

42nd Annual Conference on European Association for Computer Graphics, EUROGRAPHICS 2021

作者： Kenzel, M. Kerbl, B. Winter, M. Steinberger, M. Saarland University Computer Graphics Lab Germany TU Wien Institute of Visual Computing and Human-Centered Technology Austria Graz University of Technology Institute of Computer Graphics and Vision Austria

ISBN: (纸本)9783038681359

Since its inception, the CUDA programming model has been continuously evolving. Because the CUDA toolkit aims to consistently expose cutting-edge capabilities for general-purpose compute jobs to its users, the added features in each new version reflect the rapid changes that we observe in GPU architectures. Over the years, the changes in hardware, growing scope of built-in functions and libraries, as well as an advancing C++ standard compliance have expanded the design choices when coding for CUDA, and significantly altered the directives to achieve peak performance. In this tutorial, we give a thorough introduction to the CUDA toolkit, demonstrate how a contemporary application can benefit from recently introduced features and how they can be applied to task-based GPU scheduling in particular. For instance, we will provide detailed examples of use cases for independent thread scheduling, cooperative groups, and the CUDA standard library, libcu++, which are certain to become an integral part of clean coding for CUDA in the near future. © 2021 The Author(s).

关键词： Scheduling

3D-MetaConNet: Meta-learning for 3D Shape Classification and Segmentation

学校读者我要写书评

暂无评论

3D-MetaConNet: Meta-learning for 3D Shape Classification and...

International Conference on 3D Imaging, Modeling, Processing, visualization and Transmission (3DIMPVT)

作者： Hao Huang Xiang Li Lingjing Wang Yi Fang NYU Multimedia and Visual Computing Lab USA New York University Abu Dhabi UAE

ISBN: (纸本)9781665426893

Supervised learning on 3D shapes are extensively studied by prior literature, among which PointNet [29] and its variants PointNet++ [31] are representatives. However, these methods tackle 3D shape learning problems by training from scratch using a fixed learning algorithm over large amounts of labeled data, potentially challenged by data and computation bottlenecks. In the paper, we design a novel model, under the framework of meta-learning, to learn 3D shape representation. By training over multiple 3D tasks, each of which is defined as a supervised learning problem, our method can fast adapt to unseen tasks containing limited labeled data. Specifically, our model consists of a 3Dmeta-learner and a task-oriented 3D-learner, where the 3D-meta-learner produces parameter initialization for the 3D-learner after being trained over different tasks. With adaptively initialized parameters, the 3D-learner can be tuned rapidly in a few steps to achieve good performance on novel tasks with a small amount of training data. To further facilitate discriminative shape feature learning, we introduce a novel task-aware feature adaptation module under a contrastive learning scheme, in which all shapes in each task are considered as a whole and task-oriented compact features are learned. Therefore, we dub our model as 3DMetaConNet. Experiments on three public 3D datasets for few-shot shape classification and segmentation demonstrate that our method can learn compact and discriminative 3D shape features efficiently and robustly in a fast adaptation manner. Our method particularly outperforms the methods without a meta-learning framework and is also superior to existing meta-learning approaches.

关键词： Training Representation learning Adaptation models Solid modeling Three-dimensional displays Shape Computational modeling

EVALUATION OF DIRICHLET PROCESS GAUSSIAN MIXTURES FOR SEGMENTATION ON NOISY HYPERSPECTRAL IMAGES A PREPRINT

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Mantripragada, Kiran Qureshi, Faisal Z. Visual Computing Lab Faculty of Science Ontario Tech University 2000 Simcoe Street North OshawaONL1G OC5 Canada

Image segmentation is a fundamental step for the interpretation of Remote Sensing Images. Clustering or segmentation methods usually precede the classification task and are used as support tools for manual labeling. The most common algorithms, such as k-means, mean-shift, and MRS, require an extra manual step to find the scale parameter. The segmentation results are severely affected if the parameters are not correctly tuned and diverge from the optimal values. Additionally, the search for the optimal scale is a costly task, as it requires a comprehensive hyper-parameter search. This paper proposes and evaluates a method for segmentation of Hyperspectral Images using the Dirichlet Process Gaussian Mixture Model. Our model can self-regulate the parameters until it finds the optimal values of scale and the number of clusters in a given dataset. The results demonstrate the potential of our method to find objects in a Hyperspectral Image while bypassing the burden of manual search of the optimal parameters. In addition, our model also produces similar results on noisy datasets, while previous research usually required a pre-processing task for noise reduction and spectral smoothing. © 2022, CC BY-NC-ND.

关键词： Gaussian distribution

Regularized molecular conformation fields 22

学校读者我要写书评

暂无评论

Regularized molecular conformation fields

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： Lihao Wang Yi Zhou Yiqun Wang Xiaoqing Zheng Xuanjing Huang Hao Zhou Fudan University ByteDance AI Lab Fudan University and Shanghai Collaborative Innovation Center of Intelligent Visual Computing Institute for AI Industry Research (AIR) Tsinghua University

ISBN: (纸本)9781713871088

Predicting energetically favorable 3-dimensional conformations of organic molecules from molecular graph plays a fundamental role in computer-aided drug discovery research. However, effectively exploring the high-dimensional conformation space to identify (meta)stable conformers is anything but trivial. In this work, we introduce RMCF, a novel framework to generate a diverse set of low-energy molecular conformations through sampling from a regularized molecular conformation field. We develop a data-driven molecular segmentation algorithm to automatically partition each molecule into several structural building blocks to reduce the modeling degrees of freedom. Then, we employ a Markov Random Field to learn the joint probability distribution of fragment configurations and interfragment dihedral angles, which enables us to sample from different low-energy regions of a conformation space. Our model constantly outperforms state-of-the-art models for the conformation generation task on the GEOM-Drugs dataset. We attribute the success of RMCF to modeling in a regularized feature space and learning a global fragment configuration distribution for effective sampling. The proposed method could be generalized to deal with larger biomolecular systems.

关键词：

OmniTracker: Unifying Object Tracking by Tracking-with-Detection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Junke Chen, Dongdong Wu, Zuxuan Luo, Chong Dai, Xiyang Yuan, Lu Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center on Intelligent Visual Computing China Microsoft Cloud + AI China Microsoft Research Asia China

Object tracking (OT) aims to estimate the positions of target objects in a video sequence. Depending on whether the initial states of target objects are specified by provided annotations in the first frame or the categories, OT could be classified as instance tracking (e.g., SOT and VOS) and category tracking (e.g., MOT, MOTS, and VIS) tasks. Combing the advantages of the best practices developed in both communities, we propose a novel tracking-with-detection paradigm, where tracking supplements appearance priors for detection and detection provides tracking with candidate bounding boxes for association. Equipped with such a design, a unified tracking model, OmniTracker, is further presented to resolve all the tracking tasks with a fully shared network architecture, model weights, and inference pipeline. Extensive experiments on 7 tracking datasets, including LaSOT, TrackingNet, DAVIS16-17, MOT17, MOTS20, and YTVIS19, demonstrate that OmniTracker achieves on-par or even better results than both task-specific and unified tracking models. Copyright © 2023, The Authors. All rights reserved.

关键词： Network architecture

Hyperspectral Pixel Unmixing with Latent Dirichlet Variational Autoencoder

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Mantripragada, Kiran Qureshi, Faisal Z. Visual Computing Lab Faculty of Science Ontario Tech University 2000 Simcoe Street North OshawaONL1G OC5 Canada

We present a method for hyperspectral pixel unmixing. The proposed method assumes that (1) abundances can be encoded as Dirichlet distributions and (2) spectra of endmembers can be represented as multivariate Normal distributions. The method solves the problem of abundance estimation and endmember extraction within a variational autoencoder setting where a Dirichlet bottleneck layer models the abundances, and the decoder performs endmember extraction. The proposed method can also leverage transfer learning paradigm, where the model is only trained on synthetic data containing pixels that are linear combinations of one or more endmembers of interest. In this case, we retrieve endmembers (spectra) from the United States Geological Survey Spectral Library. The model thus trained can be subsequently used to perform pixel unmixing on "real data" that contains a subset of the endmembers used to generated the synthetic data. The model achieves state-of-the-art results on several benchmarks: Cuprite, Urban Hydice and Samson. We also present new synthetic dataset, OnTech-HSISyn-21, that can be used to study hyperspectral pixel unmixing methods. We showcase the transfer learning capabilities of the proposed model on Cuprite and OnTech-HSI-Syn-21 datasets. In summary, the proposed method can be applied for pixel unmixing a variety of domains, including agriculture, forestry, mineralogy, analysis of materials, healthcare, etc. Additionally, the proposed method eschews the need for labelled data for training by leveraging the transfer learning paradigm, where the model is trained on synthetic data generated using the endmembers present in the "real" data. © 2022, CC BY-NC-ND.

关键词： Pixels