检索结果-内蒙古大学图书馆

JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ji, Jiayi Wang, Haowei Wu, Changli Ma, Yiwei Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China

The rising importance of 3D understanding, pivotal in computer vision, autonomous driving, and robotics, is evident. However, a prevailing trend, which straightforwardly resorted to transferring 2D alignment strategies to the 3D domain, encounters three distinct challenges: (1) Information Degradation: This arises from the alignment of 3D data with mere single-view 2D images and generic texts, neglecting the need for multi-view images and detailed subcategory texts. (2) Insufficient Synergy: These strategies align 3D representations to image and text features individually, hampering the overall optimization for 3D models. (3) Underutilization: The fine-grained information inherent in the learned representations is often not fully exploited, indicating a potential loss in detail. To address these issues, we introduce JM3D, a comprehensive approach integrating point cloud, text, and image. key contributions include the Structured Multimodal Organizer (SMO), enriching vision-language representation with multiple views and hierarchical text, and the Joint Multi-modal Alignment (JMA), combining language understanding with visual representation. Our advanced model, JM3D-LLM, marries 3D representation with large language models via efficient fine-tuning. Evaluations on ModelNet40 and ScanObjectNN establish JM3D’s superiority. The superior performance of JM3D-LLM further underscores the effectiveness of our representation transfer approach. Our code and models are available at https://***/Mr-Neko/JM3D. Copyright © 2023, The Authors. All rights reserved.

关键词： Alignment

Outlier-aware slicing for post-training quantization in vision transformer 24

学校读者我要写书评

暂无评论

Outlier-aware slicing for post-training quantization in visi...

Proceedings of the 41st International Conference on Machine Learning

作者： Yuexiao Ma Huixia Li Xiawu Zheng Feng Ling Xuefeng Xiao Rui Wang Shilei Wen Fei Chao Rongrong Ji ByteDance Inc. and Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University P.R. China. ByteDance Inc. Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University P.R. China and Peng Cheng Laboratory Shenzhen China and Institute of Artificial Intelligence Xiamen University Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University P.R. China. Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University P.R. China and and Institute of Artificial Intelligence Xiamen University

Post-Training Quantization (PTQ) is a vital technique for network compression and acceleration, gaining prominence as model sizes increase. This paper addresses a critical challenge in PTQ: the severe impact of outliers on the accuracy of quantized transformer architectures. Specifically, we introduce the concept of 'reconstruction granularity' as a novel solution to this issue, which has been overlooked in previous works. Our work provides theoretical insights into the role of reconstruction granularity in mitigating the outlier problem in transformer models. This theoretical framework is supported by empirical analysis, demonstrating that varying reconstruction granularities significantly influence quantization performance. Our findings indicate that different architectural designs necessitate distinct optimal reconstruction granularities. For instance, the multi-stage Swin Transformer architecture benefits from finer granularity, a deviation from the trends observed in ViT and DeiT models. We further develop an algorithm for determining the optimal reconstruction granularity for various ViT models, achieving state-of-the-art (SOTA) performance in PTQ. For example, applying our method to 4-bit quantization, the Swin-Base model achieves a Top-1 accuracy of 82.24% on the ImageNet classification task. This result surpasses the RepQ-ViT by 3.92% (82.24% VS 78.32%). Similarly, our approach elevates the ViT-Small to a Top-1 accuracy of 80.50%, outperforming NoisyQuant by 3.64% (80.50% VS 76.86%).

关键词：

Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Jinlu Zhou, Yiyi Zheng, Qiancheng Du, Xiaoxiong Luo, Gen Peng, Jun Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China Peng Cheng Laboratory Shenzhen518000 China

Text-to-3D-aware face (T3D Face) generation and manipulation is an emerging research hot spot in machine learning, which still suffers from low efficiency and poor quality. In this paper, we propose an End-to-End efficient and Effective network for fast and accurate T3D face generation and manipulation, termed E3-FaceNet. Different from existing complex generation paradigms, E3-FaceNet resorts to a direct mapping from text instructions to 3D-aware visual space. We introduce a novel Style Code Enhancer to enhance cross-modal semantic alignment, alongside an innovative Geometric Regularization objective to maintain consistency across multi-view generations. Extensive experiments on three benchmark datasets demonstrate that E3-FaceNet can not only achieve picture-like 3D face generation and manipulation, but also improve inference speed by orders of magnitudes. For instance, compared with Latent3D, E3-FaceNet speeds up the five-view generations by almost 470 times, while still exceeding in generation quality. Our code is released at https://***/Aria-Zhangjl/E3-FaceNet. Copyright © 2024, The Authors. All rights reserved.

关键词： Mapping

TOWARDS efficient AUTOMATIC SELF-PRUNING OF LARGE LANGUAGE MODELS

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Huang, Weizhong Zhang, Yuxin Zheng, Xiawu Chao, Fei Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China Institute of Artificial Intelligence Xiamen University China

Despite exceptional capabilities, Large Language Models (LLMs) still face deployment challenges due to their enormous size. Post-training structured pruning is a promising solution that prunes LLMs without the need for retraining, reducing computational overhead, and it is hardware-deployment friendly. However, the training-free nature of post-training structured pruning leads to significant performance degradation. We argue that the key to mitigating this issue lies in accurately determining the pruning rate for each layer. Meanwhile, we find that LLMs may have prior knowledge about their own redundancy. Based on this insight, we introduce Self-Pruner an end-to-end automatic self-pruning framework for LLMs, which efficiently search layer-wise pruning rates. Specifically, Self-Pruner leverages LLMs to autonomously execute the entire evolutionary search process to search for pruning rate configurations. In this process, LLMs are used to generate populations, select parent solutions from the current population, and perform crossover and mutation operations to produce offspring solutions. In this way, LLMs automatically generate and evaluate a large number of candidate solutions, effectively converging to find the pruning rate configurations with minimal human intervention. Extensive experiments demonstrate Self-Pruner’s better performance compared to existing state-of-the-art methods. Notably, Self-Pruner prunes LLaMA-2-70B to 49B level with only 0.80% drop in accuracy across seven commonsense reasoning tasks, achieving a 1.39× speedup on NVIDIA A100 80GB GPU. Further pruning to 35B level resulted in only a 3.80% decrease in accuracy while obtaining a 1.70× speedup. Copyright © 2025, The Authors. All rights reserved.

关键词： Graphics processing unit

Visual-Linguistic Representation Learning with Deep Cross-Modality Fusion for Referring Multi-Object Tracking

学校读者我要写书评

暂无评论

Visual-Linguistic Representation Learning with Deep Cross-Mo...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Wenyan He Yajun Jian Yang Lu Hanzi Wang Fujian Key Laboratory of Sensing and Computing for Smart City School of Informatics Xiamen University Xiamen China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University Xiamen China

Referring multi-object tracking is a new rising research topic that aims at detecting and tracking the referred objects in a video sequence based on a natural language expression. Compared with traditional multi-object tracking, this setting guides object tracking with high-level semantic information, which may bring more flexible and robust tracking performance in practical scenarios. However, existing methods perform cross-modal fusion in only one phase. The limitation of visual-linguistic representation is prone to causing visionlanguage mismatching and producing poor tracking results. To effectively fuse vision and language modalities, we propose DeepRMOT with deep cross-modality fusion, including an enhanced early-fusion module, a bidirectional crossmodality encoder, and a cross-modality decoder. Therefore, DeepRMOT can boost object detection and data association by the enhanced visual-linguistic representation. Extensive experiments on the Refer-KITTI demonstrate the effectiveness of our method.

关键词：

Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

学校读者我要写书评

暂无评论

arXiv 2025年

In this paper, we address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective. Specifically, we identify a critical issue of "reconstruction error explosion" in existing LLMs sparsification methods. This refers to the cumulative effect of reconstruction errors throughout the sparsification process, where errors from earlier layers propagate and amplify in subsequent layers. As a result, the overall reconstruction error increases significantly, leading to a substantial degradation in model performance. Through theoretical analysis, we derive a simple yet effective approach to layerwise sparsity allocation that mitigates this issue. Our method uses a monotonically increasing arithmetic progression, reducing the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter. Remarkably, this allows for the optimal layer-wise sparsity rates to be identified with just a few trials. Both our theoretical analysis and experimental results demonstrate that this sparsity allocation scheme is near optimal. Extensive experiments show that our method significantly improves the performance of sparse LLMs across various architectures, outperforming existing layer-wise sparsity methods. Furthermore, it enhances the performance of various compression techniques and is applicable to vision and multimodal models. Notably, our method achieves a reduction of 52.10 in perplexity for the 70% sparse LLaMA2-7B model obtained via Wanda, improves average zero-shot accuracy by 10.50%, and delivers speedups of 2.63× and 2.23× on CPU and GPU, respectively. Copyright © 2025, The Authors. All rights reserved.

关键词： Errors

Spatio-Temporal Correlation Learning for Multiple Object Tracking

学校读者我要写书评

暂无评论

Spatio-Temporal Correlation Learning for Multiple Object Tra...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Yajun Jian Chihui Zhuang Wenyan He Kaiwen Du Yang Lu Hanzi Wang Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University Xiamen China Fujian Key Laboratory of Sensing and Computing for Smart City School of Informatics Xiamen University Xiamen China

Multi-object tracking (MOT) has gained remarkable progress in recent years, while due to the complexity of real-world environments, there are still many challenges that remain unsolved, such as object occlusion and deformation. To effectively alleviate this problem, we propose a simple yet effective Transformer-based tracker, named CLNet, consisting of an Instance-Aware Localization (IAL) module and a Temporal Context Aggregation (TCA) module. Specifically, the former learns the correlation of object positions for potential location estimation, and the latter learns the correlation of background contexts to obtain robust re-ID features for data association. Experimental results show that CLNet outperforms the baseline method by +2.2 MOTA and +2.3 IDF1 on MOT17 and +6.3 MOTA and +3.6 IDF1 on MOT20 respectively, which demonstrate the effectiveness of the proposed method.

关键词：

MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Ma, Yiwei Xu, Guohai Sun, Xiaoshuai Ji, Jiayi Lou, Jie Zhang, Debing Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Fujian361005 China Xiaohongshu Inc Shanghai200025 China

Visual instruction tuning (VIT) has emerged as a crucial technique for enabling multi-modal large language models (MLLMs) to follow user instructions adeptly. Yet, a significant gap persists in understanding the attributes of high-quality instruction tuning data and frameworks for its automated selection. To address this, we introduce MLLM-Selector, an automated approach that identifies valuable data for VIT by weighing necessity and diversity. Our process starts by randomly sampling a subset from the VIT data pool to fine-tune a pretrained model, thus creating a seed model with an initial ability to follow instructions. Then, leveraging the seed model, we calculate necessity scores for each sample in the VIT data pool to identify samples pivotal for enhancing model performance. Our findings underscore the importance of mixing necessity and diversity in data choice, leading to the creation of MLLM-Selector, our methodology that fuses necessity scoring with strategic sampling for superior data refinement. Empirical results indicate that within identical experimental conditions, MLLM-Selector surpasses LLaVA-1.5 in some benchmarks with less than 1% of the data and consistently exceeds performance across all validated benchmarks when using less than 50%. Copyright © 2025, The Authors. All rights reserved.

关键词： Visual languages

Semi-Supervised Panoptic Narrative Grounding

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yang, Danni Ji, Jiayi Sun, Xiaoshuai Wang, Haowei Li, Yinan Ma, Yiwei Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Fujian Xiamen China

Despite considerable progress, the advancement of Panoptic Narrative Grounding (PNG) remains hindered by costly annotations. In this paper, we introduce a novel Semi-Supervised Panoptic Narrative Grounding (SS-PNG) learning scheme, capitalizing on a smaller set of labeled image-text pairs and a larger set of unlabeled pairs to achieve competitive performance. Unlike visual segmentation tasks, PNG involves one pixel belonging to multiple open-ended nouns. As a result, existing multi-class based semi-supervised segmentation frameworks cannot be directly applied to this task. To address this challenge, we first develop a novel SS-PNG Network (SS-PNG-NW) tailored to the SS-PNG setting. We thoroughly investigate strategies such as Burn-In and data augmentation to determine the optimal generic configuration for the SS-PNG-NW. Additionally, to tackle the issue of imbalanced pseudo-label quality, we propose a Quality-Based Loss Adjustment (QLA) approach to adjust the semi-supervised objective, resulting in an enhanced SSPNG-NW+. Employing our proposed QLA, we improve BCE Loss and Dice loss at pixel and mask levels, respectively. We conduct extensive experiments on PNG datasets, with our SS-PNG-NW+ demonstrating promising results comparable to fully-supervised models across all data ratios. Remarkably, our SS-PNG-NW+ outperforms fully-supervised models with only 30% and 50% supervision data, exceeding their performance by 0.8% and 1.1% respectively. This highlights the effectiveness of our proposed SS-PNG-NW+ in overcoming the challenges posed by limited annotations and enhancing the applicability of PNG tasks. The source code is available at https://***/nini0919/SSPNG. © 2023, CC BY-NC-ND.

关键词： Pixels