检索结果-内蒙古大学图书馆

RH-BrainFS: regional heterogeneous multimodal brain networks fusion strategy 23

学校读者我要写书评

暂无评论

RH-BrainFS: regional heterogeneous multimodal brain networks...

Proceedings of the 37th International Conference on Neural information processing Systems

作者： Hongting Ye Yalu Zheng Yueying Li Ke Zhang Youyong Kong Yonggui Yuan Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing School of Computer Science and Engineering Southeast University Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing School of Computer Science and Engineering Southeast University and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) Ministry of Education China Department of Psychosomatics and Psychiatry Zhongda Hospital School of Medicine Southeast University

Multimodal fusion has become an important research technique in neuroscience that completes downstream tasks by extracting complementary information from multiple modalities. Existing multimodal research on brain networks mainly focuses on two modalities, structural connectivity (SC) and functional connectivity (FC). Recently, extensive literature has shown that the relationship between SC and FC is complex and not a simple one-to-one mapping. The coupling of structure and function at the regional level is heterogeneous. However, all previous studies have neglected the modal regional heterogeneity between SC and FC and fused their representations via "simple patterns", which are inefficient ways of multimodal fusion and affect the overall performance of the model. In this paper, to alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS). Briefly, we introduce a brain subgraph networks module to extract regional characteristics of brain networks, and further use a new transformer-based fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. To the best of our knowledge, this is the first paper to explicitly state the issue of structural-functional modal regional heterogeneity and to propose a solution. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art methods in a variety of neuroscience tasks. The codes are available at https://***/Yedaxia1/RH-BrainFS.

关键词：

MICM: Rethinking Unsupervised Pretraining for Enhanced Few-shot Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Zhenyu Chen, Guangyao Zou, Yixiong Huang, Zhimeng Li, Yuhua Li, Ruixuan School of Computer Science and Technology Huazhong University of Science and Technology Wuhan China National Key Laboratory for Multimedia Information Processing School of Computer Science Peking University Beijing China National Engineering Research Center of Visual Technology School of Computer Science Peking University Beijing China

Humans exhibit a remarkable ability to learn quickly from a limited number of labeled samples, a capability that starkly contrasts with that of current machine learning systems. Unsupervised Few-Shot Learning (U-FSL) seeks to bridge this divide by reducing reliance on annotated datasets during initial training phases. In this work, we first quantitatively assess the impacts of Masked Image Modeling (MIM) and Contrastive Learning (CL) on few-shot learning tasks. Our findings highlight the respective limitations of MIM and CL in terms of discriminative and generalization abilities, which contribute to their underperformance in U-FSL contexts. To address these trade-offs between generalization and discriminability in unsupervised pretraining, we introduce a novel paradigm named Masked Image Contrastive Modeling (MICM). MICM creatively combines the targeted object learning strength of CL with the generalized visual feature learning capability of MIM, significantly enhancing its efficacy in downstream few-shot learning inference. Extensive experimental analyses confirm the advantages of MICM, demonstrating significant improvements in both generalization and discrimination capabilities for few-shot learning. Our comprehensive quantitative evaluations further substantiate the superiority of MICM, showing that our two-stage U-FSL framework based on MICM markedly outperforms existing leading baselines. The repository of this project is available at https://***/iCGY96/MICM. Copyright © 2024, The Authors. All rights reserved.

关键词： Contrastive Learning

MDG:Multilingual Co-speech Gesture Generation with Low-level Audio Representation and Diffusion Models

学校读者我要写书评

暂无评论

MDG:Multilingual Co-speech Gesture Generation with Low-level...

International Conference on Asian Language processing (IALP)

作者： Jie Yang Feilong Bao Inner Mongolia University Hohhot China National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology College of Computer Science

ISBN: (数字)9798331540852

ISBN: (纸本)9798331540869

Co-speech gesture generation is crucial for enhancing human-computer interaction. Most existing research focuses on English, overlooking multilingual speech and the ability to generalize across languages. Additionally, these studies often rely on pre-trained audio feature extraction models, insufficient for non-English languages. To address these issues, we introduce MDG, a novel diffusion model-based method for speech-driven gesture generation using a multilingual dataset. Our approach incorporates Language ID to distinguish between languages and utilizes low-level audio features to improve multilingual generalization. Extensive experiments demonstrate that MDG significantly outperforms baseline models in multilingual co-speech gesture generation. For more details, please visit our demo page 1 .

关键词： Training Human computer interaction Feature extraction Diffusion models Data models Data mining Speech processing

Conditional Video-Text Reconstruction Network with Cauchy Mask for Weakly Supervised Temporal Sentence Grounding

学校读者我要写书评

暂无评论

Conditional Video-Text Reconstruction Network with Cauchy Ma...

IEEE International Conference on Multimedia and Expo (ICME)

作者： Jueqi Wei Yuanwu Xu Mohan Chen Yuejie Zhang Rui Feng Shang Gao School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University School of Information Technology Deakin University

Temporal sentence grounding aims to detect the target segment most related to a given query in an untrimmed video. To alleviate the expensive annotation cost for temporal labels, researchers paid more attention to weakly supervised setting. Prior studies neglected the utilization of video representation reconstruction, which led to an unbalanced alignment learning. Moreover, they used different strategies to generate proposals which ignored the temporal structure in a query. In this paper, we propose a novel Conditional Video-Text Reconstruction Network (CVTRN). It supports conditional reconstruction of video and text representation. Specifically, video and text features are fused to compute semantic alignment, which is the condition of reconstruction. A new mask strategy for mask conditioned sentence reconstruction is also devised. This strategy focuses more on boundary regions than the widely used Gaussian mask in previous methods. Experimental results on two public benchmark datasets show that our CVTRN outperforms the state-of-the-art methods.

关键词：

SPTNET: Span-based Prompt Tuning for Video Grounding

学校读者我要写书评

暂无评论

SPTNET: Span-based Prompt Tuning for Video Grounding

IEEE International Conference on Multimedia and Expo (ICME)

作者： Yiren Zhang Yuanwu Xu Mohan Chen Yuejie Zhang Rui Feng Shang Gao School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University School of Information Technology Deakin University

When a Pre-trained Language Model (PLM) is adopted in video grounding task, it usually acts as a text encoder without having its knowledge fully utilized. Also, there exists an inconsistency problem between the pre-training and downstream objectives. To solve the issues, we propose a new paradigm, named Span-based Prompt Tuning (SPTNet). It can convert the video grounding task into a cloze form. Specifically, a query is first changed into a form with mask token by a template, then the video and the query embeddings are integrated through a cross-modal transformer. The start and end points of the query matching time span are predicted with the embedding of the mask token. Experimental results on two public benchmarks ActivityNet Captions and Charades-STA show that our SPTNet achieves surpassing performance compared with state-of-the-art methods.

关键词：

A New Partial Task Offloading Method in a Cooperation Mode under Multi-Constraints for Multi-UE

学校读者我要写书评

暂无评论

computers, Materials & Continua 2023年第9期76卷 2879-2900页

作者： Shengyao Sun Ying Du Jiajun Chen Xuan Zhang Jiwei Zhang Yiyi Xu School of Information Science and Technology Zhengzhou Normal UniversityZhengzhou450044China Henan Key Laboratory of Big Data Analysis and Processing Henan UniversityKaifeng475004China School of Geography and Tourism Zhengzhou Normal UniversityZhengzhou450044China Science and Engineering College South China University of TechnologyGuangzhou510641China Department of Electrical and Electronic Engineering Luohe Vocational Technology CollegeLuohe462002China School of Computer Science Beijing University of Posts and TelecommunicationsBeijing100876China Cardiff School of Engineering Cardiff UniversityCardiffCF103XQ 15UK

In Multi-access Edge Computing(MEC),to deal with multiple user equipment(UE)’s task offloading problem of parallel relationships under the multi-constraints,this paper proposes a cooperation partial task offloading method(named CPMM),aiming to reduce UE’s energy and computation consumption,while meeting the task completion delay as much as *** first studies the task offloading of single-UE and then considers the task offloading ofmulti-UE based on single-UE task *** the critical path algorithmto divide the modules into key and non-key *** to some constraints of UE-self when offloading tasks,it gives priority to non-key modules for offloading and uses the evaluation decision method to select some appropriate key modules for *** on fully considering the competition between multiple UEs for communication resources and MEC service resources,CPMM uses the weighted queuing method to alleviate the competition for communication resources and uses the branch decision algorithm to determine the location of module offloading by BS according to the MEC servers’*** achieves its goal by selecting reasonable modules to offload and using the cooperation ofUE,MEC,andCloudCenter to determine the execution location of *** experiments demonstrate that CPMM obtains superior performances in task computation consumption reducing around 6%on average,task completion delay reducing around 5%on average,and better task execution success rate than other similar methods.

关键词： MEC partial task offloading parallel dependencies completion delay

SCBench: A Sports Commentary Benchmark for Video LLMs

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ge, Kuangzhi Chen, Lingjun Zhang, Kevin Luo, Yulin Shi, Tianyu Fan, Liaoyuan Li, Xiang Wang, Guanqun Zhang, Shanghang State Key Laboratory of Multimedia Information Processing School of Computer Science Peking University China Department of Computer Science University of Toronto Canada The University of Hong Kong Hong Kong Southern University of Science and Technology China

Recently, significant advances have been made in Video Large Language Models (Video LLMs) in both academia and industry. However, methods to evaluate and benchmark the performance of different Video LLMs, especially their fine-grained, temporal visual capabilities, remain very limited. On one hand, current benchmarks use relatively simple videos (e.g., subtitled movie clips) where the model can understand the entire video by processing just a few frames. On the other hand, their datasets lack diversity in task format, comprising only QA or multi-choice QA, which overlooks the models' capacity for generating in-depth and precise texts. Sports videos, which feature intricate visual information, sequential events, and emotionally charged commentary, present a critical challenge for Video LLMs, making sports commentary an ideal benchmarking task. Inspired by these challenges, we propose a novel task: sports video commentary generation, developed SCBench for Video LLMs. To construct such a benchmark, we introduce (1) SCORES, a six-dimensional metric specifically designed for our task, upon which we propose a GPT-based evaluation method, and (2) CommentarySet, a dataset consisting of 5,775 annotated video clips and ground-truth labels tailored to our metric. Based on SCBench, we conduct comprehensive evaluations on multiple Video LLMs (e.g. VILA, Video-LLaVA, etc.) and chain-of-thought baseline methods. Our results found that InternVL-Chat-2 achieves the best performance with 5.44, surpassing the second-best by 1.04. Our work provides a fresh perspective for future research, aiming to enhance models' overall capabilities in complex visual understanding tasks. Our dataset will be released soon. © 2024, CC BY-NC-ND.

关键词： Footage counters

RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Dai, Qiyu Ni, Xingyu Shen, Qianfan Chen, Wenzheng Chen, Baoquan Chu, Mengyu School of Intelligence Science and Technology Peking University China School of Computer Science Peking University China School of EECS Peking University China Wangxuan Institute of Computer Technology Peking University China State Key Laboratory of Multimedia Information Processing Peking University China State Key Laboratory of General Artificial Intelligence Peking University China

We consider the problem of adding dynamic rain effects to in-the-wild scenes in a physically-correct manner. Recent advances in scene modeling have made significant progress, with NeRF and Gaussian Splatting techniques emerging as powerful tools for reconstructing complex scenes. However, while effective for novel view synthesis, these methods typically struggle with challenging scene editing tasks, such as physics-based rain simulation. In contrast, traditional physics-based simulations can generate realistic rain effects, such as raindrops and splashes, but they often rely on skilled artists to carefully set up high-fidelity scenes. This process lacks flexibility and scalability, limiting its applicability to broader, open-world environments. In this work, we introduce RainyGS, a novel approach that leverages the strengths of both physics-based modeling and Gaussian Splatting to generate photorealistic, dynamic rain effects in open-world scenes with physical accuracy. At the core of our method is the integration of physically-based raindrop and shallow water simulation techniques within the fast Gaussian Splatting rendering framework, enabling realistic and efficient simulations of raindrop behavior, splashes, and reflections. Our method supports synthesizing rain effects at over 30 fps, offering users flexible control over rain intensity—from light drizzles to heavy downpours. We demonstrate that RainyGS performs effectively for both real-world outdoor scenes and large-scale driving scenarios, delivering more photorealistic and physically-accurate rain effects compared to state-of-the-art methods. Project page can be found at https: //***/RainyGS/. Copyright © 2025, The Authors. All rights reserved.

关键词： Dynamics

Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yu, Ting Fu, Kunhao Wang, Shuhui Huang, Qingming Yu, Jun School of Information Science and Technology Hangzhou Normal University Hangzhou311121 China Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing100190 China School of Computer Science and Technology University of Chinese Academy of Sciences Beijing101408 China Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University Hangzhou310018 China

Video Question Answering (VideoQA) represents a crucial intersection between video understanding and language processing, requiring both discriminative unimodal comprehension and sophisticated cross-modal interaction for accurate inference. Despite advancements in multi-modal pre-trained models and video-language foundation models, these systems often struggle with domain-specific VideoQA due to their generalized pre-training objectives. Addressing this gap necessitates bridging the divide between broad cross-modal knowledge and the specific inference demands of VideoQA tasks. To this end, we introduce HeurVidQA, a framework that leverages domain-specific entity-action heuristics to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model’s focus toward precise cues that enhance reasoning. By delivering fine-grained heuristics, we improve the model’s ability to identify and interpret key entities and actions, thereby enhancing its reasoning capabilities. Extensive evaluations across multiple VideoQA datasets demonstrate that our method significantly outperforms existing models, underscoring the importance of integrating domain-specific knowledge into video-language models for more accurate and context-aware VideoQA. Copyright © 2024, The Authors. All rights reserved.

关键词： Inference engines