检索结果-内蒙古大学图书馆

Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chen, Yin Li, Jia Zhang, Yu Hu, Zhenzhen Shan, Shiguang Wang, Meng Hong, Richang School of Computer Science and Information Engineering Hefei University of Technology Hefei230601 China Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing100190 China University of Chinese Academy of Sciences Beijing100049 China

Dynamic facial expression recognition (DFER) infers emotions from the temporal evolution of expressions, unlike static facial expression recognition (SFER), which relies solely on a single snapshot. This temporal analysis provides richer information and promises greater recognition capability. However, current DFER methods often exhibit unsatisfied performance largely due to fewer training samples compared to SFER. Given the inherent correlation between static and dynamic expressions, we hypothesize that leveraging the abundant SFER data can enhance DFER. To this end, we propose Static-for-Dynamic (S4D), a unified dual-modal learning framework that integrates SFER data as a complementary resource for DFER. Specifically, S4D employs dual-modal self-supervised pre-training on facial images and videos using a shared Vision Transformer (ViT) encoder-decoder architecture, yielding improved spatiotemporal representations. The pre-trained encoder is then fine-tuned on static and dynamic expression datasets in a multi-task learning setup to facilitate emotional information interaction. Unfortunately, vanilla multitask learning in our study results in negative transfer. To address this, we propose an innovative Mixture of Adapter Experts (MoAE) module that facilitates task-specific knowledge acquisition while effectively extracting shared knowledge from both static and dynamic expression data. Extensive experiments demonstrate that S4D achieves a deeper understanding of DFER, setting new state-of-the-art performance on FERV39K, MAFW, and DFEW benchmarks, with weighted average recall (WAR) of 53.65%, 58.44%, and 76.68%, respectively. Additionally, a systematic correlation analysis between SFER and DFER tasks is presented, which further elucidates the potential benefits of leveraging SFER. Copyright © 2024, The Authors. All rights reserved.

关键词： Self-supervised learning

Multimodal Knowledge Graph Embeddings via Lorentz-based Contrastive Learning

学校读者我要写书评

暂无评论

Multimodal Knowledge Graph Embeddings via Lorentz-based Cont...

IEEE International Conference on Multimedia and Expo (ICME)

作者： Ruizhou Liu Zongsheng Cao Zhe Wu Qianqian Xu Qingming Huang School of Computer Science and Technology University of Chinese Academy Sciences Beijing China Department of Networked Intelligence Pengcheng Laboratory Shenzhen China Institute of Information Engineering Chinese Academy Sciences Beijing China Key Laboratory of Intelligent Information Processing Institute of Computing Technology CAS Beijing China

ISBN: (数字)9798350390155

ISBN: (纸本)9798350390162

Multimodal knowledge graph embeddings (MKGE) have recently garnered significant attention. Unlike traditional unimodal knowledge graph embeddings, MKGE integrates both structural and multimodal knowledge to represent entities within a unified framework. However, real-world entities exhibit heterogeneity, often resulting in semantic inconsistencies where structurally similar embeddings may diverge significantly in their multimodal representations. Previous approaches primarily focus on directly fusing structural and multimodal embeddings, thus overlooking the issue of semantic-embedding inconsistency. To tackle this issue, we propose a new multimodal knowledge graph embedding method via Lorentz-based contrastive learning (LCKGE). we firstly introduce a well-designed nearest-neighbor fusion module via contrastive learning for multimodal fusion. Then, an attention-based Lorentz transformation is proposed for capturing more complex geometric information in MKGs. Furthermore, a series of comprehensive experiments are conducted to demonstrate the effectiveness of our model. We provide the code and appendix of LCKGE in https://***/RuizhouLiu/LCKGE

关键词： Codes Semantics Knowledge graphs Contrastive learning Benchmark testing

LEARNING MONOTONIC ATTENTION IN TRANSDUCER FOR STREAMING GENERATION

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Zhengrui Feng, Yang Zhang, Min Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences China Key Laboratory of AI Safety Chinese Academy of Sciences China University of Chinese Academy of Sciences China School of Future Science and Engineering Soochow University China

Streaming generation models are increasingly utilized across various fields, with the Transducer architecture being particularly popular in industrial applications. However, its input-synchronous decoding mechanism presents challenges in tasks requiring non-monotonic alignments, such as simultaneous translation, leading to suboptimal performance in these contexts. In this research, we address this issue by tightly integrating Transducer’s decoding with the history of input stream via a learnable monotonic attention mechanism. Our approach leverages the forward-backward algorithm to infer the posterior probability of alignments between the predictor states and input timestamps, which is then used to estimate the context representations of monotonic attention in training. This allows Transducer models to adaptively adjust the scope of attention based on their predictions, avoiding the need to enumerate the exponentially large alignment space. Extensive experiments demonstrate that our MonoAttn-Transducer significantly enhances the handling of non-monotonic alignments in streaming generation, offering a robust solution for Transducer-based frameworks to tackle more complex streaming generation tasks. Codes: https://***/ictnlp/MonoAttn-Transducer. Copyright © 2024, The Authors. All rights reserved.

关键词： Decoding

A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Zhengrui Fang, Qingkai Zhang, Shaolei Guo, Shoutao Feng, Yang Zhang, Min Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences China Key Laboratory of AI Safety Chinese Academy of Sciences China University of Chinese Academy of Sciences China School of Future Science and Engineering Soochow University China

Simultaneous translation models play a crucial role in facilitating communication. However, existing research primarily focuses on text-to-text or speech-to-text models, necessitating additional cascade components to achieve speech-to-speech translation. These pipeline methods suffer from error propagation and accumulate delays in each cascade component, resulting in reduced synchronization between the speaker and listener. To overcome these challenges, we propose a novel non-autoregressive generation framework for simultaneous speech translation (NAST-S2x1), which integrates speech-to-text and speech-to-speech tasks into a unified end-to-end framework. We develop a non-autoregressive decoder capable of concurrently generating multiple text or acoustic unit tokens upon receiving fixed-length speech chunks. The decoder can generate blank or repeated tokens and employ CTC decoding to dynamically adjust its latency. Experimental results show that NAST-S2x outperforms state-of-the-art models in both speech-to-text and speech-to-speech tasks. It achieves high-quality simultaneous interpretation within a delay of less than 3 seconds and provides a 28× decoding speedup in offline generation. Copyright © 2024, The Authors. All rights reserved.

关键词： Decoding

GNSS Signal Acquisition Based on Compressive Sensing and Improved Measurement Matrix

学校读者我要写书评

暂无评论

GNSS Signal Acquisition Based on Compressive Sensing and Imp...

International Conference on Electronic Engineering and Informatics (EEI)

作者： Wenjun Zhang Jie Chen Yincun Guo Yongchen Zhao School of Communication and Information Engineering Shanghai University Shanghai China Key Laboratory of Intelligent Computing for Spatial Heterogeneous Network Shanghai Aerospace Electronic Technology institute Shanghai Shanghai Aerospace Electronic Technology Institute Shanghai China

ISBN: (数字)9798350353594

ISBN: (纸本)9798350353600

The main task of Global Navigation Satellite System (GNSS) receiver is to obtain satellite signals, but the signal acquisition operations performed in this process consume a large amount of hardware resources. Compressed sensing is a new signal sampling theory that can reduce the calculation amount of the acquisition process, but it will also bring a certain signal-to-noise ratio loss. In order to improve the acquisition performance on the basis of reducing the amount of signal acquisition calculations, this paper proposes an improved measurement matrix compressed sensing signal partial matched filter-fast Fourier transform (PMF -FFT) acquisition method, which improves the Gaussian random matrix through singular value decomposition and improves the incoherence of the measurement matrix. Simulation results show that this method significantly reduces the amount of acquisition calculations without greatly reducing the acquisition performance.

关键词： Global navigation satellite system Phase measurement Simulation Signal sampling Receivers Hardware Matrix decomposition Compressed sensing Singular value decomposition Signal to noise ratio

PostoMETRO: Pose Token Enhanced Mesh Transformer for Robust 3D Human Mesh Recovery

学校读者我要写书评

暂无评论

PostoMETRO: Pose Token Enhanced Mesh Transformer for Robust ...

IEEE Workshop on Applications of Computer Vision (WACV)

作者： Wendi Yang Zihang Jiang Shang Zhao S. Kevin Zhou Division of Life Sciences and Medicine School of Biomedical Engineering University of Science and Technology of China (USTC) Hefei Anhui China Center for Medical Imaging Robotics Analytic Computing & Learning (MIRACLE) Suzhou Institute for Advance Research USTC Suzhou Jiangsu China Key Laboratory of Precision and Intelligent Chemistry USTC Hefei Anhui China Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS) Institute of Computing Technology CAS

ISBN: (数字)9798331510831

ISBN: (纸本)9798331510848

With the recent advancements in single-image-based 3D human pose and shape estimation (3DHPSE), there is a growing amount of works that can achieve good results on standard benchmarks but struggle to yield accurate hu-man mesh in extreme scenarios like occlusion. Previous works propose to leverage 2D poses to help 3D HPSE model improve performance under occlusion, but usually rely on manual design to integrate 2D poses and only aim for specific kinds of occlusion. In this paper, we present PostoMETRO (Pose token enhanced MEsh TRansfOrmer), which integrates 2D pose prior knowledge as tokens into transformers to improve model's performance under occlusion. Using a VQ- VAE-based pose tokenizer; we efficiently represent 2D poses as tokens andfeed them to transformers together with image tokens. Subsequently, these tokens are queried by vertex andJoint tokens to decode 3D coordinates of mesh vertices and human Joints. Our proposed 2D poses integration strategy is manual-design-free and suitable for various kinds of occlusion. Experiments on both standard and occlusion-specific benchmarks demonstrate the effectiveness of PostoMETRO. Code will be made available 1 1 https://***/PostoMETRO/PostoMETRO-Paper.

关键词： Solid modeling Computer vision Three-dimensional displays Shape Estimation Manuals Benchmark testing Transformers Robustness Standards

Exploiting Robust Model Watermarking Against the Model Fine-Tuning Attack via Flat Minima Aware Optimizers

学校读者我要写书评

暂无评论

Exploiting Robust Model Watermarking Against the Model Fine-...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Dongdong Lin Yue Li Bin Li Jiwu Huang Guangdong Provincial Key Laboratory of Intelligent Information Processing Shenzhen University Shenzhen China Shenzhen Key Laboratory of Media Security Shenzhen University Shenzhen China SZU-AFS Joint Innovation Center for AI Technology Shenzhen University Shenzhen China Xiamen Key Laboratory of Data Security and Blockchain Technology Huaqiao University Xiamen China Guangdong Laboratory of Machine Perception and Intelligent Computing Shenzhen MSU-BIT University Shenzhen China

ISBN: (数字)9798350368741

ISBN: (纸本)9798350368758

With the rapid advancement of deep neural networks (DNNs), model watermarking has emerged as a widely adopted technique for safeguarding model copyrights. A prevalent method involves utilizing a watermark decoder to retrieve watermark bits from generated outputs, but such methods are often vulnerable to model fine-tuning attacks. Traditionally, this challenge is mitigated through adversarial training or data augmentation, both of which significantly increase the computational burden. In this paper, we present a solution employing Flat Minima Aware (FMA) optimizers to bolster the robustness of model watermarking without requiring additional training data. By optimizing the watermark loss with flat minima awareness, our approaches significantly enhance the robustness of watermarks against the model fine-tuning attack. Comprehensive experiments have demonstrated our method’s superior ability to preserve watermark integrity. These findings suggest that this innovative optimization strategy offers a robust and efficient pathway for protecting models, thereby contributing to more secure and reliable model copyright protection mechanisms.

关键词： Training Resistance Computational modeling Training data Watermarking Signal processing Diffusion models Robustness Speech processing Optimization

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Pengxiang Gao, Zhi Zhang, Bofei Yuan, Tao Wu, Yuwei Harandi, Mehrtash Jia, Yunde Zhu, Song-Chun Li, Qing Beijing Key Laboratory of Intelligent Information Technology School of Computer Science & Technology Beijing Institute of Technology China State Key Laboratory of General Artificial Intelligence BIGAI China State Key Laboratory of General Artificial Intelligence Peking University China Department of Electrical and Computer System Engineering Monash University Australia Guangdong Laboratory of Machine Perception and Intelligent Computing Shenzhen MSU-BIT University China Department of Automation Tsinghua University China

Vision language models (VLMs) have achieved impressive progress in diverse applications, becoming a prevalent research direction. In this paper, we build FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn conversations that are derived from 27 source datasets, empowering VLMs to spontaneously refine their responses based on user feedback across diverse tasks. To scale up the data collection, FIRE is collected in two components: FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a benchmark to comprehensively evaluate the feedback-refining capability of VLMs, which contains 11K feedback-refinement conversations as the test data, two evaluation settings, and a model to provide feedback for VLMs. We develop the FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows remarkable feedback-refining capability on FIRE-Bench and outperforms untrained VLMs by 50%, making more efficient user-agent interactions and underscoring the significance of the FIRE dataset. Copyright © 2024, The Authors. All rights reserved.

关键词： Fireclay

MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Huang, Langlin Bu, Mengyu Feng, Yang Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences China Key Laboratory of AI Safety Chinese Academy of Sciences China University of Chinese Academy of Sciences China Washington University St. Louis United States

Byte-based machine translation systems have shown significant potential in massively multilingual settings. Unicode encoding, which maps each character to specific byte(s), eliminates the emergence of unknown words, even in new languages. This avoids out-of-vocabulary risk in multilingual translation and enables broad language scalability. However, byte-level tokenization results in sequences that are hard to interpret due to limited semantic information per byte. Local contextualization has proven effective in assigning initial semantics to tokens, improving sentence comprehension. Nevertheless, variations in encoding rules across languages necessitate an adaptive approach for effective contextualization. To this end, we propose Mixture of Contextualization Experts (MoCE), adaptively selecting and mixing attention heads, which are treated as contextualization experts. This enhances the flexibility of contextualization scales and allows models to search for better contextualization combinations. Experiment results show that our method outperforms existing methods without extensive manual adjustment of hyper-parameters and surpasses subword-based models with fewer parameters in Ted-59 dataset. Our code is available at https://***/ictnlp/MoCE. © 2024, CC BY-NC-ND.

关键词： Encoding (symbols)