检索结果-内蒙古大学图书馆

Instruct-NeuralTalker: Editing Audio-Driven Talking Radiance Fields with Instructions

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Sun, Yuqi He, Ruian Tan, Weimin Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

Recent neural talking radiance field methods have shown great success in photorealistic audio-driven talking face synthesis. In this paper, we propose the first novel interactive framework that utilizes human instructions to edit talking radiance fields to achieve personalized talking face generation. Given a short speech video, we first build an efficient talking radiance field and then apply the latest conditional diffusion model to edit images with human instructions during the optimization. To ensure audio-lip synchronization during the editing process, we propose a progressive dataset updating strategy to ensure successful editing while preventing distortion of lip shapes. We also introduce a lightweight refinement network for complementing image details and achieving controllable detail generation in the final rendered image. Our method enables real-time rendering at up to 30FPS on consumer hardware. Multiple metrics and user verification show that our approach significantly improves rendering quality compared to state-of-the-art methods. © 2023, CC BY-NC-ND.

关键词： Rendering (computer graphics)

Fine-Grained Blind Face Inpainting with 3D Face Component Disentanglement

学校读者我要写书评

暂无评论

Fine-Grained Blind Face Inpainting with 3D Face Component Di...

International Conference on Acoustics, Speech, and Signal processing (ICASSP)

作者： Yu Bai Ruian He Weimin Tan Bo Yan Yangle Lin School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

Inpainting is a task to restore occlusion or other corruption on images. However, previous works require mask of the occluded area to restore the occluded image, which is inconvenient for application. Blind face inpainting aims to automatically restore the occluded face without position information of the corrupt region. In this paper, we propose a novel fine-grained blind face inpainting framework, combining 3D face components disentanglement with generative network. Canonical face texture and shape disentangled by unsupervised 3D face model is restored separately to get occlusion-free rendered result. Finally, the pixel-to-pixel generative module utilize the occlusion image and the coarse de-occlusion face to get refined inpainted result. We also build up a new dataset called CelebO-3D which consists of occluded face images synthesized with 3D occlusion and rendered by 3D face model. Extensive experiments show that the proposed method is effective and robust in face blind inpainting both in synthesized and real images. Extensive evaluations and comparison with previous methods also show our superior effectiveness and lightweight architecture.

关键词： Solid modeling Three-dimensional displays Shape Speech enhancement Signal processing Acoustics Image restoration

MVFlow: Deep Optical Flow Estimation of Compressed Videos with Motion Vector Prior

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhou, Shili Jiang, Xuhao Tan, Weimin He, Ruian Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

In recent years, many deep learning-based methods have been proposed to tackle the problem of optical flow estimation and achieved promising results. However, they hardly consider that most videos are compressed and thus ignore the pre-computed information in compressed video streams. Motion vectors, one of the compression information, record the motion of the video frames. They can be directly extracted from the compression code stream without computational cost and serve as a solid prior for optical flow estimation. Therefore, we propose an optical flow model, MVFlow, which uses motion vectors to improve the speed and accuracy of optical flow estimation for compressed videos. In detail, MVFlow includes a key Motion-Vector Converting Module, which ensures that the motion vectors can be transformed into the same domain of optical flow and then be utilized fully by the flow estimation module. Meanwhile, we construct four optical flow datasets for compressed videos containing frames and motion vectors in pairs. The experimental results demonstrate the superiority of our proposed MVFlow, which can reduce the AEPE by 1.09 compared to existing models or save 52% time to achieve similar accuracy to existing models. Copyright © 2023, The Authors. All rights reserved.

关键词： Optical flows

Uncertainty-Guided Spatial Pruning Architecture for Efficient Frame Interpolation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cheng, Ri Jiang, Xuhao He, Ruian Zhou, Shili Tan, Weimin Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

The video frame interpolation (VFI) model applies the convolution operation to all locations, leading to redundant computations in regions with easy motion. We can use dynamic spatial pruning method to skip redundant computation, but this method cannot properly identify easy regions in VFI tasks without supervision. In this paper, we develop an Uncertainty-Guided Spatial Pruning (UGSP) architecture to skip redundant computation for efficient frame interpolation dynamically. Specifically, pixels with low uncertainty indicate easy regions, where the calculation can be reduced without bringing undesirable visual results. Therefore, we utilize uncertainty-generated mask labels to guide our UGSP in properly locating the easy region. Furthermore, we propose a self-contrast training strategy that leverages an auxiliary non-pruning branch to improve the performance of our UGSP. Extensive experiments show that UGSP maintains performance but reduces FLOPs by 34%/52%/30% compared to baseline without pruning on Vimeo90K/UCF101/MiddleBury datasets. In addition, our method achieves state-of-the-art performance with lower FLOPs on multiple benchmarks. Copyright © 2023, The Authors. All rights reserved.

关键词： Benchmarking

Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yuan, Zheng Zhang, Jie Wang, Yude Shan, Shiguang Chen, Xilin Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing100190 China University of Chinese Academy of Sciences Beijing100049 China

The attention mechanism has been proven effective on various visual tasks in recent years. In the semantic segmentation task, the attention mechanism is applied in various methods, including the case of both Convolution Neural Networks (CNN) and Vision Transformer (ViT) as backbones. However, we observe that the attention mechanism is vulnerable to patch-based adversarial attacks. Through the analysis of the effective receptive field, we attribute it to the fact that the wide receptive field brought by global attention may lead to the spread of the adversarial patch. To address this issue, in this paper, we propose a Robust Attention Mechanism (RAM) to improve the robustness of the semantic segmentation model, which can notably relieve the vulnerability against patch-based attacks. Compared to the vallina attention mechanism, RAM introduces two novel modules called Max Attention Suppression and Random Attention Dropout, both of which aim to refine the attention matrix and limit the influence of a single adversarial patch on the semantic segmentation results of other positions. Extensive experiments demonstrate the effectiveness of our RAM to improve the robustness of semantic segmentation models against various patch-based attack methods under different attack settings. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantics

TransCoop: Cooperation of Transformers and CNNs for Camouflaged Object Segmentation

学校读者我要写书评

暂无评论

TransCoop: Cooperation of Transformers and CNNs for Camoufla...

2022 IEEE International Conference on Multimedia and Expo, ICME 2022

作者： Wu, Fucai Li, Xuanya Zhang, Yuan Hu, Kai Xiangtan University Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education Xiangtan411105 China Baidu Inc. Beijing100085 China Xiangnan University Key Laboratory of Medical Imaging and Artifical Intelligence of Hunan Province Chenzhou423000 China

ISBN: (数字)9781665485630

ISBN: (纸本)9781665485630

Camouflaged object segmentation (COS) is a challenging task due to the existence of high intrinsic similarities between the object and background. To overcome this challenge, we pro-pose a new framework, called TransCoop, for COS through the cooperation of Transformers and convolutional neural net-works (CNNs). Specifically, Transformer is used to model global context and structural information for accurately positioning potential target objects. Meanwhile, a well-designed texture feature fusion module (TFFM) is used to fuse the features encoded by CNN with the shallow Transformer to fully mine the low-level features in the scene. Furthermore, we propose a noise removal module (NRM), which can eliminate the background noise of low-level features with the guidance of precise target location. Notably, our TFFM and NRM can effectively realize the interaction between Transformer and CNN features. Extensive experiments on four benchmark datasets demonstrate the superiority of our TransCoop against existing state-of-the-art methods. © 2022 IEEE.

关键词： Convolutional neural networks

Boundary Cue Guidance and Contextual Feature Mining for Glasss Segmentation

学校读者我要写书评

暂无评论

Boundary Cue Guidance and Contextual Feature Mining for Glas...

International Conference on Acoustics, Speech, and Signal processing (ICASSP)

作者： Qiquan Xiao Yuan Zhang Xuanya Li Kai Hu Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education Xiangtan University Xiangtan China Baidu Inc. Beijing China

Glass is ubiquitous in the real world, and its perception has many applications, including robot navigation and drone tracking. However, due to the transparent property of glass, the interior of a glass area can be any surrounding scene or object, which brings challenges for computer vision. Inspired by the human senses, boundary cues are one of the crucial factors for people to judge the location of glass contours. Hence, we propose a boundary cue guidance and contextual feature mining network (BCNet) to accurately and efficiently segment glass. Specifically, we first design a multi-branch boundary extraction module (MBEM) for learning accurate boundary cues combined with multi-level encoded features. Second, we propose a boundary cue guidance module (BCGM), inject the boundary cues into the representation learning, and provide constraints with object structure semantics to guide feature extraction. Besides, we design a contextual feature mining module (CFMM) to dynamically capture the contextual information of different receptive fields for the detection of different sizes and shapes of the glass. Finally, extensive experiments on two benchmark glass datasets, GDD and GSD. The results demonstrate that our BCNet achieves state-of-the-art segmentation performance against existing methods.

关键词： Representation learning Shape Semantics Neural networks Glass Benchmark testing Signal processing

Unified segment-to-segment framework for simultaneous sequence generation 23

学校读者我要写书评

暂无评论

Unified segment-to-segment framework for simultaneous sequen...

Proceedings of the 37th International Conference on Neural information processing Systems

作者： Shaolei Zhang Yang Feng Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences (ICT/CAS) and University of Chinese Academy of Sciences

Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks. Code is available at: https://***/ictnlp/Seg2Seg.

关键词：

SKT-MOT and DyTracker: A Multiobject Tracking Dataset and a Dynamic Tracker for Speed Skating Video

学校读者我要写书评

暂无评论

Scientific Programming 2023年第1期2023卷

作者： Wang, Junwu Li, Zongmin Li, Yachuan Yang, Shaobo Wang, Ben Li, Hua Qingdao266580 China Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing100190 China

Speed skating serves as a significant application domain for multiobject tracking (MOT), presenting unique challenges such as frequent occlusion, highly similar appearances, and motion blur. To address these challenges, this paper constructs an MOT dataset called SKT-MOT for speed skating and analyzes the shortcomings of existing datasets and methods. Accordingly, we propose a dynamic MOT method called DyTracker. The method builds upon the DeepSORT baseline and enhances three key modules. At the global level, we design the track dynamic management (TDM) algorithm. In the motion branch, a novel metric is proposed to evaluate occlusion and Kalman filter dynamic update (KFDU) is implemented. In the appearance branch, we account for the difference in human posture and propose the feature dynamic selection and updating (FDSU) strategy. This makes our DyTracker flexible and efficient to achieve a multiobject tracking accuracy (MOTA) of 93.70% and identification F1 (IDF1) score of 92.39% on SKT-MOT, which is a significant advantage over existing SOTA methods. To validate the generalization of our proposed module, two dynamic update modules are inserted into other methods and validated on the public dataset MOT17, and the accuracy is generally improved by 0.2%-0.6%. © 2023 Junwu Wang et al.

关键词：