检索结果-内蒙古大学图书馆

3DPortraitGAN: Learning One-Quarter Headshot 3D GANs from a Single-View Portrait Dataset with Diverse Body Poses

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wu, Yiqian Xu, Hao Tang, Xiangjun Fu, Hongbo Jin, Xiaogang State Key Lab of CAD&CG Zhejiang University China City University of Hong Kong China ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab China

3D-aware face generators are typically trained on 2D real-life face image datasets that primarily consist of near-frontal face data, and as such, they are unable to construct one-quarter headshot 3D portraits with complete head, neck, and shoulder geometry. Two reasons account for this issue: First, existing facial recognition methods struggle with extracting facial data captured from large camera angles or back views. Second, it is challenging to learn a distribution of 3D portraits covering the one-quarter headshot region from single-view data due to significant geometric deformation caused by diverse body poses. To this end, we first create the dataset 360◦-Portrait-HQ (360◦PHQ for short) which consists of high-quality single-view real portraits annotated with a variety of camera parameters (the yaw angles span the entire 360◦ range) and body poses. We then propose 3DPortraitGAN, the first 3D-aware one-quarter headshot portrait generator that learns a canonical 3D avatar distribution from the 360◦PHQ dataset with body pose self-learning. Our model can generate view-consistent portrait images from all camera angles with a canonical one-quarter headshot 3D representation. Our experiments show that the proposed framework can accurately predict portrait body poses and generate view-consistent, realistic portrait images with complete geometry from all camera angles. We will release our 360◦PHQ dataset, code and pre-trained models for reproducible research. Copyright © 2023, The Authors. All rights reserved.

关键词： Geometry

LPFF: A Portrait Dataset for Face Generators Across Large Poses

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wu, Yiqian Zhang, Jing Fu, Hongbo Jin, Xiaogang State Key Lab of Cad&cg Zhejiang University China ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab China City University of Hong Kong Hong Kong

The creation of 2D realistic facial images and 3D face shapes using generative networks has been a hot topic in recent years. Existing face generators exhibit exceptional performance on faces in small to medium poses (with respect to frontal faces) but struggle to produce realistic results for large poses. The distorted rendering results on large poses in 3D-aware generators further show that the generated 3D face shapes are far from the distribution of 3D faces in reality. We find that the above issues are caused by the training dataset's pose imbalance. In this paper, we present LPFF, a large-pose Flickr face dataset comprised of 19,590 high-quality real large-pose portrait images. We utilize our dataset to train a 2D face generator that can process large-pose face images, as well as a 3D-aware generator that can generate realistic human face geometry. To better validate our pose-conditional 3Daware generators, we develop a new FID measure to evaluate the 3D-level performance. Through this novel FID measure and other experiments, we show that LPFF can help 2D face generators extend their latent space and better manipulate the large-pose data, and help 3D-aware face generators achieve better view consistency and more realistic 3D reconstruction results. Copyright © 2023, The Authors. All rights reserved.

关键词： Large dataset

RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Pan, Xiaoyu Zheng, Bowen Jiang, Xinwei Zeng, Zijiao Kou, Qilong Wang, He Jin, Xiaogang State Key Lab of CAD & CG Zhejiang University China ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab China Tencent Games Digital Content Technology Center China Co. LTD China Department of Computer Science UCL Centre for Artificial Intelligence University College London United Kingdom

Optical motion capture (MoCap) is the "gold standard" for accurately capturing full-body motions. To make use of raw MoCap point data, the system labels the points with corresponding body part locations and solves the fullbody motions. However, MoCap data often contains mislabeling, occlusion and positional errors, requiring extensive manual correction. To alleviate this burden, we introduce RoMo, a learning-based framework for robustly labeling and solving raw optical motion capture data. In the labeling stage, RoMo employs a divide-and-conquer strategy to break down the complex full-body labeling challenge into manageable subtasks: alignment, full-body segmentation and part-specific labeling. To utilize the temporal continuity of markers, RoMo generates marker tracklets using a K-partite graph-based clustering algorithm, where markers serve as nodes, and edges are formed based on positional and feature similarities. For motion solving, to prevent error accumulation along the kinematic chain, we introduce a hybrid inverse kinematic solver that utilizes joint positions as intermediate representations and adjusts the template skeleton to match estimated joint positions. We demonstrate that RoMo achieves high labeling and solving accuracy across multiple metrics and various datasets. Extensive comparisons show that our method outperforms state-of-the-art research methods. On a real dataset, RoMo improves the F1 score of hand labeling from 0.94 to 0.98, and reduces joint position error of body motion solving by 25%. Furthermore, RoMo can be applied in scenarios where commercial systems are inadequate. The code and data for RoMo are available at https:// ***/non-void/RoMo. Copyright © 2024, The Authors. All rights reserved.

关键词： Motion capture

A Locality-based Neural Solver for Optical Motion Capture

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Pan, Xiaoyu Zheng, Bowen Jiang, Xinwei Xu, Guanglong Gu, Xianli Li, Jingxiang Kou, Qilong Wang, He Shao, Tianjia Zhou, Kun Jin, Xiaogang State Key Lab of CAD & CG Zhejiang University ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab Hangzhou China Tencent Games Digital Content Technology Center Shanghai China Co. LTD Shenzhen China London United Kingdom

We present a novel locality-based learning method for cleaning and solving optical motion capture data. Given noisy marker data, we propose a new heterogeneous graph neural network which treats markers and joints as different types of nodes, and uses graph convolution operations to extract the local features of markers and joints and transform them to clean motions. To deal with anomaly markers (e.g. occluded or with big tracking errors), the key insight is that a marker’s motion shows strong correlations with the motions of its immediate neighboring markers but less so with other markers, a.k.a. locality, which enables us to efficiently fill missing markers (e.g. due to occlusion). Additionally, we also identify marker outliers due to tracking errors by investigating their acceleration profiles. Finally, we propose a training regime based on representation learning and data augmentation, by training the model on data with masking. The masking schemes aim to mimic the occluded and noisy markers often observed in the real data. Finally, we show that our method achieves high accuracy on multiple metrics across various datasets. Extensive comparison shows our method outperforms state-of-the-art methods in terms of prediction accuracy of occluded marker position error by approximately 20%, which leads to a further error reduction on the reconstructed joint rotations and positions by 30%. The code and data for this paper are available at https://***/non-void/LocalMoCap. Copyright © 2023, The Authors. All rights reserved.

关键词： Machine learning

Decoupling Contact for Fine-Grained Motion Style Transfer

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Tang, Xiangjun Wu, Linjun Wang, He Wu, Yiqian Hu, Bo Li, Songnan Gong, Xu Liao, Yuchen Kou, Qilong Jin, Xiaogang State Key Lab of CAD&CG Zhejiang University China Department of Computer Science UCL Centre for Artificial Intelligence University College London United Kingdom Tencent Technology Co. Ltd. China State Key Lab of CAD&CG Zhejiang University ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab China

Motion style transfer changes the style of a motion while retaining its content and is useful in computer animations and games. Contact is an essential component of motion style transfer that should be controlled explicitly in order to express the style vividly while enhancing motion naturalness and quality. However, it is unknown how to decouple and control contact to achieve fine-grained control in motion style transfer. In this paper, we present a novel style transfer method for fine-grained control over contacts while achieving both motion naturalness and spatial-temporal variations of style. Based on our empirical evidence, we propose controlling contact indirectly through the hip velocity, which can be further decomposed into the trajectory and contact timing, respectively. To this end, we propose a new model that explicitly models the correlations between motions and trajectory/contact timing/style, allowing us to decouple and control each separately. Our approach is built around a motion manifold, where hip controls can be easily integrated into a Transformer-based decoder. It is versatile in that it can generate motions directly as well as be used as post-processing for existing methods to improve quality and contact controllability. In addition, we propose a new metric that measures a correlation pattern of motions based on our empirical evidence, aligning well with human perception in terms of motion naturalness. Based on extensive evaluation, our method outperforms existing methods in terms of style expressivity and motion quality. Copyright © 2024, The Authors. All rights reserved.

关键词： Computer games

Real-time Controllable Motion Transition for Characters

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Tang, Xiangjun Wang, He Hu, Bo Gong, Xu Yi, Ruifan Kou, Qilong Jin, Xiaogang State Key Lab of CAD&CG Zhejiang University ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab China University of Leeds United Kingdom Co. Ltd. China

Real-time in-between motion generation is universally required in games and highly desirable in existing animation pipelines. Its core challenge lies in the need to satisfy three critical conditions simultaneously: quality, controllability and speed, which renders any methods that need offline computation (or post-processing) or cannot incorporate (often unpredictable) user control undesirable. To this end, we propose a new real-time transition method to address the aforementioned challenges. Our approach consists of two key components: motion manifold and conditional transitioning. The former learns the important low-level motion features and their dynamics;while the latter synthesizes transitions conditioned on a target frame and the desired transition duration. We first learn a motion manifold that explicitly models the intrinsic transition stochasticity in human motions via a multi-modal mapping mechanism. Then, during generation, we design a transition model which is essentially a sampling strategy to sample from the learned manifold, based on the target frame and the aimed transition duration. We validate our method on different datasets in tasks where no post-processing or offline computation is allowed. Through exhaustive evaluation and comparison, we show that our method is able to generate high-quality motions measured under multiple metrics. Our method is also robust under various target frames (with extreme cases). © 2022, CC BY.

关键词： Animation

Predicting Loose-Fitting Garment Deformations Using Bone-Driven Motion Networks

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Pan, Xiaoyu Mai, Jiaming Jiang, Xinwei Tang, Dongxue Li, Jingxiang Shao, Tianjia Zhou, Kun Jin, Xiaogang Manocha, Dinesh University of Maryland United States Tencent NExT Studios China State Key Lab of CAD&CG Zhejiang University ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab China State Key Lab of CAD&CG Zhejiang University China

We present a learning algorithm that uses bone-driven motion networks to predict the deformation of loose-fitting garment meshes at interactive rates. Given a garment, we generate a simulation database and extract virtual bones from simulated mesh sequences using skin decomposition. At runtime, we separately compute low- and high-frequency deformations in a sequential manner. The low-frequency deformations are predicted by transferring body motions to virtual bones’ motions, and the high-frequency deformations are estimated leveraging the global information of virtual bones’ motions and local information extracted from low-frequency meshes. In addition, our method can estimate garment deformations caused by variations of the simulation parameters (e.g., fabric’s bending stiffness) using an RBF kernel ensembling trained networks for different sets of simulation parameters. Through extensive comparisons, we show that our method outperforms state-of-the-art methods in terms of prediction accuracy of mesh deformations by about 20% in RMSE and 10% in Hausdorff distance and STED. The code and data are available at https://***/non-void/VirtualBones. Copyright © 2022, The Authors. All rights reserved.

关键词： Deformation

Predicting Loose-Fitting Garment Deformations Using Bone-Driven Motion Networks 22

学校读者我要写书评

暂无评论

Predicting Loose-Fitting Garment Deformations Using Bone-Dri...

ACM SIGGRAPH 2022 Conference Proceedings

作者： Xiaoyu Pan Jiaming Mai Xinwei Jiang Dongxue Tang Jingxiang Li Tianjia Shao Kun Zhou Xiaogang Jin Dinesh Manocha State Key Lab of CAD&CG Zhejiang Univerisity China and ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab China Tencent NExT Studios China State Key Lab of CAD&CG Zhejiang Univerisity China University of Maryland United States of America

ISBN: (纸本)9781450393379

关键词： cloth animation skinning decomposition deep learning

FASTSWARM: A data-driven framework for real-time flying insect swarm simulation

学校读者我要写书评

暂无评论

FASTSWARM: A data-driven framework for real-time flying inse...

作者： Xiang, Wei Yao, Xinran Wang, He Jin, Xiaogang State Key Lab of CAD&CG Zhejiang University Hangzhou China School of Computing University of Leeds Leeds United Kingdom ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab Hangzhou China

Insect swarms are common phenomena in nature and therefore have been actively pursued in computer animation. Realistic insect swarm simulation is difficult due to two challenges: high-fidelity behaviors and large scales, which make the simulation practice subject to laborious manual work and excessive trial-and-error processes. To address both challenges, we present a novel data-driven framework, FASTSWARM, to model complex behaviors of flying insects based on real-world data and simulate plausible animations of flying insect swarms. FASTSWARM has a linear time complexity and achieves real-time performance for large swarms. The high-fidelity behavior model of FASTSWARM explicitly takes into consideration the most common behaviors of flying insects, including the interactions among insects such as repulsion and attraction, self-propelled behaviors such as target following and obstacle avoidance, and other characteristics such as random movements. To achieve scalability, an energy minimization problem is formed with different behaviors modeled as energy terms, where the minimizer is the desired behavior. The minimizer is computed from the real-world data, which ensures the plausibility of the simulation results. Extensive simulation results and evaluations show that FASTSWARM is versatile in simulating various swarm behaviors, high fidelity measured by various metrics, easily controllable in inducing user controls and highly scalable. © 2020 John Wiley & Sons, Ltd.

关键词： Animation