检索结果-内蒙古大学图书馆

High-Fidelity Editable Portrait Synthesis with 3D GAN Inversion

学校读者我要写书评

暂无评论

High-Fidelity Editable Portrait Synthesis with 3D GAN Invers...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Jindong Xie Jiachen Liu Yupei Lin Jinbao Wang Xianxu Hou Linlin Shen Computer Vision Institute School of Computer Science and Software Engineering Shenzhen University National Engineering Laboratory for Big Data System Computing Technology Shenzhen University School of Information Engineering Guangdong University of Technology Guangzhou China Guangdong Provincial Key Laboratory of Intelligent Information Processing School of AI and Advanced Computing Xi’an Jiaotong-Liverpool University

ISBN: (数字)9798350368741

ISBN: (纸本)9798350368758

The 3D generative adversarial network (GAN) inversion converts an image into 3D representation to attain high-fidelity reconstruction and facilitate realistic image manipulation within the 3D latent space. However, previous approaches face challenges regarding the trade-off between the reconstruction ability and editability. That is, reversing a real-world image to a low-dimensional latent code would inevitably lead to information loss, and achieving a near-perfect reconstruction using high-rate triplane representation often limits the ability to manipulate the image freely in the latent space. To address these issues, we propose a novel latent conditioning encoder-based framework with the alignment between the low-dimensional latent and high-dimensional triplane. A non-semantic guided editing strategy bridges the intrinsic relation between the latent condition and triplane generation, making it possible to edit the high-dimensional representation by latent manipulation. As a result, our method can achieve high-fidelity reconstruction and editing simultaneously by directly controlling the latent code. Experimental results demonstrate that our approach excels in reconstruction and editing quality compared to previous 3D inversion methods. Furthermore, our method can also edit even real faces with large poses and out-of-domain cases.

关键词： Bridges Three-dimensional displays Codes Signal processing Generative adversarial networks Vectors Acoustics Speech processing Image reconstruction Faces

Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features 38

学校读者我要写书评

暂无评论

Not All Diffusion Model Activations Have Been Evaluated as D...

38th Conference on Neural information Processing Systems, NeurIPS 2024

作者： Meng, Benyuan Xu, Qianqian Wang, Zitai Cao, Xiaochun Huang, Qingming Institute of Information Engineering CAS China School of Cyber Security University of Chinese Academy of Sciences China Key Lab. of Intelligent Information Processing Institute of Computing Technology CAS China Peng Cheng Laboratory China School of Cyber Science and Tech. Shenzhen Campus of Sun Yat-sen University China School of Computer Science and Tech. University of Chinese Academy of Sciences China Key Laboratory of Big Data Mining and Knowledge Management CAS China

Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at this url. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Pengfei Min, Weiqing Fu, Chaoran Jin, Ying Huang, Mingyu Li, Xiangyang Mei, Shuhuan Jiang, Shuqiang The Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing100190 China The University of Chinese Academy of Sciences Beijing100049 China The Institute of Intelligent Computing Technology Chinese Academy of Sciences Suzhou215124 China Technology Co. Ltd. Suzhou215024 China

Food is foundational to human life, serving not only as a source of nourishment but also as a cornerstone of cultural identity and social interaction. As the complexity of global dietary needs and preferences grows, food intelligence is needed to enable food perception and reasoning for various tasks, ranging from recipe generation and dietary recommendation to diet-disease correlation discovery and understanding. Towards this goal, for powerful capabilities across various domains and tasks in Large Language Models (LLMs), we introduce Food-oriented LLM FoodSky to comprehend food data through perception and reasoning. Considering the complexity and typicality of Chinese cuisine, we first construct one comprehensive Chinese food corpus FoodEarth from various authoritative sources, which can be leveraged by FoodSky to achieve deep understanding of food-related data. We then propose Topic-based Selective State Space Model (TS3M) and the Hierarchical Topic Retrieval Augmented Generation (HTRAG) mechanism to enhance FoodSky in capturing fine-grained food semantics and generating context-aware food-relevant text, respectively. Our extensive evaluations demonstrate that FoodSky significantly outperforms general-purpose LLMs in both chef and dietetic examinations, with an accuracy of 67.2% and 66.4% on the Chinese National Chef Exam and the National Dietetic Exam, respectively. FoodSky not only promises to enhance culinary creativity and promote healthier eating patterns, but also sets a new standard for domain-specific LLMs that address complex real-world issues in the food domain. An online demonstration of FoodSky is available at http://222.92.101.211:8200. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantics

MULTI-MODAL AGENT TUNING: BUILDING A VLM-DRIVEN AGENT FOR EFFICIENT TOOL USAGE

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gao, Zhi Zhang, Bofei Li, Pengxiang Ma, Xiaojian Yuan, Tao Fan, Yue Wu, Yuwei Jia, Yunde Zhu, Song-Chun Li, Qing School of Intelligence Science and Technology Peking University China State Key Laboratory of General Artificial Intelligence BIGAI China Beijing Key Laboratory of Intelligent Information Technology Beijing Institute of Technology China Guangdong Laboratory of Machine Perception and Intelligent Computing Shenzhen MSU-BIT University China Department of Automation Tsinghua University China

The advancement of large language models (LLMs) prompts the development of multi-modal agents, which are used as a controller to call external tools, providing a feasible way to solve practical tasks. In this paper, we propose a multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes a vision-language model (VLM) as the controller for powerful tool-usage reasoning. To preserve the data quality, we prompt the GPT-4o mini model to generate queries, files, and trajectories, followed by query-file and trajectory verifiers. Based on the data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks with trajectories of tool usage. Then, we develop the T3-Agent via Trajectory Tuning on VLMs for Tool usage using MM-Traj. Evaluations on the GTA and GAIA benchmarks show that the T3-Agent consistently achieves improvements on two popular VLMs: MiniCPM-V-8.5B and Qwen2-VL-7B, which outperforms untrained VLMs by 20%, showing the effectiveness of the proposed data synthesis pipeline, leading to high-quality data for tool-usage capabilities. Copyright © 2024, The Authors. All rights reserved.

关键词： Visual languages

Integrating Multi-scale Contextualized information for Byte-based Neural Machine Translation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Huang, Langlin Feng, Yang Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences China Key Laboratory of AI Safety Chinese Academy of Sciences China University of Chinese Academy of Sciences China

Subword tokenization is a common method for vocabulary building in Neural Machine Translation (NMT) models. However, increasingly complex tasks have revealed its disadvantages. First, a vocabulary cannot be modified once it is learned, making it hard to adapt to new words. Second, in multilingual translation, the imbalance in data volumes across different languages spreads to the vocabulary, exacerbating translations involving low-resource languages. While byte-based tokenization addresses these issues, byte-based models struggle with the low information density inherent in UTF-8 byte sequences. Previous works enhance token semantics through local contextualization but fail to select an appropriate contextualizing scope based on the input. Consequently, we propose the Multi-Scale Contextualization (MSC) method, which learns contextualized information of varying scales across different hidden state dimensions. It then leverages the attention module to dynamically integrate the multi-scale contextualized information. Experiments show that MSC significantly outperforms subword-based and other byte-based methods in both multilingual and out-of-domain scenarios. We have uploaded the code to github1 © 2024, CC BY-NC-ND.

关键词： Neural machine translation

Cross-Modality Perturbation Synergy Attack for Person Re-identification 38

学校读者我要写书评

暂无评论

Cross-Modality Perturbation Synergy Attack for Person Re-ide...

38th Conference on Neural information Processing Systems, NeurIPS 2024

作者： Gong, Yunpeng Zhong, Zhun Qu, Yansong Luo, Zhiming Ji, Rongrong Jiang, Min School of Informatics Xiamen University China School of Computer Science and Information Engineering Hefei University of Technology China The Department of Artificial Intelligence Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Key Laboratory of Digital Protection and Intelligent Processing of Intangible CulturalHeritage of Fujian and Taiwan Ministry of Culture and Tourism Xiamen University Fujian Xiamen361005 China

In recent years, there has been significant research focusing on addressing security concerns in single-modal person re-identification (ReID) systems that are based on RGB images. However, the safety of cross-modality scenarios, which are more commonly encountered in practical applications involving images captured by infrared cameras, has not received adequate attention. The main challenge in cross-modality ReID lies in effectively dealing with visual differences between different modalities. For instance, infrared images are typically grayscale, unlike visible images that contain color information. Existing attack methods have primarily focused on the characteristics of the visible image modality, overlooking the features of other modalities and the variations in data distribution among different modalities. This oversight can potentially undermine the effectiveness of these methods in image retrieval across diverse modalities. This study represents the first exploration into the security of cross-modality ReID models and proposes a universal perturbation attack specifically designed for cross-modality ReID. This attack optimizes perturbations by leveraging gradients from diverse modality data, thereby disrupting the discriminator and reinforcing the differences between modalities. We conducted experiments on three widely used cross-modality datasets, namely RegDB, SYSU, and LLCM. The results not only demonstrate the effectiveness of our method but also provide insights for future improvements in the robustness of cross-modality ReID systems. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

Hierarchical Policies of Subgoals for Safe Deep Reinforcement Learning 2nd

学校读者我要写书评

暂无评论

Hierarchical Policies of Subgoals for Safe Deep Reinforcem...

2nd International Conference on Ubiquitous Security, UbiSec 2022

作者： Yu, Fumin Gao, Feng Yuan, Yao Xing, Xiaofei Dai, Yinglong College of Information Science and Engineering Hunan Normal University Changsha410081 China School of Computer Science and Cyber Engineering Guangzhou University Guangzhou510006 China College of Liberal Arts and Sciences National University of Defense Technology Changsha410073 China Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing Changsha410081 China

ISBN: (纸本)9789819902712

Reinforcement learning is a machine learning method that relies on the agent to learn by trial and error to solve decision optimization problems. It is well known that an agent based on deep reinforcement learning in complex environments is difficult to train. Moreover, the agent will generate unsafe and strange actions due to the lack of sufficient reward feedback from the environment. To make the agent converge to a better policy and make its behavior safer and more controllable under sparse rewards, we propose a subgoal embedding method based on prior knowledge and hierarchical strategy that can make the training process converge faster. The subgoal embedding method can be combined with existing reinforcement learning methods. In this paper, we combine the subgoal embedding method with REINFORCE algorithm and PPO(Proximal Policy Optimization) algorithm to test the method in the MiniGrid-DoorKey game environment of the gym platform. The experiments demonstrate the effectiveness of the subgoal embedding method. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

关键词： Reinforcement learning

Key Factor-Based Initialization for 3D Hand Tracking

学校读者我要写书评

暂无评论

Chinese Journal of Electronics 2023年第4期22卷 751-756页

作者： FENG Zhiquan YANG Bo XU Tao ZHENG Yanwei GAO Jian School of Information Science and Engineering University of Jinun Jinan China Shandong Provincial Key Laboratory of Network based Intelligent Computing Jinan China University of Science and Technology Qingdao China

The discovery of three-dimensional (3D) hand models corresponding to the user's 3D hand pose in initial frames is significant in 3D human hand tracking and interacting. This study proposes an approach to initialize 3D hand gestures for the user to have an easier, more pleasurable, and satisfactory experience. This paper covers the following points. First, a new approach to selecting a human hand gesture from the hand postures database is presented. Second, both techniques of visualization and human-computer interaction are used in the initialization process, through which the 3D human hand model is fine tuned time after time until the required accuracy is satisfied. Lastly, the proposed initialization method is applied to our online virtual assembling system. We introduce a key factor to improve time cost of initialization. Our experimental results show that the proposed approach is not only fast, accurate, and robust but also direct, natural, and convenient for operators to handle.

关键词： Human computer interaction Solid modeling Visualization Three-dimensional displays Costs Databases Computational modeling

StylizedGS: Controllable Stylization for 3D Gaussian Splatting

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Dingxi Chen, Zhuoxun Yuan, Yu-Jie Zhang, Fang-Lue He, Zhenliang Shan, Shiguang Gao, Lin The University of Chinese Academy of Sciences Beijing China The Beijing Key Laboratory of Mobile Computing and Pervasive Device Institute of Computing Technology Chinese Academy of Sciences Beijing China Victoria University of Wellington New Zealand The Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing China

With the rapid development of XR, 3D generation and editing are becoming more and more important, among which, stylization is an important tool of 3D appearance editing. It can achieve consistent 3D artistic stylization given a single reference style image and thus is a user-friendly editing way. However, recent NeRF-based 3D stylization methods face efficiency issues that affect the actual user experience and the implicit nature limits its ability to transfer the geometric pattern styles. Additionally, the ability for artists to exert flexible control over stylized scenes is considered highly desirable, fostering an environment conducive to creative exploration. In this paper, we introduce StylizedGS, a 3D neural style transfer framework with adaptable control over perceptual factors based on 3D Gaussian Splatting (3DGS) representation. The 3DGS brings the benefits of high efficiency. We propose a GS filter to eliminate floaters in the reconstruction which affects the stylization effects before stylization. Then the nearest neighbor-based style loss is introduced to achieve stylization by fine-tuning the geometry and color parameters of 3DGS, while a depth preservation loss with other regularizations is proposed to prevent the tampering of geometry content. Moreover, facilitated by specially designed losses, StylizedGS enables users to control color, stylized scale and regions during the stylization to possess customized capabilities. Our method can attain high-quality stylization results characterized by faithful brushstrokes and geometric consistency with flexible controls. Extensive experiments across various scenes and styles demonstrate the effectiveness and efficiency of our method concerning both stylization quality and inference FPS. Copyright © 2024, The Authors. All rights reserved.

关键词： Efficiency