检索结果-内蒙古大学图书馆

I2MD: 3D Action Representation Learning with Inter- and Intra-modal Mutual Distillation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Mao, Yunyao Deng, Jiajun Zhou, Wengang Lu, Zhenbo Ouyang, Wanli Li, Houqiang CAS Key Laboratory of Technology in GIPAS Department of Electronic Engineeringand Information Science University of Science and Technology of China Hefei China The University of Adelaide Adelaide Australia Institute of Artificial Intelligence Hefei Comprehensive National Science Center Hefei China Shanghai Artificial Intelligent Laboratory Shanghai China

Recent progresses on self-supervised 3D human action representation learning are largely attributed to contrastive learning. However, in conventional contrastive frameworks, the rich complementarity between different skeleton modalities remains under-explored. Moreover, optimized with distinguishing self-augmented samples, models struggle with numerous similar positive instances in the case of limited action categories. In this work, we tackle the aforementioned problems by introducing a general Inter- and Intra-modal Mutual Distillation (I2MD) framework. In I2MD, we first re-formulate the cross-modal interaction as a Cross-modal Mutual Distillation (CMD) process. Different from existing distillation solutions that transfer the knowledge of a pre-trained and fixed teacher to the student, in CMD, the knowledge is continuously updated and bidirectionally distilled between modalities during pre-training. To alleviate the interference of similar samples and exploit their underlying contexts, we further design the Intra-modal Mutual Distillation (IMD) strategy, In IMD, the Dynamic Neighbors Aggregation (DNA) mechanism is first introduced, where an additional cluster-level discrimination branch is instantiated in each modality. It adaptively aggregates highly-correlated neighboring features, forming local cluster-level contrasting. Mutual distillation is then performed between the two branches for cross-level knowledge exchange. Extensive experiments on three datasets show that our approach sets a series of new records. Copyright © 2023, The Authors. All rights reserved.

关键词： Distillation

DIRE for Diffusion-Generated Image Detection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Zhendong Bao, Jianmin Zhou, Wengang Wang, Weilun Hu, Hezhen Chen, Hong Li, Houqiang CAS Key Laboratory of GIPAS EEIS Department University of Science and Technology of China China Microsoft Research Asia China Institute of Artificial Intelligence Hefei Comprehensive National Science Center China Merchants Union Consumer Finance Company

Diffusion models have shown remarkable success in visual synthesis, but have also raised concerns about potential abuse for malicious purposes. In this paper, we seek to build a detector for telling apart real images from diffusion-generated images. We find that existing detectors struggle to detect images generated by diffusion models, even if we include generated images from a specific diffusion model in their training data. To address this issue, we propose a novel image representation called DIffusion Reconstruction Error (DIRE), which measures the error between an input image and its reconstruction counterpart by a pre-trained diffusion model. We observe that diffusion-generated images can be approximately reconstructed by a diffusion model while real images cannot. It provides a hint that DIRE can serve as a bridge to distinguish generated and real images. DIRE provides an effective way to detect images generated by most diffusion models, and it is general for detecting generated images from unseen diffusion models and robust to various perturbations. Furthermore, we establish a comprehensive diffusion-generated benchmark including images generated by eight diffusion models to evaluate the performance of diffusion-generated image detectors. Extensive experiments on our collected benchmark demonstrate that DIRE exhibits superiority over previous generated-image detectors. The code and dataset are available at https://***/ZhendongWang6/DIRE. Copyright © 2023, The Authors. All rights reserved.

关键词： Diffusion

A Circuit Domain Generalization Framework for Efficient Logic Synthesis in Chip Design

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Zhihai Chen, Lei Wang, Jie Li, Xing Bai, Yinqi Li, Xijun Yuan, Mingxuan Hao, Jianye Zhang, Yongdong Wu, Feng CAS Key Laboratory of Technology in GIPAS University of Science and Technology of China Hefei230027 China Institute of Artificial Intelligence Hefei Comprehensive National Science Center Hefei230091 China Huawei Noah’s Ark Lab Canada College of Intelligence and Computing Tianjin University China

—Logic Synthesis (LS) plays a vital role in chip design—a cornerstone of the semiconductor industry. A key task in LS is to transform circuits—modeled by directed acyclic graphs (DAGs)—into simplified circuits with equivalent functionalities. To tackle this task, many LS operators apply transformations to subgraphs—rooted at each node on an input DAG—sequentially. However, we found that a large number of transformations are ineffective, which makes applying these operators highly time-consuming. In particular, we notice that the runtime of the Resub and Mfs2 operators often dominates the overall runtime of LS optimization processes. To address this challenge, we propose a novel data-driven LS operator paradigm, namely PruneX, to reduce ineffective transformations. The major challenge of developing PruneX is to learn models that well generalize to unseen circuits, i.e., the out-of-distribution (OOD) generalization problem. Thus, the major technical contribution of PruneX is the novel circuit domain generalization framework, which learns domain-invariant representations based on the transformation-invariant domain-knowledge. To the best of our knowledge, PruneX is the first approach to tackle the OOD problem in LS operators. We integrate PruneX with the aforementioned Resub and Mfs2 operators. Experiments demonstrate that PruneX significantly improves their efficiency while keeping comparable optimization performance on industrial and very large-scale circuits, achieving up to 3.1× faster runtime. Copyright © 2023, The Authors. All rights reserved.

关键词： Logic Synthesis

CLIP2GAN: Towards Bridging Text with the Latent Space of GANs

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Wang, Yixuan Zhou, Wengang Bao, Jianmin Wang, Weilun Li, Li Li, Houqiang CAS Key Laboratory of GIPAS EEIS Department University of Science and Technology of China China Microsoft Research Asia China Institute of Artificial Intelligence Hefei Comprehensive National Science Center China

In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods. Copyright © 2022, The Authors. All rights reserved.

关键词： Mapping

Rethinking Graph Convolutional Networks in Knowledge Graph Completion

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhang, Zhanqiu Wang, Jie Ye, Jieping Wu, Feng CAS Key Laboratory of Technology in GIPAS University of Science and Technology of China China Institute of Artificial Intelligence Hefei Comprehensive National Science Center University of Michigan United States KE Holdings Inc China

Graph convolutional networks (GCNs)-which are effective in modeling graph structures-have been increasingly popular in knowledge graph completion (KGC). GCN-based KGC models first use GCNs to generate expressive entity representations and then use knowledge graph embedding (KGE) models to capture the interactions among entities and relations. However, many GCN-based KGC models fail to outperform state-of-the-art KGE models though introducing additional computational complexity. This phenomenon motivates us to explore the real effect of GCNs in KGC. Therefore, in this paper, we build upon representative GCN-based KGC models and introduce variants to find which factor of GCNs is critical in KGC. Surprisingly, we observe from experiments that the graph structure modeling in GCNs does not have a significant impact on the performance of KGC models, which is in contrast to the common belief. Instead, the transformations for entity representations are responsible for the performance improvements. Based on the observation, we propose a simple yet effective framework named LTE-KGE, which equips existing KGE models with linearly transformed entity embeddings. Experiments demonstrate that LTE-KGE models lead to similar performance improvements with GCN-based KGC methods, while being more computationally efficient. These results suggest that existing GCNs are unnecessary for KGC, and novel GCN-based KGC models should count on more ablation studies to validate their effectiveness. The code of all the experiments is available on GitHub at https://***/MIRALab-USTC/GCN4KGC. © 2022, CC BY.

关键词： Knowledge graph

Generalization in Visual Reinforcement Learning with the Reward Sequence Distribution

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Jie Yang, Rui Geng, Zijie Shi, Zhihao Ye, Mingxuan Zhou, Qi Ji, Shuiwang Li, Bin Zhang, Yongdong Wu, Feng CAS Key Laboratory of Technology in GIPAS University of Science and Technology of China Hefei230027 China Institute of Artificial Intelligence Hefei Comprehensive National Science Center Hefei230091 China The Department of Computer Science and Engineering Texas A&M University College Station TX77843 United States

Generalization in partially observed markov decision processes (POMDPs) is critical for successful applications of visual reinforcement learning (VRL) in real scenarios. A widely used idea is to learn task-relevant representations that encode task-relevant information of common features in POMDPs, i.e., rewards and transition dynamics. As transition dynamics in the latent state space—which are task-relevant and invariant to visual distractions—are unknown to the agents, existing methods alternatively use transition dynamics in the observation space to extract task-relevant information in transition dynamics. However, such transition dynamics in the observation space involve task-irrelevant visual distractions, degrading the generalization performance of VRL methods. To tackle this problem, we propose the reward sequence distribution conditioned on the starting observation and the predefined subsequent action sequence (RSD-OA). The appealing features of RSD-OA include that: (1) RSD-OA is invariant to visual distractions, as it is conditioned on the predefined subsequent action sequence without task-irrelevant information from transition dynamics, and (2) the reward sequence captures long-term task-relevant information in both rewards and transition dynamics. Experiments demonstrate that our representation learning approach based on RSD-OA significantly improves the generalization performance on unseen environments, outperforming several state-of-the-arts on DeepMind Control tasks with visual distractions. Copyright © 2023, The Authors. All rights reserved.

关键词： Reinforcement learning

学校读者我要写书评

暂无评论

Proceedings of the 35th International Conference on Neural Information Processing Systems

作者： Jianbo Ouyang Hui Wu Min Wang Wengang Zhou Houqiang Li CAS Key Laboratory of Technology in GIPAS EEIS Department University of Science and Technology of China Institute of Artificial Intelligence Hefei Comprehensive National Science Center CAS Key Laboratory of Technology in GIPAS EEIS Department University of Science and Technology of China and Institute of Artificial Intelligence Hefei Comprehensive National Science Center

ISBN: (纸本)9781713845393

In content-based image retrieval, the first-round retrieval result by simple visual feature comparison may be unsatisfactory, which can be refined by visual re-ranking techniques. In image retrieval, it is observed that the contextual similarity among the top-ranked images is an important clue to distinguish the semantic relevance. Inspired by this observation, in this paper, we propose a visual re-ranking method by contextual similarity aggregation with self-attention. In our approach, for each image in the top-K ranking list, we represent it into an affinity feature vector by comparing it with a set of anchor images. Then, the affinity features of the top-K images are refined by aggregating the contextual information with a transformer encoder. Finally, the affinity features are used to recalculate the similarity scores between the query and the top-K images for re-ranking of the latter. To further improve the robustness of our re-ranking model and enhance the performance of our method, a new data augmentation scheme is designed. Since our re-ranking model is not directly involved with the visual feature used in the initial retrieval, it is ready to be applied to retrieval result lists obtained from various retrieval algorithms. We conduct comprehensive experiments on four benchmark datasets to demonstrate the generality and effectiveness of our proposed visual re-ranking method.

关键词：

Joint Inductive and Transductive Learning for Video Object Segmentation

学校读者我要写书评

暂无评论

Joint Inductive and Transductive Learning for Video Object S...

International Conference on Computer Vision (ICCV)

作者： Yunyao Mao Ning Wang Wengang Zhou Houqiang Li CAS Key Laboratory of Technology in GIPAS University of Science and Technology of China Institute of Artificial Intelligence Hefei Comprehensive National Science Center

ISBN: (纸本)9781665428132

Semi-supervised video object segmentation is a task of segmenting the target object in a video sequence given only a mask annotation in the first frame. The limited information available makes it an extremely challenging task. Most previous best-performing methods adopt matching-based transductive reasoning or online inductive learning. Nevertheless, they are either less discriminative for similar instances or insufficient in the utilization of spatio-temporal information. In this work, we propose to integrate transductive and inductive learning into a unified framework to exploit the complementarity between them for accurate and robust video object segmentation. The proposed approach consists of two functional branches. The transduction branch adopts a lightweight transformer architecture to aggregate rich spatio-temporal cues while the induction branch performs online inductive learning to obtain discriminative target information. To bridge these two diverse branches, a two-head label encoder is introduced to learn the suitable target prior for each of them. The generated mask encodings are further forced to be disentangled to better retain their complementarity. Extensive experiments on several prevalent benchmarks show that, without the need of synthetic training data, the proposed approach sets a series of new state-of-the-art records. Code is available at https://***/maoyunyao/JOINT.

关键词： Computer vision Video sequences Training data Object segmentation Computer architecture Benchmark testing Transformers

Learning Deep Local Features with Multiple Dynamic Attentions for Large-Scale Image Retrieval

学校读者我要写书评

暂无评论

Learning Deep Local Features with Multiple Dynamic Attention...

International Conference on Computer Vision (ICCV)

作者： Hui Wu Min Wang Wengang Zhou Houqiang Li CAS Key Laboratory of Technology in GIPAS University of Science and Technology of China Institute of Artificial Intelligence Hefei Comprehensive National Science Center

ISBN: (纸本)9781665428132

In image retrieval, learning local features with deep convolutional networks has been demonstrated effective to improve the performance. To discriminate deep local features, some research efforts turn to attention learning. However, existing attention-based methods only generate a single attention map for each image, which limits the exploration of diverse visual patterns. To this end, we propose a novel deep local feature learning architecture to simultaneously focus on multiple discriminative local patterns in an image. In our framework, we first adaptively reorganize the channels of activation maps for multiple heads. For each head, a new dynamic attention module is designed to learn the potential attentions. The whole architecture is trained as metric learning of weighted-sum-pooled global image features, with only image-level relevance label. After the architecture training, for each database image, we select local features based on their multi-head dynamic attentions, which are further indexed for efficient retrieval. Extensive experiments show the proposed method outperforms the state-of-the-art methods on the Revisited Oxford and Paris datasets. Besides, it typically achieves competitive results even using local features with lower dimensions. Code will be released at https://***/CHANWH/MDA.

关键词： Training Representation learning Measurement Visualization Head Quantization (signal) Databases