检索结果-内蒙古大学图书馆

Semi-Supervised Image Captioning Considering Wasserstein Graph Matching

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yang, Yang The Nanjing University of Science and Technology Nanjing210094 China PCA Lab Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education Jiangsu Key Lab of Image and Video Understanding for Social Security School of Computer Science and Engineering Nanjing University of Science and Technology China Ministry of Education State Key Lab. for Novel Software Technology Nanjing University China

Image captioning can automatically generate captions for the given images, and the key challenge is to learn a mapping function from visual features to natural language features. Existing approaches are mostly supervised ones, i.e., each image has a corresponding sentence in the training set. However, considering that describing images always requires a huge of manpower, we usually have limited amount of described images (i.e., image-text pairs) and a large number of undescribed images in real-world applications. Thereby, a dilemma is the "Semi-Supervised Image Captioning". To solve this problem, we propose a novel Semi-Supervised Image Captioning method considering Wasserstein Graph Matching (SSIC-WGM), which turns to adopt the raw image inputs to supervise the generated sentences. Different from traditional single modal semi-supervised methods, the difficulty of semi-supervised cross-modal learning lies in constructing intermediately comparable information among heterogeneous modalities. In this paper, SSIC-WGM adopts the successful scene graphs as intermediate information, and constrains the generated sentences from two aspects: 1) inter-modal consistency. SSIC-WGM constructs the scene graphs of the raw image and generated sentence respectively, then employs the wasserstein distance to better measure the similarity between region embeddings of different graphs. 2) intra-modal consistency. SSIC-WGM takes the data augmentation techniques for the raw images, then constrains the consistency among augmented images and generated sentences. Consequently, SSIC-WGM combines the cross-modal pseudo supervision and structure invariant measure for efficiently using the undescribed images, and learns more reasonable mapping function. Experiments show that our method can outperform state-of-the-art comparison methods on the MS-COCO "Karpathy" offline test split, under different complex semi-supervised scenarios. Copyright © 2024, The Authors. All rights reserved.

关键词： Visual languages

StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Suzhen Ma, Yifeng Ding, Yu Hu, Zhipeng Fan, Changjie Lv, Tangjie Deng, Zhidong Yu, Xin Fuxi AI Lab Netease Zhejiang Hangzhou China Department of Computer Science and Technology BNRist THUAI State Key Laboratory of Intelligent Technology and Systems Tsinghua University China The School of Computer Science The University of Queensland Brisbane Australia

Individuals have unique facial expression and head pose styles that reflect their personalized speaking styles. Existing one-shot talking head methods cannot capture such personalized characteristics and therefore fail to produce diverse speaking styles in the final videos. To address this challenge, we propose a one-shot style-controllable talking face generation method that can obtain speaking styles from reference speaking videos and drive the one-shot portrait to speak with the reference speaking styles and another piece of audio. Our method aims to synthesize the style-controllable coefficients of a 3D Morphable Model (3DMM), including facial expressions and head movements, in a unified framework. Specifically, the proposed framework first leverages a style encoder to extract the desired speaking styles from the reference videos and transform them into style codes. Then, the framework uses a style-aware decoder to synthesize the coefficients of 3DMM from the audio input and style codes. During decoding, our framework adopts a two-branch architecture, which generates the stylized facial expression coefficients and stylized head movement coefficients, respectively. After obtaining the coefficients of 3DMM, an image renderer renders the expression coefficients into a specific person's talking-head video. Extensive experiments demonstrate that our method generates visually authentic talking head videos with diverse speaking styles from only one portrait image and an audio clip. Copyright © 2024, The Authors. All rights reserved.

关键词： Deep neural networks

CLICK: Controllable Text Generation with Sequence Likelihood Contrastive Learning

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zheng, Chujie Ke, Pei Zhang, Zheng Huang, Minlie The CoAI Group DCST Institute for Artificial Intelligence State Key Lab of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology Tsinghua University Beijing100084 China

It has always been an important yet challenging problem to control language models to avoid generating texts with undesirable attributes, such as toxic language and unnatural repetition. We introduce CLICK for controllable text generation, which needs no modification to the model architecture and facilitates out-of-the-box use of trained models. It employs a contrastive loss on sequence likelihood, which fundamentally decreases the generation probability of negative samples (i.e., generations with undesirable attributes). It also adopts a novel likelihood ranking-based strategy to construct contrastive samples from model generations. On the tasks of language detoxification, sentiment steering, and repetition reduction, we show that CLICK outperforms strong baselines of controllable text generation and demonstrate the superiority of CLICK’s sample construction strategy.1 Copyright © 2023, The Authors. All rights reserved.

关键词： Detoxification

Safety Assessment of Chinese Large Language Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Sun, Hao Zhang, Zhexin Deng, Jiawen Cheng, Jiale Huang, Minlie The CoAI Group DCST Institute for Artificial Intelligence State Key Lab of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology Tsinghua University Beijing100084 China

With the rapid popularity of large language models such as ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023), a growing amount of attention is paid to their safety concerns. These models may generate insulting and discriminatory content, reflect incorrect social values, and may be used for malicious purposes such as fraud and dissemination of misleading information. Evaluating and enhancing their safety is particularly essential for the wide application of large language models (LLMs). To further promote the safe deployment of LLMs, we develop a Chinese LLM safety assessment benchmark. Our benchmark explores the comprehensive safety performance of LLMs from two perspectives: 8 kinds of typical safety scenarios and 6 types of more challenging instruction attacks. Our benchmark is based on a straightforward process in which it provides the test prompts and evaluates the safety of the generated responses from the evaluated model. In evaluation, we utilize the LLM’s strong evaluation ability and develop it as a safety evaluator by prompting. On top of this benchmark, we conduct safety assessments and analyze 15 LLMs including the OpenAI GPT series and other well-known Chinese LLMs, where we observe some interesting findings. For example, we find that instruction attacks are more likely to expose safety issues of all LLMs. Moreover, to promote the development and deployment of safe, responsible, and ethical AI, we publicly release SAFETYPROMPTS including 100k augmented prompts and responses by LLMs. 1 2 Copyright © 2023, The Authors. All rights reserved.

关键词： Benchmarking

Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Guan, Jian Huang, Minlie The CoAI Group DCST Institute for Artificial Intelligence State Key Lab of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology Tsinghua University Beijing100084 China

Despite the huge progress in a myriad of generation tasks, pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts with maximization-based decoding algorithms for open-ended generation. We attribute their overestimation of token-level repetition probabilities to the learning bias: LMs capture simple repetitive patterns faster with the MLE loss. We propose self-contrastive training to penalize the output of a premature checkpoint of the same model when it incorrectly predicts repetition, which is shown to mitigate repetition effectively while maintaining fluency on two datasets. Furthermore, we find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops. Copyright © 2023, The Authors. All rights reserved.

关键词：

ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Zhexin Wen, Jiaxin Huang, Minlie The CoAI group DCST Institute for Artificial Intelligence State Key Lab of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology Tsinghua University Beijing100084 China

Large pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named ETHICIST for targeted training data Extraction THrough loss smoothed soft prompting and calIbrated ConfIdence eSTimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that ETHICIST significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is available at https://***/thu-coai/Targeted-Data-Extraction. Copyright © 2023, The Authors. All rights reserved.

关键词： Extraction

DA-Ada: learning domain-aware adapter for domain adaptive object detection 24

学校读者我要写书评

暂无评论

DA-Ada: learning domain-aware adapter for domain adaptive ob...

Proceedings of the 38th International Conference on Neural Information Processing systems

作者： Haochen Li Rui Zhang Hantao Yao Xin Zhang Yifan Hao Xinkai Song Xiaqing Li Yongwei Zhao Ling Li Yunji Chen Intelligent Software Research Center Institute of Software CAS Beijing China and University of Chinese Academy of Sciences Beijing China State Key Lab of Processors Institute of Computing Technology CAS Beijing China State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation CAS Beijing China State Key Lab of Processors Institute of Computing Technology CAS Beijing China and University of Chinese Academy of Sciences Beijing China

ISBN: (纸本)9798331314385

Domain adaptive object detection (DAOD) aims to generalize detectors trained on an annotated source domain to an unlabelled target domain. As the visual-language models (VLMs) can provide essential general knowledge on unseen images, freezing the visual encoder and inserting a domain-agnostic adapter can learn domaininvariant knowledge for DAOD. However, the domain-agnostic adapter is inevitably biased to the source domain. It discards some beneficial knowledge discriminative on the unlabelled domain, i.e. domain-specific knowledge of the target domain. To solve the issue, we propose a novel Domain-Aware Adapter (DA-Ada) tailored for the DAOD task. The key point is exploiting domain-specific knowledge between the essential general knowledge and domain-invariant knowledge. DA-Ada consists of the Domain-Invariant Adapter (DIA) for learning domain-invariant knowledge and the Domain-Specific Adapter (DSA) for injecting the domain-specific knowledge from the information discarded by the visual encoder. Comprehensive experiments over multiple DAOD tasks show that DA-Ada can efficiently infer a domain-aware visual encoder for boosting domain adaptive object detection. Our code is available at https://***/Therock90421/DA-Ada.

关键词：

Prototype-based classifier learning for long-tailed visual recognition

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2022年第6期65卷 62-76页

作者： Xiu-Shen WEI Shu-Lin XU Hao CHEN Liang XIAO Yuxin PENG School of Computer Science and Engineering Nanjing University of Science and Technology State Key Laboratory of Integrated Services Networks Xidian University Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education and Jiangsu Key Lab of Image and Video Understanding for Social Security Wangxuan Institute of Computer Technology Peking University

In this paper, we tackle the long-tailed visual recognition problem from the categorical prototype perspective by proposing a prototype-based classifier learning(PCL) method. Specifically, thanks to the generalization ability and robustness, categorical prototypes reveal their advantages of representing the category semantics. Coupled with their class-balance characteristic, categorical prototypes also show potential for handling data imbalance. In our PCL, we propose to generate the categorical classifiers based on the prototypes by performing a learnable mapping function. To further alleviate the impact of imbalance on classifier generation, two kinds of classifier calibration approaches are designed from both prototype-level and example-level aspects. Extensive experiments on five benchmark datasets, including the large-scale iNaturalist, Places-LT,and ImageNet-LT, justify that the proposed PCL can outperform state-of-the-arts. Furthermore, validation experiments can demonstrate the effectiveness of tailored designs in PCL for long-tailed problems.

关键词： long-tailed distribution categorical prototype classifier generation classifier calibration class imbalance

Coordinated Navigation Control of Cross-Domain Unmanned systems via Guiding Vector Fields

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Hu, Bin-Bin Zhang, Hai-Tao Liu, Bin Ding, Jianing Xu, Yifan Luo, Chuanshang Cao, Haosen The School of Artificial Intelligence and Automation The Engineering Research Center of Autonomous Intelligent Unmanned Systems The Key Laboratory of Image Processing and Intelligent Control The State Key Lab of Digital Manufacturing Equipment and Technology Huazhong University of Science and Technology Wuhan430074 China Dongguan University of Technology Dongguan523106 China

This paper proposes a distributed guiding-vector-field (DGVF) controller for cross-domain unmanned systems (CDUSs) consisting of heterogeneous unmanned aerial vehicles (UAVs) and unmanned surface vehicles (USVs), to achieve coordinated navigation whereas maneuvering along their prescribed paths. In particular, the DGVF controller provides a hierarchical architecture of an upper-level heterogeneous guidance velocity controller and a lower-level signal tracking regulator. Therein, the upper-level controller is to govern multiple heterogeneous USVs and UAVs to approach and maneuver along the prescribed paths and coordinate the formation simultaneously, whereas the low-level regulator is to track the corresponding desired guidance signals provided by the upper-level module. Significantly, the heterogeneous coordination among neighboring UAVs and USVs is achieved merely by the lightweight communication of a scalar (i.e., the additional virtual coordinate), which substantially decreases the communication and computational costs. Sufficient conditions assuring asymptotical convergence of the closed-loop system are derived in presence of the exponentially vanishing tracking errors. Finally, real-lake experiments are conducted on a self-established cross-domain heterogeneous platform consisting of three M-100 UAVs, two HUSTER-16 USVs, a HUSTER-12C USV, and a WiFi 5G wireless communication station to verify the effectiveness of the present DGVF controller. Copyright © 2023, The Authors. All rights reserved.

关键词： Air navigation