检索结果-内蒙古大学图书馆

Multimodal transformer with multi-view visual representation for image captioning

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Yu, Jun Li, Jing Yu, Zhou Huang, Qingming Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University Hangzhou310018 China School of Computer and Control Engineering University of Chinese Academy of Sciences Beijing101408 China

Image captioning aims to automatically generate a natural language description of a given image, and most state-ofthe-art models have adopted an encoder-decoder framework. The framework consists of a convolution neural network (CNN)-based image encoder that extracts region-based visual features from the input image, and an recurrent neural network (RNN)-based caption decoder that generates the output caption words based on the visual features with the attention mechanism. Despite the success of existing studies, current methods only model the co-attention that characterizes the inter-modal interactions while neglecting the self-attention that characterizes the intramodal interactions. Inspired by the success of the Transformer model in machine translation, here we extend it to a Multimodal Transformer (MT) model for image captioning. Compared to existing image captioning approaches, the MT model simultaneously captures intra- and inter-modal interactions in a unified attention block. Due to the in-depth modular composition of such attention blocks, the MT model can perform complex multimodal reasoning and output accurate captions. Moreover, to further improve the image captioning performance, multi-view visual features are seamlessly introduced into the MT model. We quantitatively and qualitatively evaluate our approach using the benchmark MSCOCO image captioning dataset and conduct extensive ablation studies to investigate the reasons behind its effectiveness. The experimental results show that our method significantly outperforms the previous state-of-the-art methods. With an ensemble of seven models, our solution ranks the 1st place on the real-time leaderboard of the MSCOCO image captioning challenge at the time of the writing of this paper. Copyright © 2019, The Authors. All rights reserved.

关键词： Recurrent neural networks

Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yu, Zhou Ouyang, Xuecheng Shao, Zhenwei Wang, Meng Yu, Jun The Key Laboratory of Complex Systems Modeling and Simulation the School of Computer Science Hangzhou Dianzi University China The HDU-ITMO Joint Institute Hangzhou Dianzi University China The School of Computer Science and Information Engineering Hefei University of Technology China

Knowledge-based visual question answering (VQA) requires external knowledge beyond the image to answer the question. Early studies retrieve required knowledge from explicit knowledge bases (KBs), which often introduces irrelevant information to the question, hence restricting the performance of their models. Recent works have resorted to using a powerful large language model (LLM) as an implicit knowledge engine to acquire the necessary knowledge for answering. Despite the encouraging results achieved by these methods, we argue that they have not fully activated the capacity of the blind LLM as the provided textual input is insufficient to depict the required visual information to answer the question. In this paper, we present Prophet—a conceptually simple, flexible, and general framework designed to prompt LLM with answer heuristics for knowledge-based VQA. Specifically, we first train a vanilla VQA model on a specific knowledge-based VQA dataset without external knowledge. After that, we extract two types of complementary answer heuristics from the VQA model: answer candidates and answer-aware examples. Finally, the two types of answer heuristics are jointly encoded into a formatted prompt to facilitate the LLM’s understanding of both the image and question, thus generating a more accurate answer. By incorporating the state-of-the-art LLM GPT-3 [1], Prophet significantly outperforms existing state-of-the-art methods on four challenging knowledge-based VQA datasets. To demonstrate the generality of our approach, we instantiate Prophet with the combinations of different VQA models (i.e., both discriminative and generative ones) and different LLMs (i.e., both commercial and open-source ones). © 2023, CC BY.

关键词： Knowledge based systems

Uniform convergence of multigrid finite element method for time-dependent riesz tempered fractional problem

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Chen, Minghua Bu, Weiping Qi, Wenya Wang, Yantao School of Mathematics and Statistics Gansu Key Laboratory of Applied Mathematics and Complex Systems Lanzhou University Lanzhou730000 China School of Mathematics and Computational Science Hunan Key Laboratory for Computation and Simulation in Science and Engineering Xiangtan University Xiangtan411105 China South University of Science and Technology of China Shenzhen518055 China

In this article a theoretical framework for the Galerkin finite element approximation to the time-dependent Riesz tempered fractional problem is provided without the fractional regularity assumption. Because the time-dependent problems should become easier to solve as the time step τ → 0, which correspond to the mass matrix dominant [R. E. Bank and T. Dupont, Math. Comp., 153 (1981), pp. 35-51]. Based on the introduced and analysis of the fractional τ-norm, the uniform convergence estimates of the V-cycle multigrid method with the time-dependent fractional problem is strictly proved, which means that the convergence rate of the V-cycle MGM is independent of the mesh size h and the time step τ. The numerical experiments are performed to verify the convergence with only O(NlogN) complexity by the fast Fourier transform method, where N is the number of the grid points. To the best of our knowledge, this is the first proof for the convergence rate of the V-cycle multigrid finite element method with τ → 0. Copyright © 2017, The Authors. All rights reserved.

关键词： Fast Fourier transforms

An Auto-Parallel Method for Deep Learning Models Based on Genetic Algorithm 29

学校读者我要写书评

暂无评论

An Auto-Parallel Method for Deep Learning Models Based on Ge...

29th IEEE International Conference on Parallel and Distributed systems, ICPADS 2023

作者： Zeng, Yan Huang, Chengchuang Ni, Yijie Zhou, Chunbao Zhang, Jilin Wang, Jue Zhou, Mingyao Xue, Meiting Zhang, Yunquan Hangzhou Dianzi University School of Computer Science and Technology Hangzhou310018 China Ministry of Education Key Laboratory for Modeling and Simulation of Complex Systems Hangzhou310018 China Data Security Governance Zhejiang Engineering Research Center Hangzhou310018 China Hangzhou Dianzi University School of ITMO Joint Institute Hangzhou310018 China Institute of Computer Network Information Center of the Chinese Academy of Sciences Beijing100086 China HuaWei China Institute of Computing Technology of the Chinese Academy of Sciences State Key Laboratory of Computer Architecture Beijing100086 China

ISBN: (纸本)9798350330717

As the size of datasets and neural network models increases, automatic parallelization methods for models have become a research hotspot in recent years. The existing auto-parallel methods based on machine learning or graph algorithms still have issues with search efficiency and applicability. This paper proposes an automatic parallel method based on a dual-population genetic algorithm, TGA, which transforms model partitioning and placement into an integer linear programming problem and constructs a cost model to evaluate the solution. The solution space is built using the neural network's dataflow graph and device cluster's topology, and the dual-population genetic algorithm is used to search for the optimal model parallel strategy. Experiments with various models show that the proposed method can improve single-step execution time by up to 42% compared to the Baechi method and up to 37.7% compared to the Hierarchical method. © 2023 IEEE.

关键词： Auto-Parallel Distributed machine learning Genetic algorithm Integer linear programming Model parallelism

Roto-Translation Invariant Formation of Fixed-Wing UAVs in 3D: Feasibility and Control

学校读者我要写书评

暂无评论

arXiv 2023年

作者： He, Xiaodong Li, Zhongkui Wang, Xiangke Geng, Zhiyong The State Key Laboratory for Turbulence and Complex Systems Department of Mechanics and Engineering Science College of Engineering Peking University Beijing100871 China The College of Intelligence Science and Technology National University of Defense Technology Changsha410073 China

This paper investigates the formation of fixed-wing UAVs in 3D, which communicate via a directed acyclic graph. Different from common formation problems, we consider the roto-translation invariant (RTI) formation, where the "roto-translation" refers to a rigid-body motion obtained by composing rotation and translation. Besides, the fixed-wing UAV is modelled by a 3-D nonholonomic constrained rigid body instead of a particle agent. The main results of this paper include proposing the formation feasibility conditions and designing the formation controller. Firstly, we define the RTI formation and propose the conditions to guarantee that the formations are feasible for the fixed-wing UAVs under the nonholonomic and input saturation constraints. Secondly, given feasible formations, we design a formation controller by introducing a virtual leader and employing the compensation of rotation, followed by proving the stability of the closed-loop system. Finally, simulation examples are presented to verify the theoretical *** with the motion coupling among multiple robots. Finally, numerical simulations are conducted to verify the theoretical results. Copyright © 2023, The Authors. All rights reserved.

关键词： Unmanned aerial vehicles (UAV)

ActivityNet-QA: A dataset for understanding complex web videos via question answering

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Yu, Zhou Xu, Dejing Yu, Jun Yu, Ting Zhao, Zhou Zhuang, Yueting Tao, Dacheng Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University Hangzhou China College of Computer Science Zhejiang University Hangzhou China UBTECH Sydney AI Centre SIT FEIT University of Sydney Australia

Recent developments in modeling language and vision have been successfully applied to image question answering. It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA). Compared to the image domain where large scale and fully annotated benchmark datasets exists, VideoQA datasets are limited to small scale and are automatically generated, etc. These limitations restrict their applicability in practice. Here we introduce ActivityNet-QA, a fully annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex web videos derived from the popular ActivityNet dataset. We present a statistical analysis of our ActivityNet-QA dataset and conduct extensive experiments on it by comparing existing VideoQA baselines. Moreover, we explore various video representation strategies to improve VideoQA performance, especially for long videos. The dataset is available at https://***/MILVLG/ activitynet-qa. Copyright © 2019, The Authors. All rights reserved.

关键词： Modeling languages

Multi-modal factorized bilinear pooling with co-attention learning for visual question answering

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Yu, Zhou Yu, Jun Fan, Jianping Tao, Dacheng Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University China Department of Computer Science University of North Carolina Charlotte United States UBTECH Sydney AI Centre School of IT FEIT University of Sydney Australia

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multimodal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multimodal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a 'co-attention' mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-theart performance on the real-world VQA dataset. Code available at https://***/yuzcccc/mfb. Copyright © 2017, The Authors. All rights reserved.

关键词： Network architecture

Metastable States and Wetting Transition of Submerged Superhydrophobic Structures

学校读者我要写书评

暂无评论

Physical Review Letters 2014年第19期112卷 196101-196101页

作者： Pengyu Lv Yahui Xue Yipeng Shi Hao Lin Huiling Duan State Key Laboratory for Turbulence and Complex Systems Department of Mechanics and Engineering Science College of Engineering Peking University Beijing 100871 China Key Laboratory of High Energy Density Physics Simulation Center for Applied Physics and Technology Peking University Beijing 100871 China Mechanical and Aerospace Engineering Rutgers The State University of New Jersey 98 Brett Road Piscataway New Jersey 08854 USA

Superhydrophobicity on structured surfaces is frequently achieved via the maintenance of liquid-air interfaces adjacent to the trapped air pockets. These interfaces, however, are subject to instabilities due to the Cassie-Baxter-to-Wenzel transition and total wetting. The current work examines in situ liquid-air interfaces on a submerged surface patterned with cylindrical micropores using confocal microscopy. Both the pinned Cassie-Baxter and depinned metastable states are directly observed and measured. The metastable state dynamically evolves, leading to a transition to the Wenzel state. This process is extensively quantified under different ambient pressure conditions, and the data are in good agreement with a diffusion-based model prediction. A similarity law along with a characteristic time scale is derived which governs the lifetime of the air pockets and which can be used to predict the longevity of underwater superhydrophobicity.

关键词： HYDROPHOBIC surfaces RESEARCH METASTABLE states WETTING LIQUID air PHYSICS research

Multimodal unified attention networks for vision-and-language interactions

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Yu, Zhou Cui, Yuhao Yu, Jun Tao, Dacheng Tian, Qi Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University China UBTECH Sydney AI Centre School of Computer Science Faculty of Engineering University of Sydney Australia Noah's Ark Lab Huawei China

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general 'unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top level performance on both tasks without bells and whistles. Copyright © 2019, The Authors. All rights reserved.

关键词： Deep learning