检索结果-内蒙古大学图书馆

Deep Modular Co-Attention Networks for Visual Question Answering

学校读者我要写书评

暂无评论

Deep Modular Co-Attention Networks for Visual Question Answe...

IEEE/CVF Conference on computer Vision and Pattern Recognition

作者： Zhou Yu Jun Yu Yuhao Cui Dacheng Tao Qi Tian Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University UBTECH Sydney AI Centre School of Computer Science FEIT University of Sydney Noah's Ark Lab

ISBN: (纸本)9781728132945

Visual Question Answering (VQA) requires a finegrained and simultaneous understanding of both the visual content of images and the textual content of questions. Therefore, designing an effective 'co-attention' model to associate key words in questions with key objects in images is central to VQA performance. So far, most successful attempts at co-attention learning have been achieved by using shallow models, and deep co-attention models show little improvement over their shallow counterparts. In this paper, we propose a deep Modular Co-Attention Network (MCAN) that consists of Modular Co-Attention (MCA) layers cascaded in depth. Each MCA layer models the self-attention of questions and images, as well as the question-guided-attention of images jointly using a modular composition of two basic attention units. We quantitatively and qualitatively evaluate MCAN on the benchmark VQA-v2 dataset and conduct extensive ablation studies to explore the reasons behind MCAN's effectiveness. Experimental results demonstrate that MCAN significantly outperforms the previous state-of-the-art. Our best single model delivers 70.63% overall accuracy on the test-dev set.

关键词： Modularity images Visualization system buses compensatory amounts Maximum credible accidents Layer models

On exploring undetermined relationships for visual relationship detection

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Zhan, Yibing Yu, Jun Yu, Ting Tao, Dacheng Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University China UBTECH Sydney AI Centre School of Computer Science FEIT University of Sydney DarlingtonNSW2008 Australia

In visual relationship detection, human-notated relationships can be regarded as determinate relationships. However, there are still large amount of unlabeled data, such as object pairs with less significant relationships or even with no relationships. We refer to these unlabeled but potentially useful data as undetermined relationships. Although a vast body of literature exists, few methods exploit these undetermined relationships for visual relationship detection. In this paper, we explore the beneficial effect of undetermined relationships on visual relationship detection. We propose a novel multi-modal feature based undetermined relationship learning network (MF-URLN) and achieve great improvements in relationship detection. In detail, our MF-URLN automatically generates undetermined relationships by comparing object pairs with human-notated data according to a designed criterion. Then, the MF-URLN extracts and fuses features of object pairs from three complementary modals: Visual, spatial, and linguistic modals. Further, the MF-URLN proposes two correlated subnetworks: One subnetwork decides the determinate confidence, and the other predicts the relationships. We evaluate the MF-URLN on two datasets: The Visual Relationship Detection (VRD) and the Visual Genome (VG) datasets. The experimental results compared with state-of-the-art methods verify the significant improvements made by the undetermined relationships, e.g., the top-50 relation detection recall improves from 19.5% to 23.9% on the VRD dataset. Copyright © 2019, The Authors. All rights reserved.

关键词： Machine learning

Deep Modular Co-Attention networks for visual question answering

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Yu, Zhou Yu, Jun Cui, Yuhao Tao, Dacheng Tian, Qi Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University UBTECH Sydney AI Centre School of Computer Science FEIT University of Sydney Noah's Ark Lab Huawei

Visual Question Answering (VQA) requires a finegrained and simultaneous understanding of both the visual content of images and the textual content of questions. Therefore, designing an effective 'co-attention' model to associate key words in questions with key objects in images is central to VQA performance. So far, most successful attempts at co-attention learning have been achieved by using shallow models, and deep co-attention models show little improvement over their shallow counterparts. In this paper, we propose a deep Modular Co-Attention Network (MCAN) that consists of Modular Co-Attention (MCA) layers cascaded in depth. Each MCA layer models the self-attention of questions and images, as well as the guided-attention of images jointly using a modular composition of two basic attention units. We quantitatively and qualitatively evaluate MCAN on the benchmark VQA-v2 dataset and conduct extensive ablation studies to explore the reasons behind MCAN's effectiveness. Experimental results demonstrate that MCAN significantly outperforms the previous state-ofthe- A rt. Our best single model delivers 70.63% overall accuracy on the test-dev set. Code is available at http://***/MILVLG/mcan-vqa. Copyright © 2019, The Authors. All rights reserved.

关键词： Machine learning

ActivityNet-QA: A dataset for understanding complex web videos via question answering

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Yu, Zhou Xu, Dejing Yu, Jun Yu, Ting Zhao, Zhou Zhuang, Yueting Tao, Dacheng Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University Hangzhou China College of Computer Science Zhejiang University Hangzhou China UBTECH Sydney AI Centre SIT FEIT University of Sydney Australia

Recent developments in modeling language and vision have been successfully applied to image question answering. It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA). Compared to the image domain where large scale and fully annotated benchmark datasets exists, VideoQA datasets are limited to small scale and are automatically generated, etc. These limitations restrict their applicability in practice. Here we introduce ActivityNet-QA, a fully annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex web videos derived from the popular ActivityNet dataset. We present a statistical analysis of our ActivityNet-QA dataset and conduct extensive experiments on it by comparing existing VideoQA baselines. Moreover, we explore various video representation strategies to improve VideoQA performance, especially for long videos. The dataset is available at https://***/MILVLG/ activitynet-qa. Copyright © 2019, The Authors. All rights reserved.

关键词： modeling languages

Multimodal unified attention networks for vision-and-language interactions

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Yu, Zhou Cui, Yuhao Yu, Jun Tao, Dacheng Tian, Qi Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University China UBTECH Sydney AI Centre School of Computer Science Faculty of Engineering University of Sydney Australia Noah's Ark Lab Huawei China

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general 'unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top level performance on both tasks without bells and whistles. Copyright © 2019, The Authors. All rights reserved.

关键词： Deep learning

Multi-modal factorized bilinear pooling with co-attention learning for visual question answering

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Yu, Zhou Yu, Jun Fan, Jianping Tao, Dacheng Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University China Department of Computer Science University of North Carolina Charlotte United States UBTECH Sydney AI Centre School of IT FEIT University of Sydney Australia

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multimodal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multimodal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a 'co-attention' mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-theart performance on the real-world VQA dataset. Code available at https://***/yuzcccc/mfb. Copyright © 2017, The Authors. All rights reserved.

关键词： Network architecture

Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Yu, Zhou Yu, Jun Xiang, Chenchao Fan, Jianping Tao, Dacheng Key Laboratory of Complex Systems Modeling and Simulation School of Computer Science and Technology Hangzhou Dianzi University China Department of Computer Science University of North Carolina Charlotte United States UBTECH Sydney Artificial Intelligence Centre School of Information Technologies Faculty of Engineering and Information Technologies University of Sydney DarlingtonNSW2008 Australia

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question;2) multi-modal feature fusion that is able to capture the complex interactions between multi-modal features;3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a 'co-attention' mechanism is developed by using a deep neural network architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multi-modal feature fusion, a generalized Multi-modal Factorized High-order pooling approach (MFH) is developed to achieve more effective fusion of multi-modal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the KL (Kullback-Leibler) divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A deep neural network architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA datasets and win the runner-up in VQA Challenge 2017. Copyright © 2017, The Authors. All rights reserved.

关键词： Deep neural networks

26th Annual Computational Neuroscience Meeting (CNS*2017) of the Organization for Computational Neuroscience Antwerp, Belgium, July 15-20, 2017

学校读者我要写书评

暂无评论

BMC NEUROSCIENCE 2017年第SUPPL 1期18卷 59-59页

作者： [Anonymous] Indiana University Purdue University Indianapolis Indianapolis IN 46032 USA Stark Neurosciences Research Institute Indiana University School of Medicine Indianapolis IN 46032 USA Department of Mathematics East Carolina University Greenville NC 27858 USA Jülich Supercomputing Centre Forschungszentrum Jülich 52425 Jülich Germany Future Systems Swiss National Supercomputing Centre 8092 Zurich Switzerland User Engagement and Support Swiss National Supercomputing Centre 6900 Lugano Switzerland Institut de Neurosciences des Systèmes Aix Marseille Univ 13005 Marseille France Simulation Lab Neuroscience Forschungszentrum Jülich Jülich Germany Department of Experimental Psychology Ghent University 9000 Ghent Belgium Donders Center for Cognitive Neuroimaging Radboud University 6525HR Nijmegen The Netherlands Department of Electrical Computer and Energy Engineering University of Colorado Boulder CO 80309 USA Department of Neurosurgery Johns Hopkins School of Medicine Baltimore MD 21287 USA Department of Neurology Johns Hopkins School of Medicine Baltimore MD 21287 USA Department of Otolaryngology Johns Hopkins School of Medicine Baltimore MD 21287 USA INSERM U968 Paris France Sorbonne Universités UPMC University Paris 06 UMR_S 968 Institut de la Vision Paris France CNRS UMR_7210 Paris France Department of Computer Architecture and Technology University of Granada (CITIC) Granada Spain Sorbonne Universités UPMC Univ Paris 06 INSERM CNRS Institut de la Vision Paris France Department of Adaptive Machine Systems Osaka University Osaka Japan Department of Computer Science University of Cergy-Pontoise Cergy-Pontoise France Department of Physics and Astronomy College of Charleston Charleston SC 29424 USA School of Physics Faculty of Science University of Sydney Sydney NSW 2006 Australia Center of Excellence for Integrative Brain Function Australian Research Council Sydney Australia Max Planck Institute for Human Cognitive and Brain Sciences Saxony Lei

26th Annual Computational Neuroscience Meeting (CNS*2017): Part 3 Antwerp, Belgium. 15-20 July 2017 Abstracts

学校读者我要写书评

暂无评论

BMC NEUROSCIENCE 2017年第SUPPL 1期18卷 95-176页

作者： [Anonymous] Department of Neuroscience Yale University New Haven CT 06520 USA Department Physiology & Pharmacology SUNY Downstate Brooklyn NY 11203 USA NYU School of Engineering 6 MetroTech Center Brooklyn NY 11201 USA Departament de Matemàtica Aplicada Universitat Politècnica de Catalunya Barcelona 08028 Spain Institut de Neurobiologie de la Méditerrannée (INMED) INSERM UMR901 Aix-Marseille Univ Marseille France Center of Neural Science New York University New York NY USA Aix-Marseille Univ INSERM INS Inst Neurosci Syst Marseille France Laboratoire de Physique Théorique et Modélisation CNRS UMR 8089 Université de Cergy-Pontoise 95300 Cergy-Pontoise Cedex France Department of Mathematics and Computer Science ENSAT Abdelmalek Essaadi’s University Tangier Morocco Laboratory of Natural Computation Department of Information and Electrical Engineering and Applied Mathematics University of Salerno 84084 Fisciano SA Italy Department of Medicine University of Salerno 84083 Lancusi SA Italy Dipartimento di Fisica Università degli Studi Aldo Moro Bari and INFN Sezione Di Bari Italy Data Analysis Department Ghent University Ghent Belgium Coma Science Group University of Liège Liège Belgium Cruces Hospital and Ikerbasque Research Center Bilbao Spain BIOtech Department of Industrial Engineering University of Trento and IRCS-PAT FBK 38010 Trento Italy Department of Data Analysis Ghent University Ghent 9000 Belgium The Wellcome Trust Centre for Neuroimaging University College London London WC1N 3BG UK Department of Electronic Engineering NED University of Engineering and Technology Karachi Pakistan Blue Brain Project École Polytechnique Fédérale de Lausanne Lausanne Switzerland Departement of Mathematics Swansea University Swansea Wales UK Laboratory for Topology and Neuroscience at the Brain Mind Institute École polytechnique fédérale de Lausanne Lausanne Switzerland Institute of Mathematics University of Aberdeen Aberdeen Scotland UK Department of Integrativ