检索结果-内蒙古大学图书馆

M2ConceptBase: A Fine-Grained Aligned Concept-Centric Multimodal Knowledge Base

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zha, Zhiwei Wang, Jiaan Li, Zhixu Zhu, Xiangru Song, Wei Xiao, Yanghua Shanghai Key Laboratory of Data Science School of Computer Science Fudan University Shanghai China School of Computer Science and Technology Soochow University Suzhou China Research Center for Intelligent Robotics Zhejiang Lab Hangzhou China

Multimodal knowledge bases (MMKBs) provide cross-modal aligned knowledge crucial for multimodal tasks. However, the images in existing MMKBs are generally collected for entities in encyclopedia knowledge graphs. Therefore, detailed groundings of visual semantics with linguistic concepts are lacking, which are essential for the visual concept cognition ability of multimodal models. Addressing this gap, we introduce M2ConceptBase, the first concept-centric MMKB. M2ConceptBase models concepts as nodes with associated images and detailed textual descriptions. We propose a context-aware multimodal symbol grounding approach to align concept-image and concept-description pairs using context information from image-text datasets. Comprising 951K images and 152K concepts, M2ConceptBase links each concept to an average of 6.27 images and a single description, ensuring comprehensive visual and textual semantics. Human studies confirm more than 95% alignment accuracy, underscoring its quality. Additionally, our experiments demonstrate that M2ConceptBase significantly enhances VQA model performance on the OK-VQA task. M2ConceptBase also substantially improves the fine-grained concept understanding capabilities of multimodal large language models through retrieval augmentation in two concept-related tasks, highlighting its value.1 Copyright © 2023, The Authors. All rights reserved.

关键词： Semantics

An Out-of-Distribution Attack Resistance Approach to Emotion Categorization

学校读者我要写书评

暂无评论

IEEE Transactions on Artificial Intelligence

IEEE Transactions on Artificial Intelligence 2021年第6期2卷 564-573页

作者： Shehu, Harisu Abdullahi Browne, Will N. Eisenbarth, Hedwig The School of Engineering and Computer Science Victoria University of Wellington Wellington6012 New Zealand The School of Electrical Engineering and Robotics Queensland University of Technology Brisbane4000 Australia The School of Psychology Victoria University of Wellington Wellington6012 New Zealand

Deep neural networks are a powerful model for feature extraction. They produce features that enable state-of-the-art performance on many tasks, including emotion categorization. However, their homogeneous representation of knowledge has made them prone to attacks, i.e., small modification in train or test data to mislead the models. Emotion categorization can usually be performed to be either in-distribution (train and test with the same dataset) or out-of-distribution (train on one or more dataset(s) and test on a different dataset). Our already developed landmark-based technique, which is robust for in-distribution improvement against attacks in emotion categorization, could translate to out-of-distribution classification problems. This is important as different databases might have different variations such as in color or level of expressiveness of emotion. We compared the landmark-based method with four state-of-the-art deep models (EfficientNetB0, InceptionV3, ResNet50, and VGG19), as well as emotion categorization tools (i.e., Python Facial Expression Analysis Toolbox and the Microsoft Azure face application programming interface) by performing a cross-database experiment across six commonly used databases, i.e., extended Cohn-Kanade, Japanese female facial expression, Karolinska directed emotional faces, National Institute of Mental Health Child Emotional Faces Picture Set, real-world affective faces, and psychological image collection at Stirling databases. The landmark-based method has achieved a significantly higher accuracy, achieving an average of 47.44% compared with most of the deep networks ( 97% accuracy. Impact Statement-Recognising emotions from people's faces has real-world applications for computer-based perception as it is often vital for interpersonal communication. Emotion recognition tasks nowadays are addressed using deep learning models that model colour distribution so classify images rather than emotion. This homogeneous knowledge represe

关键词： Emotion Recognition

Long-Term Invariant Local Features via Implicit Cross-Domain Correspondences

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Pataki, Zador Altillawi, Mohammad Kanakis, Menelaos Pautrat, Rémi Shen, Fengyi Liu, Ziyuan Van Gool, Luc Pollefeys, Marc The Computer Vision and Geometry Lab Department of Computer Science ETH Zurich Switzerland The Computer Vision Center CVC-Barcelona The Intelligent Robotics Cloud Technology lab of Huawei-Munich Germany The Computer Vision Lab Department Electrical Engineering ETH Zurich Switzerland The Intelligent Robotics Cloud Technology lab of Huawei-Munich Germany The Intelligent Robotics Cloud Technology lab of Huawei-Munich Germany The Center for Processing Speech and Images KU Leuven The Computer Vision Lab ETH Zurich Switzerland

Modern learning-based visual feature extraction networks perform well in intra-domain localization, however, their performance significantly declines when image pairs are captured across long-term visual domain variations, such as different seasonal and daytime variations. In this paper, our first contribution is a benchmark to investigate the performance impact of long-term variations on visual localization. We conduct a thorough analysis of the performance of current state-of-the-art feature extraction networks under various domain changes and find a significant performance gap between intra- and cross-domain localization. We investigate different methods to close this gap by improving the supervision of modern feature extractor networks. We propose a novel data-centric method, Implicit Cross-Domain Correspondences (iCDC). iCDC represents the same environment with multiple Neural Radiance Fields, each fitting the scene under individual visual domains. It utilizes the underlying 3D representations to generate accurate correspondences across different long-term visual conditions. Our proposed method enhances cross-domain localization performance, significantly reducing the performance gap. When evaluated on popular long-term localization benchmarks, our trained networks consistently outperform existing methods. This work serves as a substantial stride toward more robust visual localization pipelines for long-term deployments, and opens up research avenues in the development of long-term invariant descriptors. Copyright © 2023, The Authors. All rights reserved.

关键词： Feature extraction

Deep Learning-Enabled ISAC-OTFS Pre-equalization Design for Aerial-Terrestrial Networks

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Weihao Guo, Jing Wang, Siqiang Wang, Xinyi Yuan, Weijie Fei, Zesong the School of Information and Electronics Beijing Institute of Technology Beijing100081 China the School of System Design and Intelligent Manufacturing The Shenzhen Key Laboratory of Robotics and Computer Vision Southern University of Science and Technology Shenzhen518055 China

Orthogonal time frequency space (OTFS) modulation has been viewed as a promising technique for integrated sensing and communication (ISAC) systems and aerial-terrestrial networks, due to its delay-Doppler domain transmission property and strong Doppler-resistance capability. However, it also suffers from high processing complexity at the receiver. In this work, we propose a novel pre-equalization based ISAC-OTFS transmission framework, where the terrestrial base station (BS) executes pre-equalization based on its estimated channel state information (CSI). In particular, the mean square error of OTFS symbol demodulation and Cramér-Rao lower bound of sensing parameter estimation are derived, and their weighted sum is utilized as the metric for optimizing the pre-equalization matrix. To address the formulated problem while taking the time-varying CSI into consideration, a deep learning enabled channel prediction-based pre-equalization framework is proposed, where a parameter-level channel prediction module is utilized to decouple OTFS channel parameters, and a low-dimensional prediction network is leveraged to correct outdated CSI. A CSI processing module is then used to initialize the input of the pre-equalization module. Finally, a residual-structured deep neural network is cascaded to execute pre-equalization. Simulation results show that under the proposed framework, the demodulation complexity at the receiver as well as the pilot overhead for channel estimation, are significantly reduced, while the symbol detection performance approaches those of conventional minimum mean square error equalization and perfect CSI. Copyright © 2024, The Authors. All rights reserved.

关键词： Channel estimation

X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Guo, Pinxue Li, Wanyun Huang, Hao Hong, Lingyi Zhou, Xinyu Chen, Zhaoyu Li, Jinglun Jiang, Kaixun Zhang, Wei Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China

Multi-modal Video Object Segmentation (VOS), including RGB-Thermal, RGB-Depth, and RGB-Event, has garnered attention due to its capability to address challenging scenarios where traditional VOS methods struggle, such as extreme illumination, rapid motion, and background distraction. Existing approaches often involve designing specific additional branches and performing full-parameter fine-tuning for fusion in each task. However, this paradigm not only duplicates research efforts and hardware costs but also risks model collapse with the limited multi-modal annotated data. In this paper, we propose a universal framework named X-Prompt for all multi-modal video object segmentation tasks, designated as RGB+X. The X-Prompt framework first pretrains a video object segmentation foundation model using RGB data, and then utilize the additional modality of the prompt to adapt it to downstream multi-modal tasks with limited data. Within the X-Prompt framework, we introduce the Multi-modal Visual Prompter (MVP), which allows prompting foundation model with the various modalities to segment objects precisely. We further propose the Multi-modal Adaptation Experts (MAEs) to adapt the foundation model with pluggable modality-specific knowledge without compromising the generalization capacity. To evaluate the effectiveness of the X-Prompt framework, we conduct extensive experiments on 3 tasks across 4 benchmarks. The proposed universal X-Prompt framework consistently outperforms the full fine-tuning paradigm and achieves state-of-the-art performance. Code: https://***/PinxueGuo/***. Copyright © 2024, The Authors. All rights reserved.

关键词： Image segmentation

Two-stream Dynamic Heterogeneous Graph Recurrent Neural Network for Multi-label Multi-modal Emotion Recognition

学校读者我要写书评

暂无评论

IEEE Transactions on Affective Computing 2025年

作者： Wang, Jing Feng, Zhiyang Ning, Xiaojun Lin, Youfang Chen, Badong Jia, Ziyu Beijing Jiaotong University Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence School of Computer Science and Technology Beijing China Xi'an Jiaotong University Institute of Artificial Intelligence and Robotics Xi'an710049 China Beijing Jiaotong University School of Computer Science and Technology Beijing100044 China Chinese Academy of Sciences Brainnetome Center Institute of Automation Beijing100190 China

The study of the relationship between emotions and physiological signals of subjects under multimedia stimulation is an emerging field, and many important advances are made. However, there are still some challenges: 1) How to effectively utilize the complementarity among spatial-spectral-temporal domain information. 2) How to employ the heterogeneity and the correlation among multi-modal physiological signals simultaneously. 3) How to improve the robustness of the model dealing with missing channels. 4) How to model the dependency among different emotions. In this paper, we propose a novel two-stream Dynamic Heterogeneous Graph Recurrent Neural Network called DHGRNN. Specifically, DHGRNN consists of a spatial-temporal stream, a spatial-spectral stream, a fusion layer, and a multi-label classifier. Each stream is composed of a graph transformer network, evolved graph convolutional neural network, and gated recurrent units. We propose a graph-based two-stream structure to fuse the information of the spatial-spectral-temporal domain simultaneously. Graph transformer network and evolved graph convolutional neural network are used to model the heterogeneity and correlation of multi-modal physiological signals, respectively. To deal with the problem of robustness in the face of missing channel data, we transform it into the problem of dynamic graphs and use a dynamic graph neural network to improve the robustness. In addition, we propose a multi-label classifier to model the dependency among different emotion dimensions. Experiments on three public datasets demonstrate that our proposed model outperforms existing state-of-the-art methods. © 2010-2012 IEEE.

关键词： Graph neural networks

A study on the noise reduction method in turbid water environments using Spatial frequency characteristics

学校读者我要写书评

暂无评论

A study on the noise reduction method in turbid water enviro...

International Symposium on Environmentally Conscious Design and Inverse Manufacturing (EcoDesign)

作者： Ayumu Koura Hyun-Woo Kim Myungjin Cho Min-Chul Lee Graduate School of Computer Science and Systems Engineering Kyushu Institute of Technology Iizuka city Japan School of ICT Robotics and Mechanical Engineering IITC Hankyong National University Anseong-si Korea

ISBN: (数字)9798350363043

ISBN: (纸本)9798350363050

Scattering noise reduction is a challenging project to remove noise under scattering media conditions such as fog or turbid water. In previous study, blurring caused by scattering medium particles in fog or turbid water was removed by low-pass filtering. However, in this method, the accurate color information of the object may be lost due to rough low-frequency filtering. In addition, it is difficult to the obtain object shape information in heavy turbid environments. To solve this problem, we propose a new de-scattering algorithm that combines Fourier-domain filtering and integral imaging by creating a filter tailored to the frequency to be removed. To prove the proposed method, we perform a numerical analysis of experimental results using conventional and proposed methods.

关键词： Water Three-dimensional displays Image color analysis Shape Numerical analysis Noise reduction Scattering Reconstruction algorithms Colored noise Image reconstruction

Unified regularity measures for sample-wise learning and generalization

学校读者我要写书评

暂无评论

Visual Intelligence 2024年第1期2卷 1-20页

作者： Zhang, Chi Yuan, Meng Ma, Xiaoning Liu, Yu Lu, Haoang Wang, Le Su, Yuanqi Liu, Yuehu Institute of Artificial Intelligence and Robotics Xi’an Jiaotong University Xi’an 710049 China National KeyLaboratory of Human-Machine Hybrid Augmented Intelligence Xi’an Jiaotong University Xi’an 710049 China National Engineering Research Center for Visual Information and Applications Xi’an Jiaotong University Xi’an 710049 China School of Computer Science and Technology Xi’an Jiaotong University Xi’an 710049 China

Fundamental machine learning theory shows that different samples contribute unequally to both the learning and testing processes. Recent studies on deep neural networks (DNNs) suggest that such sample differences are rooted in the distribution of intrinsic pattern information, namely sample regularity. Motivated by recent discoveries in network memorization and generalization, we propose a pair of sample regularity measures with a formulation-consistent representation for both processes. Specifically, the cumulative binary training/generalizing loss (CBTL/CBGL), the cumulative number of correct classifications of the training/test sample within the training phase, is proposed to quantify the stability in the memorization-generalization process, while forgetting/mal-generalizing events (ForEvents/MgEvents), i.e., the misclassification of previously learned or generalized samples, are utilized to represent the uncertainty of sample regularity with respect to optimization dynamics. The effectiveness and robustness of the proposed approaches for mini-batch stochastic gradient descent (SGD) optimization are validated through sample-wise analyses. Further training/test sample selection applications show that the proposed measures, which share the unified computing procedure, could benefit both tasks. © The Author(s) 2024.

关键词： Forgetting events Memorization-generalization mechanism Network generalization Sample regularity Sample reweighting

Crop protection and disease detection using artificial intelligence and computer vision: a comprehensive review

学校读者我要写书评

暂无评论

Multimedia Tools and Applications 2025年第7期84卷 3723-3743页

作者： Shah, Kanish Sushra, Rajat Shah, Manan Shah, Dhairya Shah, Haard Raval, Megh Prajapati, Mitul Software Engineer Walmart SunnyvaleCA United States Atmiya Vidya Mandir Gujarat Surat India Department of Chemical Engineering School of Energy Technology Pandit Deendayal Energy University Gujarat Gandhinagar India Robotics ECE Concentration Northeastern University BostonMA United States Department of Computer Science Concentration in AI Hofstra University HempsteadNY United States Department of Chemical Engineering S. S Agrawal Institute of Engineering & Technology Gujarat Navsari India R &D Engineer Span Inspection Systems Pvt. Ltd Gujarat Gandhinagar India

The technological advancements in the field of agriculture have increased to a great extent in recent years, and many techniques have evolved from other techniques. Some methods are improved or upgraded from the previous versions by implementing a new model or using better hardware devices. This has been helpful for the farmers in increasing crop productivity, and the life expectancy of crops has also increased as the diseases inside or outside the crops can be detected much earlier, and learning at an early stage helps prevent other crops. In this paper, we have presented a study where many varieties of fruits and vegetables have been taken to determine which method was used for a particular crop. By analyzing the various works carried out by the authors, it was inferred that most of the works revolved around image processing and hyperspectral imaging. Due to this, we had also included most of the papers, particularly as the models and hardware components used were much better than other works. Then, a comparative study was done where different fruits and vegetables highlighted the two main areas: the method used and the accuracy obtained. © The Author(s), under exclusive licence to Springer science+Business Media, LLC, part of Springer Nature 2024.

关键词： Crops