检索结果-内蒙古大学图书馆

Visual Relationship Detection With Image Position and Feature Information Embedding and Fusion

IEEE ACCESS 2022年 10卷 117170-117176页

作者： Peng, Jinghui Zhang, Ying Huang, Weichun Mil Acad Sci Inst Syst Engn Beijing 100101 Peoples R China

Visual relationship detection (VRD) is an important direction in the field of image processing, and it is a research task to explore object relationships based on object recognition and localization regression. At the same time, it is also one of the key contents of scene graph generation and construction of multimodal knowledge graph. In order to better improve the visual relationship detection effect, the fusion method of image location and feature information embedding is adopted on the vrd dataset. First, analyze the causes of errors in existing methods, and build a pre-training pre-dataset vrd_P for data enhancement;then, combine the characteristics of the vrd dataset to establish an entities relationship network (Entities-net) to guide target recognition;finally, a visual relationship detection model of LTransE is proposed to achieve the joint representation of target features and target locations for relationship detection. The results show that the method of embedding fusion of image location and feature information can effectively improve the visual relationship detection effect.

关键词： VRD data-enhancement VTransE scene graph generation LTransE

来源：评论

学校读者我要写书评

暂无评论

SANGRIA: Surgical Video scene graph Optimization for Surgical Workflow Prediction 6th

SANGRIA: Surgical Video Scene Graph Optimization for Surgic...

引用

6th International Workshop on graphs in Biomedical Image Analysis, GRAIL 2024

作者： Köksal, Çağhan Ghazaei, Ghazal Holm, Felix Farshad, Azade Navab, Nassir Carl Zeiss AG Oberkochen Germany Technische Universität München Munich Germany Munich Center for Machine Learning Munich Germany

ISBN: (纸本)9783031832420

graph-based holistic scene representations facilitate surgical workflow understanding and have recently demonstrated significant success. However, this task is often hindered by the limited availability of densely annotated surgical scene data. In this work, we introduce an end-to-end framework for the generation and optimization of surgical scene graphs on a downstream task. Our approach leverages the flexibility of graph-based spectral clustering and the generalization capability of foundation models to generate unsupervised scene graphs with learnable properties. We reinforce the initial spatial graph with sparse temporal connections using local matches between consecutive frames to predict temporally consistent clusters across a temporal neighborhood. By jointly optimizing the spatiotemporal relations and node features of the dynamic scene graph with the downstream task of phase segmentation, we address the costly and annotation-burdensome task of semantic scene comprehension and scene graph generation in surgical videos using only weak surgical phase labels. Further, by incorporating effective intermediate scene representation disentanglement steps within the pipeline, our solution outperforms the SOTA on the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow recognition. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

关键词： scene graph generation Surgical Phase Segmentation Surgical scene Understanding Unsupervised Video Segmentation

来源：评论

学校读者我要写书评

暂无评论

Haystack: A Panoptic scene graph Dataset to Evaluate Rare Predicate Classes

Haystack: A Panoptic Scene Graph Dataset to Evaluate Rare Pr...

引用

IEEE/CVF International Conference on Computer Vision (ICCV)

作者： Lorenz, Julian Barthel, Florian Kienzle, Daniel Lienhart, Rainer Univ Augsburg Augsburg Germany

ISBN: (纸本)9798350307443

Current scene graph datasets suffer from strong long-tail distributions of their predicate classes. Due to a very low number of some predicate classes in the test sets, no reliable metrics can be retrieved for the rarest classes. We construct a new panoptic scene graph dataset and a set of metrics that are designed as a benchmark for the predictive performance especially on rare predicate classes. To construct the new dataset, we propose a model-assisted annotation pipeline that efficiently finds rare predicate classes that are hidden in a large set of images like needles in a haystack. to prior scene graph datasets, Haystack contains explicit negative annotations, i.e. annotations that a given relation does not have a certain predicate class. Negative annotations are helpful especially in the field of scene graph generation and open up a whole new set of possibilities to improve current scene graph generation models. Haystack is 100% compatible with existing panoptic scene graph datasets and can easily be integrated with existing evaluation pipelines. Our dataset and code can be found here: https://***/ haystack/. It includes annotation files and simple to use scripts and utilities, to help with integrating our dataset in existing work.

关键词： dataset evaluation metrics scene graph scene graph generation

来源：评论

学校读者我要写书评

暂无评论

Configurable graph Reasoning for Visual Relationship Detection

引用

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2022年第1期33卷 117-129页

作者： Zhu, Yi Liang, Xiwen Lin, Bingqian Ye, Qixiang Jiao, Jianbin Lin, Liang Liang, Xiaodan Univ Chinese Acad Sci UCAS Beijing Peoples R China Sun Yat Sen Univ Guangzhou Peoples R China

Visual commonsense knowledge has received growing attention in the reasoning of long-tailed visual relationships biased in terms of object and relation labels. Most current methods typically collect and utilize external knowledge for visual relationships by following the fixed reasoning path of {subject, object -> predicate} to facilitate the recognition of infrequent relationships. However, the knowledge incorporation for such fixed multidependent path suffers from the data set biased and exponentially grown combinations of object and relation labels and ignores the semantic gap between commonsense knowledge and real scenes. To alleviate this, we propose configurable graph reasoning (CGR) to decompose the reasoning path of visual relationships and the incorporation of external knowledge, achieving configurable knowledge selection and personalized graph reasoning for each relation type in each image. Given a commonsense knowledge graph, CGR learns to match and retrieve knowledge for different subpaths and selectively compose the knowledge routed path. CGR adaptively configures the reasoning path based on the knowledge graph, bridges the semantic gap between the commonsense knowledge, and the real-world scenes and achieves better knowledge generalization. Extensive experiments show that CGR consistently outperforms previous state-of-the-art methods on several popular benchmarks and works well with different knowledge graphs. Detailed analyses demonstrated that CGR learned explainable and compelling configurations of reasoning paths.

关键词： graph learning scene graph generation visual reasoning visual relationship detection (VRD)

来源：评论

学校读者我要写书评

暂无评论

Learning multimodal relationship interaction for visual relationship detection

引用

PATTERN RECOGNITION 2022年 132卷

作者： Liu, Zhixuan Zheng, Wei-Shi Sun Yat Sen Univ Sch Comp Sci & Engn Guangzhou Peoples R China

Visual relationship detection aims to recognize visual relationships in scenes as triplets (subject-predicate-object). Previous works have shown remarkable progress by introducing multimodal features, external linguistics, scene context, etc. Due to the loss of informative multimodal hyper-relations (i.e. relations of relationships), the meaningful contexts of relationships are not fully captured yet, which limits the reasoning ability. In this work, we propose a Multimodal Similarity Guided Relationship Interaction Network (MSGRIN) to explicitly model the relations of relationships in graph neural network paradigm. In a visual scene, the MSGRIN takes the visual relationships as nodes to construct an adaptive graph and enhances deep message passing by introducing Entity Appearance Reconstruction, Entity Relevance Filtering and Multimodal Similarity Attention. We have conducted extensive experiments on two datasets: Visual Relationship Detection (VRD) and Visual Genome (VG). The evaluation results demonstrate that the proposed MSGRIN has empirically performed more effectively overall. (C) 2022 Published by Elsevier Ltd.

关键词： Visual relationship detection scene graph generation Relationship context Multimodal relationship interaction

来源：评论

学校读者我要写书评

暂无评论

Semantic Privacy-Preserving for Video Surveillance Services on the Edge 8

Semantic Privacy-Preserving for Video Surveillance Services ...

引用

8th Annual IEEE/ACM Symposium on Edge Computing (SEC)

作者： Huang, Alexander Y. C. Chen, Yitao Huang, Dijiang Zhao, Ming Arizona State Univ Hamilton High Tempe AZ 85287 USA Arizona State Univ Tempe AZ 85287 USA

ISBN: (纸本)9798400701238

Intelligent Video surveillance systems, leveraging edge computing, have become increasingly prevalent in various facilities, providing advanced monitoring and management capabilities. However, these systems can inadvertently compromise personally identifiable information, such as human images, leading to privacy violations. We introduced a semantic privacy-preserving video surveillance service on the edge to address this critical issue. Unlike traditional centralized models, the solution operates as a decentralized machine learning framework within the video surveillance infrastructure at the edge. Its primary focus is protecting private information extracted from captured video streaming data. This research integrates cutting-edge machine learning techniques, including scene graph generation and semantic communication approaches, by enabling edge nodes to exchange parameters for training, referencing, and safeguarding data privacy and ownership. These innovations collectively contribute to the protection of human privacy. The performance evaluation confirms that the solution is an efficient and effective privacy protection platform, offering a significant advancement over conventional centralized solutions.

关键词： Distributed training privacy preservation edge computing scene graph generation semantic communication

来源：评论

学校读者我要写书评

暂无评论

Naive scene graphs : How Visual is Modern Visual Relationship Detection? 20

Naive Scene Graphs : How Visual is Modern Visual Relationshi...

引用

20th Conference on Robots and Vision (CRV)

作者： Abou Chacra, David Zelek, John Univ Waterloo Dept Syst Design Engn Waterloo ON Canada

ISBN: (纸本)9798350341393

Modern approaches to scene graph generation still struggle with their performance, with even state of the art approaches hovering under a 15% mean recall on certain evaluation modes. This poor performance is partially a result of networks heavily relying and fixating on non-visual data, such as class statistics, instead of the pixel-level signals present in the images. We demonstrate this by examining the 'visual-ness' of visual relationship detection approaches. We first describe and implement a new Naive Bayes-based statistical baseline for scene graph generation. Most notably, this basic classifier does not utilize the image pixels, but relies on the properties of the bounding boxes (class labels, topological configuration,... etc.) to predict the relationship labels. We demonstrate that our classical machine learning approach, one as simple as a categorical Naive Bayes classifier, can perform relationship detection in a manner that achieves relatively competitive performance to that of modern scene graph generators. This is an alarming finding regarding scene graph generation that implies that visual data in images may not be utilized in modern visual relationship detection past the point of object detection. We finally discuss how more visual modern approaches to scene graph generation appear to remedy some of these shortcomings.

关键词： scene graph generation Visual Relationship Detection Explainability Statistical Approaches

来源：评论

学校读者我要写书评

暂无评论

DIFFERENCE-AWARE ITERATIVE REASONING NETWORK FOR KEY RELATION DETECTION

DIFFERENCE-AWARE ITERATIVE REASONING NETWORK FOR KEY RELATIO...

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Zhao, Bowen Chen, Weidong Hu, Bo Xie, Hongtao Mao, Zhendong Univ Sci & Technol China Hefei Peoples R China

ISBN: (纸本)9781665468916

scene graph serves as a crucial visual representation of an image, with salient objects providing richer semantics for detecting key relations. However, most methods use a one-step reasoning manner for key relation detection, which may not utilize potential clues effectively. Humans usually review and revise to achieve the final answer, and semantics of relations offer further linguistic clues. Therefore, we propose the Difference-aware Iterative Reasoning Network (DIRNet) to predict key relations in a multi-step manner. Our model estimates visual saliency, encodes contexts globally with message passing, and then refines predictions iteratively by considering the difference in predicted relation semantics and contextual information across iterations. Extensive experiments show that our model outperforms state-of-the-art methods in key relation prediction on the VG-KR benchmark, and achieves competitive results in common relation prediction on VG, demonstrating its generalization and superiority.

关键词： scene graph generation visual relation detection key relation multi-step reasoning message passing

来源：评论

学校读者我要写书评

暂无评论

Align R-CNN: A Pairwise Head Network for Visual Relationship Detection

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2022年 24卷 1266-1276页

作者： Tajrobehkar, Mitra Tang, Kaihua Zhang, Hanwang Lim, Joo-Hwee Nanyang Technol Univ SCSE Singapore 67904777 Singapore ASTAR Inst Infocomm Res Singapore 119613 Singapore

scene graphs connect individual objects with visual relationships. They serve as a comprehensive scene representation for downstream multimodal tasks. However, by exploring recent progress in scene graph generation (SGG), we find that the performance of recent works is highly limited by the pairwise relationship modeling by naive feature concatenation. Such pairwise features lack sufficient object interaction due to the mis-aligned object parts, resulting in non-discriminative pairwise features for visual relationship prediction. For example, naive concatenated pairwise feature usually make the model fail to discriminate between riding and feeding for object pair person and horse. To this end, we design a meta-architecture- learning-to-align - for dynamic object feature concatenation. We call our model: Align R-CNN. Specifically, we introduce a novel attention-based multiple region alignment module that can be jointly optimized with SGG. Experiments on the large-scale SGG benchmark Visual Genome show that the proposed Align R-CNN can replace the naive feature concatenation and thus boost all the existing SGG methods.

关键词： Feature extraction Visualization Task analysis Predictive models Semantics Context modeling Data mining Pairwise feature alignment scene graph generation visual attention

来源：评论

学校读者我要写书评

暂无评论

Adaptive Multimodal Fusion with Cross-Attention for Robust scene Segmentation and Urban Economic Analysis

引用

APPLIED SCIENCES-BASEL 2025年第1期15卷 438-438页

作者： Zhong, Chun Zeng, Shihong Zhu, Hongqiu Hunan Univ Sci & Technol Xiangtan 411199 Peoples R China Cent South Univ Sch Automat Changsha 410017 Peoples R China

With the increasing demand for accurate multimodal data analysis in complex scenarios, existing models often struggle to effectively capture and fuse information across diverse modalities, especially when data include varying scales and levels of detail. To address these challenges, this study presents an enhanced Swin Transformer V2-based model designed for robust multimodal data processing. The method analyzes urban economic activities and spatial layout using satellite and street view images, with applications in traffic flow and business activity intensity, highlighting its practical significance. The model incorporates a multi-scale feature extraction module into the window attention mechanism, combining local and global window attention with adaptive pooling to achieve comprehensive multi-scale feature fusion and representation. This approach enables the model to effectively capture information at different scales, enhancing its expressiveness in complex scenes. Additionally, a cross-attention-based multimodal feature fusion mechanism integrates spatial structure information from scene graphs with Swin Transformer's image classification outputs. By calculating similarities and correlations between scene graph embeddings and image classifications, this mechanism dynamically adjusts each modality's contribution to the fused representation, leveraging complementary information for a more coherent multimodal understanding. Compared with the baseline method, the proposed bimodal model performs superiorly and the accuracy is improved by 3%, reaching 91.5%, which proves its effectiveness in processing and fusing multimodal information. These results highlight the advantages of combining multi-scale feature extraction and cross-modal alignment to improve performance on complex multimodal tasks.

关键词： bimodal analysis scene graph generation graph embedding economic activity prediction multimodal data fusion urban economics

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：