检索结果-内蒙古大学图书馆

Pay attention to doctor-patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis

引用

INFORMATION FUSION 2021年 75卷 168-185页

作者： Zheng, Wenbo Yan, Lan Gou, Chao Zhang, Zhi-Cheng Zhang, Jun Jason Hu, Ming Wang, Fei-Yue Xi An Jiao Tong Univ Sch Software Engn Xian 710049 Peoples R China Chinese Acad Sci Inst Automat State Key Lab Management & Control Complex Syst Beijing 100190 Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 100190 Peoples R China Sun Yat Sen Univ Sch Intelligent Syst Engn Guangzhou 510275 Peoples R China Gen Hosp Peoples Liberat Army Med Ctr 7 Beijing 100700 Peoples R China Wuhan Univ Sch Elect Engn & Automat Wuhan 430072 Peoples R China Wuhan Pulm Hosp Intens Care Unit Wuhan 430030 Peoples R China

The sudden increase in coronavirus disease 2019 (COVID-19) cases puts high pressure on healthcare services worldwide. At this stage, fast, accurate, and early clinical assessment of the disease severity is vital. In general, there are two issues to overcome: (1) Current deep learning-based works suffer from multimodal data adequacy issues;(2) In this scenario, multimodal (e.g., text, image) information should be taken into account together to make accurate inferences. To address these challenges, we propose a multi-modal knowledge graph attention embedding for COVID-19 diagnosis. Our method not only learns the relational embedding from nodes in a constituted knowledge graph but also has access to medical knowledge, aiming at improving the performance of the classifier through the mechanism of medical knowledge attention. The experimental results show that our approach significantly improves classification performance compared to other state-of-the-art techniques and possesses robustness for each modality from multi-modal data. Moreover, we construct a new COVID-19 multi-modal dataset based on text mining, consisting of 1393 doctor-patient dialogues and their 3706 images (347 X-ray + 2598 CT + 761 ultrasound) about COVID-19 patients and 607 non-COVID-19 patient dialogues and their 10754 images (9658 X-ray + 494 CT + 761 ultrasound), and the fine-grained labels of all. We hope this work can provide insights to the researchers working in this area to shift the attention from only medical images to the doctor-patient dialogue and its corresponding medical images.

关键词： COVID-19 diagnose knowledge attention mechanism knowledge-based representation learning knowledge embedding

来源：评论

学校读者我要写书评

暂无评论

KM⁴: Visual reasoning via knowledge EmbeddingMemoryModel with MutualModulation

引用

INFORMATION FUSION 2021年 67卷 14-28页

作者： Zheng, Wenbo Yan, Lan Gou, Chao Wang, Fei-Yue Xi An Jiao Tong Univ Sch Software Engn Xian 710049 Peoples R China Chinese Acad Sci Inst Automat State Key Lab Management & Control Complex Syst Beijing 100190 Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 100190 Peoples R China Sun Yat Sen Univ Sch Intelligent Syst Engn Guangzhou 510275 Peoples R China

Visual reasoning is a special kind of visual question answering, which is essentially multi-step and compositional, and also requires intensive text-visual interaction. The most important and challenging problem of visual reasoning is to design an effective and robust visual reasoning model. To this end, there are two challenges to overcome. The first is that textual and visual information must be jointly considered to make accurate inferences about reasoning. The second is that existing deep learning-based works are often too specific to a particular task. To address these issues, we propose a knowledge memory embedding model with mutual modulation for visual reasoning. This approach learns not only knowledge-based embeddings derived from key-value memory network to make the full and joint of textual and visual information, but also exploits the prior knowledge to improve the performance with knowledge-based representation learning for applying other general reasoning tasks. Experimental results on four benchmarks show that the proposed approach significantly improves performance compared with other state-of-the-art methods, guarantees the robustness with our model. Most importantly, we apply our model to four reasoning tasks, and experimentally show that our model effectively supports relational reasoning and improves performance in several tasks and datasets.

关键词： Visual reasoning knowledge-based representation learning Memory network knowledge embedding

来源：评论

学校读者我要写书评

暂无评论

GRAPH ATTENTION MODEL EMBEDDED WITH MULTI-MODAL knowledge FOR DEPRESSION DETECTION

GRAPH ATTENTION MODEL EMBEDDED WITH MULTI-MODAL KNOWLEDGE FO...

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Zheng, Wenbo Yan, Lan Gou, Chao Wang, Fei-Yue Xi An Jiao Tong Univ Sch Software Engn Xian 710049 Peoples R China Chinese Acad Sci Inst Automat State Key Lab Management & Control Complex Syst Beijing 100190 Peoples R China Sun Yat Sen Univ Sch Intelligent Syst Engn Guangzhou Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 100049 Peoples R China

ISBN: (纸本)9781728113319

With more than 300 million people depressed worldwide annually, depression is a global problem. The goal of depression detection is to improve diagnostic accuracy and availability, leading to faster intervention. The most important and challenging problem here is to design an effective and robust depression detection model. To this end, there are two challenges to overcome: 1) Multi-modal (audio, image, text, etc.) information must be jointly considered to make accurate inferences. 2) Existing deep learning-based work suffers from multi-modal data sufficiency problem. To address these issues, we propose a graph attention model embedded with multi-modal knowledge for depression detection. This approach learns not only reasonable embeddings for nodes in the knowledge graph, but also exploits medical knowledge to improve the performance of classification and prediction with the knowledge attention mechanism. Experimental results on the real-world datasets show that the proposed approach significantly improves the classification and prediction performance compared with other major state-of-the-art approaches, with guaranteed the robustness with each modality of multi-modal data. Overall, this paper shows how multi-modal knowledge attention mechanism and deep-learning-based networks can be combined to assist mental health patients and practitioners.

关键词： Depression detection knowledge-based representation learning knowledge embedding

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：