检索结果-内蒙古大学图书馆

Multi-semantic decoding of Visual Perception with Graph Neural Networks

INTERNATIONAL JOURNAL OF NEURAL SYSTEMS 2024年第4期34卷 2450016页

作者： Li, Rong Li, Jiyi Wang, Chong Liu, Haoxiang Liu, Tao Wang, Xuyang Zou, Ting Huang, Wei Yan, Hongmei Chen, Huafu Univ Elect Sci & Technol China Sichuan Prov Peoples Hosp Ctr Psychosomat Med Sichuan Prov Ctr Mental Hlth R China Chengdu 611731 Peoples R China Univ Elect Sci & Technol China Sch Life Sci & Technol Chengdu 611731 Peoples R China Univ Elect Sci & Technol China Key Lab Sichuan Prov MOE Key Lab Neuroinformat High Field Magnet Resonance Brain Imaging Chengdu 611731 Peoples R China

Constructing computational decoding models to account for the cortical representation of semantic information plays a crucial role in understanding visual perception. The human visual system processes interactive relationships among different objects when perceiving the semantic contents of natural visions. However, the existing semantic decoding models commonly regard categories as completely separate and independent visually and semantically and rarely consider the relationships from prior information. In this work, a novel semantic graph learning model was proposed to decode multiple semantic categories of perceived natural images from brain activity. The proposed model was validated on the functional magnetic resonance imaging data collected from five normal subjects while viewing 2750 natural images comprising 52 semantic categories. The results showed that the Graph Neural Network-based decoding model achieved higher accuracies than other deep neural network models. Moreover, the co-occurrence probability among semantic categories showed a significant correlation with the decoding accuracy. Additionally, the results suggested that semantic content organized in a hierarchical way with higher visual areas was more closely related to the internal visual experience. Together, this study provides a superior computational framework for multi-semantic decoding that supports the visual integration mechanism of semantic processing.

关键词： fMRI natural images visual perception semantic decoding graph neural network

来源：评论

学校读者我要写书评

暂无评论

Multi-subject data augmentation for target subject semantic decoding with deep multi-view adversarial learning

引用

INFORMATION SCIENCES 2021年 547卷 1025-1044页

作者： Li, Dan Du, Changde Wang, Shengpei Wang, Haibao He, Huiguang Chinese Acad Sci Inst Automat Res Ctr Brain Inspired Intelligence Beijing 100190 Peoples R China Chinese Acad Sci Inst Automat Natl Lab Pattern Recognit Beijing 100190 Peoples R China Univ Chinese Acad Sci Beijing 100190 Peoples R China Huawei Cloud BU EI Innovat Lab Beijing 100085 Peoples R China Chinese Acad Sci Ctr Excellence Brain Sci & Intelligence Technol Beijing 100190 Peoples R China

Functional magnetic resonance imaging (fMRI) is widely used in the field of brain semantic decoding. However, as fMRI data acquisition is time-consuming and expensive, the number of samples is usually small in the existing fMRI datasets. It is difficult to build an accurate brain decoding model for a subject with insufficient fMRI data. The majority of semantic decoding methods focus on designing predictive model with limited samples, while less attention is paid to fMRI data augmentation. Leveraging data from related but different subjects can be regarded as a new strategy to improve the performance of predictive model. There are two challenges when using information from different subjects: 1) feature mismatch;2) distribution mismatch. In this paper, we propose a multi-subject fMRI data augmentation method to address the above two challenges, which can improve the decoding accuracy of the target subject. Specifically, the subject information can be translated from one to another by using multiple subject-specific encoders, decoders and discriminators. The encoder maps each subject to a shared latent space, solving the feature mismatch problem. The decoders and discriminators form multiple generative adversarial network architectures, which solves the distribution mismatch problem. Meanwhile, to ensure that the representation of the latent space preserves information of the input space, our method not only minimizes the local data reconstruction loss, but also preserves the sparse reconstruction (semantic) relation over the whole dataset of the input space. Extensive experiments on three fMRI datasets demonstrate the effectiveness of the proposed method. (C) 2020 Elsevier Inc. All rights reserved.

关键词： Data augmentation semantic decoding Multi-view adversarial learning Sparse reconstruction relation

来源：评论

学校读者我要写书评

暂无评论

semantic language decoding across participants and stimulus modalities

引用

CURRENT BIOLOGY 2025年第5期35卷 1023-1032.e6页

作者： Tang, Jerry Huth, Alexander G. Univ Texas Austin Dept Comp Sci Austin TX 78712 USA Univ Texas Austin Dept Neurosci Austin TX 78712 USA

Brain decoders that reconstruct language from semantic representations have the potential to improve communication for people with impaired language production. However, training a semantic decoder for a participant currently requires many hours of brain responses to linguistic stimuli, and people with impaired language production often also have impaired language comprehension. In this study, we tested whether language can be decoded from a goal participant without using any linguistic training data from that participant. We trained semantic decoders on brain responses from separate reference participants and then used functional alignment to transfer the decoders to the goal participant. Cross-participant decoder predictions were semantically related to the stimulus words, even when functional alignment was performed using movies with no linguistic content. To assess how much semantic representations are shared between language and vision, we compared functional alignment accuracy using story and movie stimuli and found that performance was comparable in most cortical regions. Finally, we tested whether cross-participant decoders could be robust to lesions by excluding brain regions from the goal participant prior to functional alignment and found that cross-participant decoders do not depend on data from any single brain region. These results demonstrate that cross-participant decoding can reduce the amount of linguistic training data required from a goal participant and potentially enable language decoding from participants who struggle with both language production and language comprehension.

关键词： fMRI language decoding semantic decoding functional alignment language vision

来源：评论

学校读者我要写书评

暂无评论

Neural decoding of semantic concepts: a systematic literature review

引用

JOURNAL OF NEURAL ENGINEERING 2022年第2期19卷 021002-021002页

作者： Rybar, Milan Daly, Ian Univ Essex Sch Comp Sci & Elect Engn Brain Comp Interfacing & Neural Engn Lab Colchester Essex England

Objective. semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach. We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Main results. Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance. semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focu

关键词： semantic decoding conceptual decoding literature review functional magnetic resonance imaging (fMRI) functional near infrared spectroscopy (fNIRS) electroencephalography (EEG) intracranial electrodes

来源：评论

学校读者我要写书评

暂无评论

decoding of semantic categories of imagined concepts of animals and tools in fNIRS

引用

JOURNAL OF NEURAL ENGINEERING 2021年第4期18卷

作者： Rybar, Milan Poli, Riccardo Daly, Ian Univ Essex Sch Comp Sci & Elect Engn Brain Comp Interfacing & Neural Engn Lab Colchester Essex England

Objective. semantic decoding refers to the identification of semantic concepts from recordings of an individual's brain activity. It has been previously reported in functional magnetic resonance imaging and electroencephalography. We investigate whether semantic decoding is possible with functional near-infrared spectroscopy (fNIRS). Specifically, we attempt to differentiate between the semantic categories of animals and tools. We also identify suitable mental tasks for potential brain-computer interface (BCI) applications. Approach. We explore the feasibility of a silent naming task, for the first time in fNIRS, and propose three novel intuitive mental tasks based on imagining concepts using three sensory modalities: visual, auditory, and tactile. Participants are asked to visualize an object in their minds, imagine the sounds made by the object, and imagine the feeling of touching the object. A general linear model is used to extract hemodynamic responses that are then classified via logistic regression in a univariate and multivariate manner. Main results. We successfully classify all tasks with mean accuracies of 76.2% for the silent naming task, 80.9% for the visual imagery task, 72.8% for the auditory imagery task, and 70.4% for the tactile imagery task. Furthermore, we show that consistent neural representations of semantic categories exist by applying classifiers across tasks. Significance. These findings show that semantic decoding is possible in fNIRS. The study is the first step toward the use of semantic decoding for intuitive BCI applications for communication.

关键词： semantic decoding mental imagery functional near-infrared spectroscopy (fNIRS) machine learning general linear model (GLM) brain– computer interface (BCI) concepts

来源：评论

学校读者我要写书评

暂无评论

A semantic Communication Approach for Multiscene Target Detection in Intelligent Vehicle Networks

引用

IEEE INTERNET OF THINGS JOURNAL 2024年第22期11卷 35877-35890页

作者： Wan, Zhiping Liu, Shaojiang Xu, Zhiming Ni, Weichuan Ye, Shitong Wang, Feng Guangzhou Xinhua Univ Dongguan 523133 Guangdong Peoples R China Guangzhou Huashang Coll Dept Data Sci Guangzhou Peoples R China

As intelligent vehicle networks become integral to modern transportation systems, the need for accurate and efficient multiscene target detection in complex urban environments is increasingly critical. Addressing the challenge of processing and transmitting extensive image data rapidly and accurately under dynamic road conditions, this article proposes the multiscene target detection semantic communication (MTDSC) method. MTDSC employs convolutional neural networks and region proposal networks for effective target detection in images, and further utilizes spatial pyramid pooling and long short-term memory (LSTM) networks to assign relevant semantic labels in varied scenarios. These labels are transformed into a transmissible encoding format using a convolutional semantic encoder and a reinforcement learning strategy. The transmission phase leverages variational self-encoders techniques for channel encoding and decoding, ensuring reliable semantic information transfer in complex wireless environments. A crucial aspect of this research is semantic decoding and image reconstruction. To recover original semantic information accurately, a structured support vector machine (SVM) loss function is utilized, designed to capture the structural relationships among labels and enhance the consistency between decoded and original semantic labels. Experimental results show that MTDSC effectively reconstructs semantic images across different road scenes, ensuring key information retention and high-quality image reconstruction, even in low-bit rate and complex scenarios.

关键词： semantics Intelligent vehicles Object detection Encoding Roads decoding Transportation Image reconstruction intelligent vehicular networks semantic decoding semantic encoding

来源：评论

学校读者我要写书评

暂无评论

A Theory of semantic Communication

引用

IEEE TRANSACTIONS ON MOBILE COMPUTING 2024年第12期23卷 12211-12228页

作者： Shao, Yulin Cao, Qi Gunduz, Deniz Univ Macau Dept Elect & Comp Engn State Key Lab Internet Things Smart City Macau 999078 Peoples R China Xidian Univ Xidian Guangzhou Res Inst Xian 710126 Peoples R China Imperial Coll London Dept Elect & Elect Engn London SW7 2AZ England

semantic communication is an emerging research area that has gained a wide range of attention recently. Despite this growing interest, there remains a notable absence of a comprehensive and widely-accepted framework for characterizing semantic communication. This paper introduces a new conceptualization of semantic communication and formulates two fundamental problems, which we term language exploitation and language design. Our contention is that the challenge of language design can be effectively situated within the broader framework of joint source-channel coding theory, underpinned by a comprehensive end-to-end distortion metric. To tackle the language exploitation problem, we put forth three approaches: semantic encoding, semantic decoding, and a synergistic combination of both in the form of combined semantic encoding and decoding. Furthermore, we establish the semantic distortion-cost region as a critical framework for assessing the language exploitation problem. For each of the three proposed approaches, the achievable distortion-cost region is characterized. Overall, this paper aims to shed light on the intricate dynamics of semantic communication, paving the way for a deeper understanding of this evolving field.

关键词： semantics Receivers Transmitters decoding Symbols Channel coding Distortion Joint source-channel coding large language model semantic communication semantic decoding semantic encoding

来源：评论

学校读者我要写书评

暂无评论

semantic RECONSTRUCTION OF CONTINUOUS LANGUAGE FROM MEG SIGNALS 49

SEMANTIC RECONSTRUCTION OF CONTINUOUS LANGUAGE FROM MEG SIGN...

引用

49th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Wang, Bo Xu, Xiran Zhang, Longxiang Xiao, Boda Wu, Xihong Chen, Jing Peking Univ Sch Intelligence Sci & Technol Speech & Hearing Res Ctr Beijing Peoples R China Peking Univ Coll Future Technol Natl Biomed Imaging Ctr Beijing Peoples R China Natl Key Lab Gen Artificial Intelligence Beijing Peoples R China

ISBN: (纸本)9798350344868;9798350344851

decoding language from neural signals holds considerable theoretical and practical importance. Previous research has indicated the feasibility of decoding text or speech from invasive neural signals. However, when using non-invasive neural signals, significant challenges are encountered due to their low quality. In this study, we proposed a data-driven approach for decoding semantic of language from Magnetoencephalography (MEG) signals recorded while subjects were listening to continuous speech. First, a multi-subject decoding model was trained using contrastive learning to reconstruct continuous word embeddings from MEG data. Subsequently, a beam search algorithm was adopted to generate text sequences based on the reconstructed word embeddings. Given a candidate sentence in the beam, a language model was used to predict the subsequent words. The word embeddings of the subsequent words were correlated with the reconstructed word embedding. These correlations were then used as a measure of the probability for the next word. The results showed that the proposed continuous word embedding model can effectively leverage both subject-specific and subject-shared information. Additionally, the decoded text exhibited significant similarity to the target text, with an average BERTScore of 0.816.

关键词： semantic decoding MEG brain-computer interface text generation

来源：评论

学校读者我要写书评

暂无评论

Spatiotemporal target selection for intracranial neural decoding of abstract and concrete semantics

引用

CEREBRAL CORTEX 2022年第24期32卷 5544-5554页

作者： Nagata, Keisuke Kunii, Naoto Shimada, Seijiro Fujitani, Shigeta Takasago, Megumi Saito, Nobuhito Univ Tokyo Dept Neurosurg Bunkyo Ku 7-3-1 Hongo Tokyo 1138655 Japan

decoding the inner representation of a word meaning from human cortical activity is a substantial challenge in the development of speech brain-machine interfaces (BMIs). The semantic aspect of speech is a novel target of speech decoding that may enable versatile communication platforms for individuals with impaired speech ability;however, there is a paucity of electrocorticography studies in this field. We decoded the semantic representation of a word from single-trial cortical activity during an imageability-based property identification task that required participants to discriminate between the abstract and concrete words. Using high gamma activity in the language-dominant hemisphere, a support vector machine classifier could discriminate the 2-word categories with significantly high accuracy (73.1 +/- 7.5%). Activities in specific time components from two brain regions were identified as significant predictors of abstract and concrete dichotomy. Classification using these feature components revealed that comparable prediction accuracy could be obtained based on a spatiotemporally targeted decoding approach. Our study demonstrated that mental representations of abstract and concrete word processing could be decoded from cortical high gamma activities, and the coverage of implanted electrodes and time window of analysis could be successfully minimized. Our findings lay the foundation for the future development of semantic-based speech BMIs.

关键词： abstract and concrete brain-machine interface electrocorticography language semantic decoding

来源：评论

学校读者我要写书评

暂无评论

Using data from cue presentations results in grossly overestimating semantic BCI performance

引用

SCIENTIFIC REPORTS 2024年第1期14卷 1-13页

作者： Rybar, Milan Poli, Riccardo Daly, Ian Univ Essex Sch Comp Sci & Elect Engn Brain Comp Interfaces & Neural Engn Lab Colchester CO4 3SQ England

Neuroimaging studies have reported the possibility of semantic neural decoding to identify specific semantic concepts from neural activity. This offers promise for brain-computer interfaces (BCIs) for communication. However, translating these findings into a BCI paradigm has proven challenging. Existing EEG-based semantic decoding studies often rely on neural activity recorded when a cue is present, raising concerns about decoding reliability. To address this, we investigate the effects of cue presentation on EEG-based semantic decoding. In an experiment with a clear separation between cue presentation and mental task periods, we attempt to differentiate between semantic categories of animals and tools in four mental tasks. By using state-of-the-art decoding analyses, we demonstrate significant mean classification accuracies up to 71.3% during cue presentation but not during mental tasks, even with adapted analyses from previous studies. These findings highlight a potential issue when using neural activity recorded during cue presentation periods for semantic decoding. Additionally, our results show that semantic decoding without external cues may be more challenging than current state-of-the-art research suggests. By bringing attention to these issues, we aim to stimulate discussion and drive advancements in the field toward more effective semantic BCI applications.

关键词： semantic decoding Mental imagery Cue presentation Electroencephalography (EEG) Machine learning Brain-computer interface (BCI) Functional magnetic resonance imaging (fMRI)

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：