检索结果-内蒙古大学图书馆

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

IEEE Transactions on Cognitive and Developmental Systems 2025年

作者： Wang, Xiao Rong, Yao Wu, Zongzhen Zhu, Lin Jiang, Bo Tang, Jin Tian, Yonghong Anhui University Institute of Artificial Intelligence Hefei Comprehensive National Science Center School of Computer Science and Technology Hefei230601 China Beijing Institute Of Technology Beijing China Peking University National Engineering Laboratory for Video Technology School of Electronics Engineering and Computer Science Beijing China Peng Cheng Laboratory Shenzhen China

Event camera-based pattern recognition is a newly arising research topic in recent years. Current researchers usually transform the event streams into images, graphs, or voxels, and adopt deep neural networks for event-based classification. Although good performance can be achieved on simple event recognition datasets, however, their results may be still limited due to the following two issues. Firstly, they adopt spatial sparse event streams for recognition only, which may fail to capture the color and detailed texture information well. Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition. However, few of them consider achieving a balance between these two aspects. In this paper, we formally propose to recognize patterns by fusing RGB frames and event streams simultaneously and propose a new RGB frame-event recognition framework to address the aforementioned issues. The proposed method contains four main modules, i.e., memory support Transformer network for RGB frame encoding, spiking neural network for raw event stream encoding, multi-modal bottleneck fusion module for RGB-Event feature aggregation, and prediction head. Due to the scarcity of RGB-Event based classification dataset, we also propose a large-scale PokerEvent dataset which contains 114 classes, and 27102 frame-event pairs recorded using a DVS346 event camera. Extensive experiments on two RGB-Event based classification datasets fully validated the effectiveness of our proposed framework. We hope this work will boost the development of pattern recognition by fusing RGB frames and event streams © 2016 IEEE.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

1000× Faster Camera and Machine Vision with Ordinary Devices

arXiv

引用

arXiv 2022年

作者： Huang, Tiejun Zheng, Yajing Yu, Zhaofei Chen, Rui Li, Yuan Xiong, Ruiqin Ma, Lei Zhao, Junwei Dong, Siwei Zhu, Lin Li, Jianing Jia, Shanshan Fu, Yihua Shi, Boxin Wu, Si Tian, Yonghong Department of Computer Science and Technology National Engineering Laboratory for Video Technology Peking University Beijing China

In digital cameras, we find a major limitation: the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world. Here, we present vidar, a bit sequence array where each bit represents whether the accumulation of photons has reached a threshold, to record and reconstruct the scene radiance at any moment. By employing only consumer-level CMOS sensors and integrated circuits, we have developed a vidar camera that is 1,000× faster than conventional cameras. By treating vidar as spike trains in biological vision, we have further developed a spiking neural network-based machine vision system that combines the speed of the machine and the mechanism of biological vision, achieving high-speed object detection and tracking 1,000× faster than human vision. We demonstrate the utility of the vidar camera and the super vision system in an assistant referee and target pointing system. Our study is expected to fundamentally revolutionize the image and video concepts and related industries, including photography, movies, and visual media, and to unseal a new spiking neural network-enabled speed-free machine vision era. Copyright © 2022, The Authors. All rights reserved.

关键词： Computer vision

来源：评论

学校读者我要写书评

暂无评论

Optimal video communication strategy for intelligent video analysis in unmanned aerial vehicle applications

引用

Chinese Journal of Aeronautics 2020年第11期33卷 2921-2929页

作者： Yongqiang XIE Zhongbo LI Jin QI Kai ZHANG Bixuan ZHANG Feng QI Institute of Systems Engineering Academy of Military SciencesPLABeijing 100141China National Engineering Laboratory for Video Technology Peking UniversityBeijing 100871China

For Unmanned Aerial Vehicles(UAV),the intelligent video analysis is a key technology in intelligent autonomous control,real-time navigation and ***,poor UAV wireless links would degrade the quality of video communication,leading to difficulties in video *** meet the challenges of packet-loss and limited bandwidth in adverse UAV channel environments,this paper proposes a parameter optimization mechanism for UAV intelligent video *** the proposed method,an Optimal Strategy Library(OSL)is designed to optimize the parameters for video encoding and forward error *** to the packet-loss rate and bandwidth in practical UAV wireless network,the proposed OSL can facilitate the encoding of video sequences and the recovery of degraded videos with optimal *** results demonstrate that the proposed solution can keep intelligent video analysis working efficiently with adverse UAV wireless links,and is capable of maximizing the inference accuracy of Multi-Object Tracking(MOT)algorithms in various scenarios.

关键词： Forward error correction Intelligent video analysis Multi-Object Tracking(MOT) Optimal strategies UAVs video communication

来源：评论

学校读者我要写书评

暂无评论

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

arXiv

引用

arXiv 2023年

作者： Wang, Xiao Wu, Zongzhen Rong, Yao Zhu, Lin Jiang, Bo Tang, Jin Tian, Yonghong School of Computer Science and Technology Anhui University Institute of Artificial Intelligence Hefei Comprehensive National Science Center Hefei230601 China Beijing Institute Of Technology Beijing China Peng Cheng Laboratory Shenzhen China National Engineering Laboratory for Video Technology School of Electronics Engineering and Computer Science Peking University Beijing China

—Event camera-based pattern recognition is a newly arising research topic in recent years. Current researchers usually transform the event streams into images, graphs, or voxels, and adopt deep neural networks for event-based classification. Although good performance can be achieved on simple event recognition datasets, however, their results may be still limited due to the following two issues. Firstly, they adopt spatial sparse event streams for recognition only, which may fail to capture the color and detailed texture information well. Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition. However, seldom of them consider achieving a balance between these two aspects. In this paper, we formally propose to recognize patterns by fusing RGB frames and event streams simultaneously and propose a new RGB frame-event recognition framework to address the aforementioned issues. The proposed method contains four main modules, i.e., memory support Transformer network for RGB frame encoding, spiking neural network for raw event stream encoding, multi-modal bottleneck fusion module for RGB-Event feature aggregation, and prediction head. Due to the scarce of RGB-Event based classification dataset, we also propose a large-scale PokerEvent dataset which contains 114 classes, and 27102 frame-event pairs recorded using a DVS346 event camera. Extensive experiments on two RGB-Event based classification datasets fully validated the effectiveness of our proposed framework. We hope this work will boost the development of pattern recognition by fusing RGB frames and event streams. Both our dataset and source code of this work will be released at https://***/Event-AHU/SSTFormer. Copyright © 2023, The Authors. All rights reserved.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

Point-Voxel Absorbing Graph Representation Learning for Event Stream based Recognition

arXiv

引用

arXiv 2023年

作者： Jiang, Bo Yuan, Chengguo Wang, Xiao Bao, Zhimin Zhu, Lin Tian, Yonghong Tang, Jin School of Computer Science and Technology Anhui University Hefei230601 China Tencent YouTu Lab Hefei China Beijing Institute of Technology Beijing China Peng Cheng Laboratory Shenzhen China National Engineering Laboratory for Video Technology School of Electronics Engineering and Computer Science Peking University Beijing China

Sampled point and voxel methods are usually employed to downsample the dense events into sparse ones. After that, one popular way is to leverage a graph model which treats the sparse points/voxels as nodes and adopts graph neural networks (GNNs) to learn the representation of event data. Although good performance can be obtained, however, their results are still limited mainly due to two issues. (1) Existing event GNNs generally adopt the additional max (or mean) pooling layer to summarize all node embeddings into a single graph-level representation for the whole event data representation. However, this approach fails to capture the importance of graph nodes and also fails to be fully aware of the node representations. (2) Existing methods generally employ either a sparse point or voxel graph representation model which thus lacks consideration of the complementary between these two types of representation models. To address these issues, in this paper, we propose a novel dual point-voxel absorbing graph representation learning for event stream data representation. To be specific, given the input event stream, we first transform it into the sparse event cloud and voxel grids and build dual absorbing graph models for them respectively. Then, we design a novel absorbing graph convolutional network (AGCN) for our dual absorbing graph representation and learning. The key aspect of the proposed AGCN is its ability to effectively capture the importance of nodes and thus be fully aware of node representations in summarizing all node representations through the introduced absorbing nodes. Finally, the event representations of dual learning branches are concatenated together to extract the complementary information of two cues. The output is then fed into a linear layer for event data classification. Extensive experiments on multiple event-based classification benchmark datasets fully validated the effectiveness of our framework. Both the source code and pre-trained models wi

关键词： Classification (of information)

来源：评论

学校读者我要写书评

暂无评论

VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognition

arXiv

引用

arXiv 2024年

作者： Chen, Lan Yang, Haoxiang Shao, Pengpeng Song, Haoyu Wang, Xiao Zhao, Zhicheng Wang, Yaowei Tian, Yonghong The School of Electronic and Information Engineering Anhui University Hefei230601 China The School of Computer Science and Technology Anhui University Hefei China Tsinghua University Beijing China The School of Artificial Intelligence Anhui University Hefei China Peng Cheng Laboratory Shenzhen China Harbin Institute of Technology Shenzhen China National Engineering Laboratory for Video Technology School of Electronics Engineering and Computer Science Peking University Beijing China

Pattern recognition leveraging both RGB and Event cameras can significantly enhance performance by deploying deep neural networks that utilize a fine-tuning strategy. Inspired by the successful application of large models, the introduction of such large models can also be considered to further enhance the performance of multi-modal tasks. However, fully fine-tuning these models leads to inefficiency and lightweight fine-tuning methods such as LoRA and Adapter have been proposed to achieve a better balance between efficiency and performance. To our knowledge, there is currently no work that has conducted parameter-efficient fine-tuning (PEFT) for RGB-Event recognition based on pre-trained foundation models. To address this issue, this paper proposes a novel PEFT strategy to adapt the pre-trained foundation vision models for the RGB-Event-based classification. Specifically, given the RGB frames and event streams, we extract the RGB and event features based on the vision foundation model ViT with a modality-specific LoRA tuning strategy. The frame difference of the dual modalities is also considered to capture the motion cues via the frame difference backbone network. These features are concatenated and fed into high-level Transformer layers for efficient multi-modal feature learning via modality-shared LoRA tuning. Finally, we concatenate these features and feed them into a classification head to achieve efficient fine-tuning. The source code and pre-trained models will be released on https://***/Event-AHU/VELoRA. Copyright © 2024, The Authors. All rights reserved.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

BBA-NET: A BI-BRANCH ATTENTION NETWORK FOR CROWD COUNTING

arXiv

引用

arXiv 2022年

作者： Houyi, Yi Hou Li, Chengyang Yang, Fan Ma, Cong Zhu, Liping Li, Yuan Jia, Huizhu Xie, Xiaodong National Engineering Laboratory for Video Technology Peking University China Key Lab of Petroleum Data Mining China University of Petroleum Beijing China

In the field of crowd counting, the current mainstream CNN-based regression methods simply extract the density information of pedestrians without finding the position of each person. This makes the output of the network often found to contain incorrect responses, which may erroneously estimate the total number and not conducive to the interpretation of the algorithm. To this end, we propose a Bi-Branch Attention Network (BBA-NET) for crowd counting, which has three innovation points. i) A two-branch architecture is used to estimate the density information and location information separately. ii) Attention mechanism is used to facilitate feature extraction, which can reduce false responses. iii) A new density map generation method combining geometric adaptation and Voronoi split is introduced. Our method can integrate the pedestrian's head and body information to enhance the feature expression ability of the density map. Extensive experiments performed on two public datasets show that our method achieves a lower crowd counting error compared to other state-of-the-art methods. © 2022, CC BY.

关键词： Convolutional neural networks

来源：评论

学校读者我要写书评

暂无评论

Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation

arXiv

引用

arXiv 2025年

作者： Wang, Xiao Wang, Fuling Wang, Haowen Jiang, Bo Li, Chuanfu Wang, Yaowei Tian, Yonghong Tang, Jin Information Materials and Intelligent Sensing Laboratory of Anhui Province Anhui University Hefei230601 China Anhui Provincial Key Laboratory of Multimodal Cognitive Computation Anhui University Hefei230601 China School of Computer Science and Technology Anhui University Hefei230601 China The First Affiliated Hospital of Anhui University of Chinese Medicine Hefei230022 China Peng Cheng Laboratory Shenzhen China Harbin Institute of Technology Shenzhen China National Engineering Laboratory for Video Technology School of Electronics Engineering and Computer Science Peking University Beijing China

X-ray image based medical report generation achieves significant progress in recent years with the help of the large language model, however, these models have not fully exploited the effective information in visual image regions, resulting in reports that are linguistically sound but insufficient in describing key diseases. In this paper, we propose a novel associative memory-enhanced X-ray report generation model that effectively mimics the process of professional doctors writing medical reports. It considers both the mining of global and local visual information and associates historical report information to better complete the writing of the current report. Specifically, given an X-ray image, we first utilize a classification model along with its activation maps to accomplish the mining of visual regions highly associated with diseases and the learning of disease query tokens. Then, we employ a visual Hopfield network to establish memory associations for disease-related tokens, and a report Hopfield network to retrieve report memory information. This process facilitates the generation of high-quality reports based on a large language model and achieves state-of-the-art performance on multiple benchmark datasets, including the IU X-ray, MIMIC-CXR, and Chexpert Plus. The source code of this work is released on https://***/Event-AHU/Medical_Image_Analysis. Copyright © 2025, The Authors. All rights reserved.

关键词： Associative storage

来源：评论

学校读者我要写书评

暂无评论

New treatment for amblyopia based on rules of synaptic plasticity:a randomized clinical trial

引用

Science China(Life Sciences) 2022年第3期65卷 451-465页

作者： Xin Huang Huika Xia Qi Zhang Colin Blakemore Yan Nan Wenyao Wang Jie Gao Spencer S Ng Jing Wen Tiejun Huang Xiaoqing Li Mingliang Pu Department of Anatomy School of Basic Medical SciencesPeking UniversityBeijing 100083China Key Laboratory on Machine Perception(Ministry of Education) Peking UniversityBeijing 100083China Department of Ophthalmology Hebei General HospitalShijiazhuang 050051China Department of Biomedical Sciences City University of Hong KongHong KongChina Department of Biology University of CaliforniaLos Angeles 90095-7246USA Department of Computer Science School of Electrical Engineering and Computer SciencesPeking UniversityBeijing 100191China National Engineering Laboratory for Video Technology Peking UniversityBeijing 100871China Department of Pediatric Ophthalmology Peking University First HospitalBeijing 100034China National Amblyopia and Strabismus Prevention and Treatment Center Beijing 100034China

Amblyopia resulting from early deprivation of vision or defocus in one eye reflects an imbalance of input from the eyes to the visual *** tested the hypothesis that asynchronous stimulation of the two eyes might induce synaptic plasticity and rebalance *** on normal adults showed that repetitive brief exposure of grating stimuli,with the onset of each stimulus delayed by 8.3 ms in one eye,results in a shift in perceptual eye *** studies(Clinical trial registration number:Chi CTR2100049130),using popular 3D movies with similar asynchrony between the two eyes(amblyopic eye stimulated first)to treat anisometropic amblyopia,established that just 10.5 h of conditioning over<3 weeks produced improvement that met criteria for successful *** benefits of asynchronous conditioning accumulate over 20–3045 min sessions,and are maintained for at least 2 ***,we demonstrate that asynchronous binocular treatment alone is more effective than patching *** novel treatment is popular with children and is some 50 times more efficient than patching alone.

关键词： amblyopia visual cortex STDP synaptic plasticity SED

来源：评论

学校读者我要写书评

暂无评论

Bridging the Source-to-target Gap for Cross-domain Person Re-Identification with Intermediate Domains

arXiv

引用

arXiv 2022年

作者： Dai, Yongxing Sun, Yifan Liu, Jun Tong, Zekun Yang, Yi Duan, Ling-Yu The National Engineering Laboratory for Video Technology Peking University Beijing China Baidu Research Beijing China The Information Systems Technology and Design Pillar Singapore University of Technology and Design Singapore National University of Singapore Singapore The College of Computer Science and Technology Zhejiang University Hangzhou China Peng Cheng Laboratory Shenzhen China

Cross-domain person re-identification (re-ID), such as unsupervised domain adaptive re-ID (UDA re-ID), aims to transfer the identity-discriminative knowledge from the source to the target domain. Existing methods commonly consider the source and target domains are isolated from each other, i.e., no intermediate status is modeled between the source and target domains. Directly transferring the knowledge between two isolated domains can be very difficult, especially when the domain gap is large. This paper, from a novel perspective, assumes these two domains are not completely isolated, but can be connected through a series of intermediate domains. Instead of directly aligning the source and target domains against each other, we propose to align the source and target domains against their intermediate domains so as to facilitate a smooth knowledge transfer. To discover and utilize these intermediate domains, this paper proposes an Intermediate Domain Module (IDM) and a Mirrors Generation Module (MGM). IDM has two functions: 1) it generates multiple intermediate domains by mixing the hidden-layer features from source and target domains and 2) it dynamically reduces the domain gap between the source / target domain features and the intermediate domain features. While IDM achieves good domain alignment effect, it introduces a side effect, i.e., the mix-up operation may mix the identities into a new identity and lose the original identities. Accordingly, MGM is introduced to compensate the loss of the original identity by mapping the features into the IDM-generated intermediate domains without changing their original identity. It allows to focus on minimizing domain variations to further promote the alignment between the source / target domain and intermediate domains, which reinforces IDM into IDM++. We extensively evaluate our method under both the UDA and domain generalization (DG) scenarios and observe that IDM++ yields consistent (and usually significant) performance

关键词： Knowledge management

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：