检索结果-内蒙古大学图书馆

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Qian, Xinyuan Gao, Jiaran Zhang, Yaodan Zhang, Qiquan Liu, Hexin Garcia, Leibny Paola Li, Haizhou The School of Computer and Communication Engineering University of Science and Technology Beijing Beijing100083 China The School of Electrical Engineering and Telecommunications The University of New South Wales Sydney2052 Australia The College of Computing and Data Science Nanyang Technological University Singapore The Center for Language and Speech Processing Johns Hopkins University United States The Guangdong Provincial Key Laboratory of Big Data Computing The Chinese University of Hong Kong Shenzhen518172 China Shenzhen Research Institute of Big data Shenzhen51872 China

Speech enhancement plays an essential role in various applications, and the integration of visual information has been demonstrated to bring substantial advantages. However, the majority of current research concentrates on the examination of facial and lip movements, which can be compromised or entirely inaccessible in scenarios where occlusions occur or when the camera view is distant. Whereas contextual visual cues from the surrounding environment have been overlooked: for example, when we see a dog bark, our brain has the innate ability to discern and filter out the barking noise. To this end, in this paper, we introduce a novel task, i.e. Scene-aware AudioVisual Speech Enhancement (SAV-SE). To our best knowledge, this is the first proposal to use rich contextual information from synchronized video as auxiliary cues to indicate the type of noise, which eventually improves the speech enhancement performance. Specifically, we propose the VC-S2E method, which incorporates the Conformer and Mamba modules for their complementary strengths. Extensive experiments are conducted on public MUSIC, AVSpeech and AudioSet datasets, where the results demonstrate the superiority of VC-S2E over other competitive methods. We will make the source code publicly available. Project demo page: https://***/ © 2024, CC BY.

关键词： Speech enhancement

Sub-Loc:predicting protein sub-mitochondrial localization based on sequence embedding

学校读者我要写书评

暂无评论

Sub-Loc:predicting protein sub-mitochondrial localization ba...

IEEE International Conference on Bioinformatics and Biomedicine (BIBM)

作者： Xiaoting Wang Juan Wang Haodong Bian Maozu Guo School of Computer Science Inner Mongolia University and Ecological Big Data Engineering Research Center of the Ministry of Education Hohhot China School of Electrical and Information Engineering Beijing University of Civil Engineering and Architecture Beijing P.R. China Beijing Key Laboratory of Intelligent Processing for Building Big Data Beijing P.R. China

ISBN: (数字)9781665468190

ISBN: (纸本)9781665468206

Mitochondria are subcellular organelles existing in most eukaryotic organisms. They have a pivotal role in lots of bio-chemical processes for cells. Proteins in different compartments of mitochondria have their transport routes. Locating proteins in mitochondria can provide a solid foundation for mitochondrial pathologies. So far, there have been several computational methods for solving the issue. However, their accuracy is so low that they do not identify the localization precisely. We develop an unsupervised learning model to represent mitochondrial proteins as n-dimensional vectors, called sequence embedding, which can learn the global and context information from mitochondrial proteins. We design a new model, called Sub-Loc, to predict the sub-mitochondrial localization of proteins using the SVM classifier and the sequence embedding method. The sequence embedding method and the Sub-Loc are tested by experiments. Experimental results show the sequence embedding method remarkably enhances the performance of prediction for sub-mitochondrial localization of proteins compared with other feature representations. The Sub-Loc outperforms other approaches for predicting sub-mitochondrial localization of proteins.

关键词： Proteins Location awareness Support vector machines Deep learning Pathology Predictive models Feature extraction

Fundamental Limits of Multiple-Access Integrated Sensing and Communication Systems

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Liu, Yao Li, Min Liu, An Ong, Lawrence Yener, Aylin The College of Information Science and Electronic Engineering The Zhejiang Provincial Key Laboratory of Information Processing Communication and Networking Zhejiang University Hangzhou310027 China The School of Engineering University of Newcastle CallaghanNSW2308 Australia The Department of Electrical and Computer Engineering The Ohio State University ColumbusOH43210 United States

A memoryless state-dependent multiple access channel is considered to model an integrated sensing and communication system, where two transmitters wish to convey messages to a receiver while simultaneously estimating the sensing state sequences through echo signals. In particular, the sensing states are assumed to be correlated with the channel state, and the receiver has imperfect channel state information. In this setup, improved inner and outer bounds for capacity-distortion region are derived. The inner bound is based on an achievable scheme that combines message cooperation and joint compression via distributed Wyner-Ziv coding at each transmitter, resulting in unified cooperative communication and sensing. The outer bound is based on the ideas of dependence balance for communication rate and rate-limited constraints on sensing distortion. The proposed inner and outer bounds are proved to improve the state-of-the-art bounds. Finally, numerical examples are provided to demonstrate that our new inner and outer bounds strictly improve the existing results. © 2022, CC BY.

关键词： Transmitters

MG-SLAM: Structure Gaussian Splatting SLAM with Manhattan World Hypothesis

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Liu, Shuhong Deng, Tianchen Zhou, Heng Li, Liuzhuozheng Wang, Hongyu Wang, Danwei Li, Mingrui Department of Information Science and Technology Department of Complexity Science and Engineering The University of Tokyo Tokyo113-8654 Japan Institute of Medical Robotics Department of Automation Shanghai Jiao Tong University Key Laboratory of System Control and Information Processing Ministry of Education Shanghai200240 China Department of Mechanical Engineering Columbia University New York10027 United States School of Electrical and Electronic Engineering Nanyang Technological University Singapore Department of Computer Science Dalian University of Technology Dalian116024 China

Gaussian Splatting SLAMs have made significant advancements in improving the efficiency and fidelity of real-time reconstructions. However, these systems often encounter incomplete reconstructions in complex indoor environments, characterized by substantial holes due to unobserved geometry caused by obstacles or limited view angles. To address this challenge, we present Manhattan Gaussian SLAM, an RGB-D system that leverages the Manhattan World hypothesis to enhance geometric accuracy and completeness. By seamlessly integrating fused line segments derived from structured scenes, our method ensures robust tracking in textureless indoor areas. Moreover, The extracted lines and planar surface assumption allow strategic interpolation of new Gaussians in regions of missing geometry, enabling efficient scene completion. Extensive experiments conducted on both synthetic and real-world scenes demonstrate that these advancements enable our method to achieve state-of-the-art performance, marking a substantial improvement in the capabilities of Gaussian SLAM systems. © 2024, CC BY.

关键词： Gaussian distribution

Signal Detection in MIMO Systems with Hardware Imperfections: Message Passing on Neural Networks

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Gao, Dawei Guo, Qinghua Liao, Guisheng Eldar, Yonina C. Li, Yonghui Yu, Yanguang Vucetic, Branka The Hangzhou Institute of Technology Xidian University Hangzhou311200 China The National Laboratory of Radar Signal Processing Xidian University Xi’an710071 China The School of Electrical Computer and Telecommunications Engineering University of Wollongong NSW2522 Australia The Faculty of Math and CS Weizmann Institute of Science Rehovot7610001 Israel The School of Electrical and Information Engineering University of Sydney SydneyNSW2006 Australia

In this paper, we investigate signal detection in multiple-input-multiple-output (MIMO) communication systems with hardware impairments, such as power amplifier nonlinearity and in-phase/quadrature imbalance. To deal with the complex combined effects of hardware imperfections, neural network (NN) techniques, in particular deep neural networks (DNNs), have been studied to directly compensate for the impact of hardware impairments. However, it is difficult to train a DNN with limited pilot signals, hindering its practical applications. In this work, we investigate how to achieve efficient Bayesian signal detection in MIMO systems with hardware imperfections. Characterizing combined hardware imperfections often leads to complicated signal models, making Bayesian signal detection challenging. To address this issue, we first train an NN to ‘model’ the MIMO system with hardware imperfections and then perform Bayesian inference based on the trained NN. Modelling the MIMO system with NN enables the design of NN architectures based on the signal flow of the MIMO system, minimizing the number of NN layers and parameters, which is crucial to achieving efficient training with limited pilot signals. We then represent the trained NN with a factor graph, and design an efficient message passing based Bayesian signal detector, leveraging the unitary approximate message passing (UAMP) algorithm. The implementation of a turbo receiver with the proposed Bayesian detector is also investigated. Extensive simulation results demonstrate that the proposed technique delivers remarkably better performance than state-of-the-art methods. Copyright © 2022, The Authors. All rights reserved.

关键词： Deep neural networks

Efficient Semantic Segmentation via Lightweight Multiple-Information Interaction Network

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Qiu, Yangyang Xu, Guoan Gao, Guangwei Guo, Zhenhua Yu, Yi Lin, Chia-Wen The Institute of Advanced Technology Nanjing University of Posts and Telecommunications Nanjing210046 China Key Laboratory of Artificial Intelligence Ministry of Education Shanghai200240 China The Provincial Key Laboratory for Computer Information Processing Technology Soochow University Suzhou215006 China The Faculty of Engineering and Information Technology University of Technology Sydney Sydney Australia The Tianyijiaotong Technology Ltd. Suzhou215100 China The Graduate School of Advanced Science and Engineering Hiroshima University Hiroshima739-8511 Japan The Department of Electrical Engineering National Tsing Hua University Hsinchu30013 Taiwan

Recently, the integration of the local modeling capabilities of Convolutional Neural Networks (CNNs) with the global dependency strengths of Transformers has created a sensation in the semantic segmentation community. However, substantial computational workloads and high hardware memory demands remain major obstacles to their further application in real-time scenarios. In this work, we propose a lightweight multiple-information interaction network for real-time semantic segmentation, called LMIINet, which effectively combines CNNs and Transformers while reducing redundant computations and memory footprint. It features Lightweight Feature Interaction Bottleneck (LFIB) modules comprising efficient convolutions that enhance context integration. Additionally, improvements are made to the Flatten Transformer by enhancing local and global feature interaction to capture detailed semantic information. The incorporation of a combination coefficient learning scheme in both LFIB and Transformer blocks facilitates improved feature interaction. Extensive experiments demonstrate that LMIINet excels in balancing accuracy and efficiency. With only 0.72M parameters and 11.74G FLOPs, LMIINet achieves 72.0% mIoU at 100 FPS on the Cityscapes test set and 69.94% mIoU at 160 FPS on the CamVid test dataset using a single RTX2080Ti GPU. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantic Segmentation

MetaFruit Meets Foundation Models: Leveraging a Comprehensive Multi-Fruit Dataset for Advancing Agricultural Foundation Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Jiajia Lammers, Kyle Yin, Xunyuan Yin, Xiang He, Long Lu, Renfu Li, Zhaojian Department of Electrical and Computer Engineering Michigan State University East LansingMI United States School of Chemical and Biomedical Engineering Nanyang Technological University Singapore Department of Automation Key Laboratory of System Control and Information Processing Shanghai Jiao Tong University Shanghai China Department of Agricultural and Biological Engineering Pennsylvania State University United States United States Department of Agriculture Agricultural Research Service East LansingMI United States Department of Mechanical Engineering Michigan State University East LansingMI United States

Fruit harvesting poses a significant labor and financial burden for the industry, highlighting the critical need for advancements in robotic harvesting solutions. Machine vision-based fruit detection has been recognized as a crucial component for robust identification of fruits to guide robotic manipulation. Despite considerable progress in leveraging deep learning and machine learning techniques for fruit detection, a common shortfall is the inability to swiftly extend the developed models across different orchards and/or various fruit species. Additionally, the limited availability of pertinent data further compounds these challenges. In this work, we introduce MetaFruit, the largest publicly available multi-class fruit dataset, comprising 4,248 images and 248,015 manually labeled instances across diverse U.S. orchards. Furthermore, this study proposes an innovative open-set fruit detection system leveraging advanced Vision Foundation Models (VFMs) for fruit detection that can adeptly identify a wide array of fruit types under varying orchard conditions. This system not only demonstrates remarkable adaptability in learning from minimal data through few-shot learning but also shows the ability to interpret human instructions for subtle detection tasks. The performance of the developed foundation model is comprehensively evaluated using several metrics, which outperforms the existing state-of-the-art algorithms in both our MetaFruit dataset and other open-sourced fruit datasets, thereby setting a new benchmark in the field of agricultural technology and robotic harvesting. The MetaFruit dataset (https://***/datasets/jiajiali/metafruit) and detection framework (https://***/JiajiaLi04/FMFruit) are open-sourced to foster future research in vision-based fruit harvesting, marking a significant stride toward addressing the urgent needs of the agricultural sector. © 2024, CC BY-NC-SA.

关键词： Fruits

Metafruit Meets Foundation Models: Leveraging a Comprehensive Multi-Fruit Dataset for Advancing Agricultural Foundation Models

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

Fruit harvesting poses a significant labor and financial burden for the industry, highlighting the critical need for advancements in robotic harvesting solutions. Machine vision-based fruit detection has been recognized as a crucial component for robust identification of fruits to guide robotic manipulation. Despite considerable progress in leveraging deep learning and machine learning techniques for fruit detection, a common shortfall is the inability to swiftly extend the developed models across different orchards and/or various fruit species. Additionally, the limited availability of pertinent data further compounds these challenges. In this work, we introduce MetaFruit, the largest publicly available multi-class fruit dataset, comprising 4,248 images and 248,015 manually labeled instances across diverse U.S. orchards. Furthermore, this study proposes an innovative open-set fruit detection system leveraging advanced Vision Foundation Models (VFMs) for fruit detection that can adeptly identify a wide array of fruit types under varying orchard conditions. This system not only demonstrates remarkable adaptability in learning from minimal data through few-shot learning but also shows the ability to interpret human instructions for subtle detection tasks. The performance of the developed foundation model is comprehensively evaluated using several metrics, which outperforms the existing state-of-the-art algorithms in both our MetaFruit dataset and other open-sourced fruit datasets, thereby setting a new benchmark in the field of agricultural technology and robotic harvesting. The MetaFruit dataset and detection framework are open-sourced to foster future research in vision-based fruit harvesting, marking a significant stride toward addressing the urgent needs of the agricultural sector. © 2024, The Authors. All rights reserved.

关键词： Fruits

Graph Convolutional Networks Based Muti-Label Deep Cross-Modal Hashing

学校读者我要写书评

暂无评论

Graph Convolutional Networks Based Muti-Label Deep Cross-Mod...

International Conference on Information Science and Technology (ICIST)

作者： Yi Peng Xin Jiang Nian Zhang Jiang Xiong School of Electronic and Information Engineering Chongqing Three Gorges University Chongqing China Key Laboratory of Intelligent Information Processing and Control of Chongqing Municipal Institutions of Higher Education Chongqing Three Gorges University Chongqing China Department of Electrical and Computer Engineering University of the District of Columbia Washington D.C. USA College of Mathematics and Statistics Chongqing Three Gorges University Chongqing China

Recently, multi-label deep cross-modal hashing (MDCH), which incorporates deep neural networks, hashing and multi-label learning for cross-modal retrieval tasks, has achieved excellent cross-modal retrieval results and thus became a highly popular area of research. Nevertheless, many existing MDCH methods concentrate on extracting information from multi-modal data, while neglecting the abundant semantic information in multiple labels. Few MDCH methods incorporate multi-label information, but they often treat labels as independent entities, ignoring the relationships between categories, which hinders the establishment of semantic connections among multi-modal data. In order to tackle the aforementioned challenges, we propose a graph convolutional networks based multi-label deep cross-modal hashing method (GMCH) in this paper. GMCH leverages two deep neural networks to generate hash representations from the original image-text pairs, during this process, a graph convolutional network is introduced to capture the category correlations of multi-labels and supervise the training of hash mapping. Experimental results on two commonly employed datasets validate the efficacy of the proposed GMCH method. You can find the code for our proposed GCMH at https://***/licher12/***.

关键词：