检索结果-内蒙古大学图书馆

SpikingMiniLM: energy-efficient spiking transformer for natural language understanding

Science China(Information Sciences) 2024年第10期67卷 115-128页

作者： Jiayu ZHANG Jiangrong SHEN Zeke WANG Qinghai GUO Rui YAN Gang PAN Huajin TANG College of Computer Science and Technology Zhejiang University The State Key Lab of Brain-Machine Intelligence Zhejiang University Collaborative Innovation Center of Artificial Intelligence Zhejiang University Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd. College of Computer Science and Technology Zhejiang University of Technology MOE Frontier Science Center for Brain Science and Brain-Machine Integration Zhejiang University

In the era of large-scale pretrained models, artificial neural networks（ANNs） have excelled in natural language understanding（NLU） tasks. However, their success often necessitates substantial computational resources and energy consumption. To address this, we explore the potential of spiking neural networks（SNNs） in NLU——a promising avenue with demonstrated advantages, including reduced power consumption and improved efficiency due to their event-driven characteristics. We propose the SpikingMiniLM,a novel spiking Transformer model tailored for natural language understanding. We first introduce a multi-step encoding method to convert text embeddings into spike trains. Subsequently, we redesign the attention mechanism and residual connections to make our model operate on the pure spike-based paradigm without any normalization technique. To facilitate stable and fast convergence, we propose a general parameter initialization method grounded in the stable firing rate principle. Furthermore, we apply an ANN-to-SNN knowledge distillation to overcome the challenges of pretraining SNNs. Our approach achieves a macro-average score of 75.5 on the dev sets of the GLUE benchmark, retaining 98% of the performance exhibited by the teacher model MiniLMv2. Our smaller model also achieves similar performance to BERTMINIwith fewer parameters and much lower energy consumption, underscoring its competitiveness and resource efficiency in NLU tasks.

关键词： spiking neural networks natural language understanding spiking Transformer spike-based attention multi-step encoding ANN-to-SNN distillation

来源：评论

学校读者我要写书评

暂无评论

Continual learning of new diseases with dual distillation and ensemble strategy 23rd

Continual learning of new diseases with dual distillation an...

引用

23rd International Conference on Medical Image computing and Computer-Assisted Intervention, MICCAI 2020

作者： Li, Zhuoyun Zhong, Changhong Wang, Ruixuan Zheng, Wei-Shi School of Data and Computer Science Sun Yat-sen University Guangzhou China Key Laboratory of Machine Intelligence and Advanced Computing MOE Guangzhou China Pazhou Lab Guangzhou China

ISBN: (纸本)9783030597092

Most intelligent diagnosis systems are developed for one or a few specific diseases, while medical specialists can diagnose all diseases of certain organ or tissue. Since it is often difficult to collect data of all diseases, it would be desirable if an intelligent system can initially diagnose a few diseases, and then continually learn to diagnose more and more diseases with coming data of these new classes in the future. However, current intelligent systems are characterised by catastrophic forgetting of old knowledge when learning new classes. In this paper, we propose a new continual learning framework to alleviate this issue by simultaneously distilling both old knowledge and recently learned new knowledge and by ensembling the class-specific knowledge from the previous classifier and the learned new classifier. Experiments showed that the proposed method outperforms state-of-the-art methods on multiple medical and natural image datasets. © Springer Nature Switzerland AG 2020.

关键词： Distillation

来源：评论

学校读者我要写书评

暂无评论

Detection of driving fatigue based on grip force on steering wheel with wavelet transformation and support vector machine

Detection of driving fatigue based on grip force on steering...

引用

20th International Conference on Neural Information Processing, ICONIP 2013

作者： Li, Fan Wang, Xiao-Wei Lu, Bao-Liang Center for Brain-like Computing and Machine Intelligence Department of Computer Science and Engineering China MOE-Microsoft Key Lab. for Intelligent Computing and Intelligent Systems Shanghai Jiao Tong University 800 Dongchuan Road Shanghai 200240 China

ISBN: (纸本)9783642420504

This paper proposes an unobtrusive way to detect fatigue for drivers through grip forces on steering wheel. Simulated driving experiments are conducted in a refitted passenger car, during which grip forces of both hands are collected. Wavelet transformation is introduced to extract fatigue-related features from wavelet coefficients. We compare the performance of k-nearest neighbours, linear discriminant analysis, and support vector machine (SVM) on the task of discriminating drowsy and awake states. SVM with radial basis function reaches the best accuracy, 75% on average. The results show that variation in grip forces on steering wheel can be used to effectively detect drivers' fatigue. © Springer-Verlag 2013.

关键词： Support vector machines

来源：评论

学校读者我要写书评

暂无评论

Embedding Dynamic Attributed Networks by Modeling the Evolution Processes 28

Embedding Dynamic Attributed Networks by Modeling the Evolut...

引用

28th International Conference on Computational Linguistics, COLING 2020

作者： Xu, Zenan Ou, Zijing Su, Qinliang Yu, Jianxing Quan, Xiaojun Lin, Zhenkun School of Data and Computer Science Sun Yat-Sen University China Guangdong Key Laboratory of Big Data Analysis and Processing Guangzhou China Key Lab. of Machine Intelligence and Advanced Computing Ministry of Education China

ISBN: (纸本)9781952148279

Network embedding has recently emerged as a promising technique to embed nodes of a network into low-dimensional vectors. While fairly successful, most existing works focus on the embedding techniques for static networks. But in practice, there are many networks that are evolving over time and hence are dynamic, e.g., the social networks. To address this issue, a high-order spatio-temporal embedding model is developed to track the evolutions of dynamic networks. Specifically, an activeness-aware neighborhood embedding method is first proposed to extract the high-order neighborhood information at each given timestamp. Then, an embedding prediction framework is further developed to capture the temporal correlations, in which the attention mechanism is employed instead of recurrent neural networks (RNNs) for its efficiency in computing and flexibility in modeling. Extensive experiments are conducted on four real-world datasets from three different areas. It is shown that the proposed method outperforms all the baselines by a substantial margin for the tasks of dynamic link prediction and node classification, which demonstrates the effectiveness of the proposed methods on tracking the evolutions of dynamic networks. © 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

关键词： Recurrent neural networks

来源：评论

学校读者我要写书评

暂无评论

Real-time head detection with kinect for driving fatigue detection

Real-time head detection with kinect for driving fatigue det...

引用

20th International Conference on Neural Information Processing, ICONIP 2013

作者： Cao, Yang Lu, Bao-Liang Center for Brain-like Computing and Machine Intelligence Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240 China MOE-Microsoft Key Lab. for Intelligent Computing and Intelligent Systems Shanghai Jiao Tong University 800 Dongchuan Road Shanghai 200240 China

ISBN: (纸本)9783642420504

Nowadays, depth cameras such Microsoft Kinect make it easier and cheaper for us to capture depth images. It becomes practical to use depth images for detection in consumer-grade products. In this paper, we propose a novel and simple real-time method to detect human head in depth image for our driving fatigue detection system, based on the elliptical shape of human head. Experiments show that our method can successfully detect human head in different light conditions and across different head poses. We integrate this detection algorithm into our driving fatigue detection system, and see remarkable improvements both in detection rate and detection speed. © Springer-Verlag 2013.

关键词： Computers

来源：评论

学校读者我要写书评

暂无评论

Topology analysis system for vehicular Ad Hoc network 17

Topology analysis system for vehicular Ad Hoc network

引用

17th International Conference on Parallel and Distributed computing, Applications and Technologies, PDCAT 2016

作者： Dong, Baihong Deng, Jian Wu, Weigang Meng, Tianyu 1. School of Data and Computer Science Sun Yat-Sen Univ. China Guangdong Province Key Lab. of Big Data Analysis and Processing China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education Guangzhou China

ISBN: (纸本)9781509050819

With the development of the technology, Vehicular Ad-hoc Network is developing rapidly and continuously. And many new algorithms in VANET were put forward. For example, many people apply the Named Data Networks or Software Defined Networking6 to the VANET. However, few researches are looking into the analysis of the topology of the VANET, among most of which are based on simple situations and calculation of the track of vehicles. And therefore, these analysis are inadequate for analyzing the topology under complex situations. This paper is going to put forward a new method for analyzing the change of the topology of VANET, which is capable to quantitatively analyze the change under different and complex situations. In the real experiment, when compared with traditional analysis methods, this new method can analyze more precisely the impact of some possible wireless communication problem on the change of the topology, for example, the impact of hidden node on the vehicular wireless communication and whether the number or the density of vehicles makes the topology of VANET more stable. And this new method can provide a numerical result instead of purely qualitative analysis. © 2016 IEEE.

关键词： Topology

来源：评论

学校读者我要写书评

暂无评论

A dimension-decreasing particle swarm optimization method for portfolio optimization 15

A dimension-decreasing particle swarm optimization method fo...

引用

17th Genetic and Evolutionary Computation Conference, GECCO 2015

作者： Wang, Jia-Bin Chen, Wei-Neng Zhang, Jun Lin, Ying School of Advanced Computing Sun Yat-sen University Guangzhou China Key Lab. Machine Intelligence and Advanced Computing Ministry of Education China Collaborative Innovation Center of High Performance Computing China Engineering Research Center of Supercomputing Engineering Software Ministry of Education China Department of Psychology Sun Yat-sen University Guangzhou China

ISBN: (纸本)9781450334884

Portfolio optimization problems are challenging as they contain different kinds of constrains and their complexity becomes very high when the number of assets grows. In this paper, we develop a dimension-decreasing particle swarm optimization (DDPSO) for solving multi-constrained portfolio optimization problems. DDPSO improves the efficiency of PSO for solving portfolio optimization problems with a lot of asset and it can easily handle the cardinality constraint in portfolio optimization. To improve search diversity, the dimension-decreasing method is coupled with the comprehensive learning particle swarm optimization (CLPSO) algorithm. The proposed method is tested on benchmark problems from the OR library. Experimental results show that the proposed algorithm performs well. Copyright is held by the owner/author(s).

关键词： Particle swarm optimization (PSO)

来源：评论

学校读者我要写书评

暂无评论

Deep knn for medical image classification 23rd

Deep knn for medical image classification

引用

23rd International Conference on Medical Image computing and Computer-Assisted Intervention, MICCAI 2020

作者： Zhuang, Jiaxin Cai, Jiabin Wang, Ruixuan Zhang, Jianguo Zheng, Wei-Shi School of Data and Computer Science Sun Yat-sen University Guangzhou China Department of Computer Science and Engineering Southern University of Science and Technology Shenzhen China Key Laboratory of Machine Intelligence and Advanced Computing MOE Guangzhou China Pazhou Lab Guangzhou China

ISBN: (纸本)9783030597092

Human-level diagnostic performance from intelligent systems often depends on large set of training data. However, the amount of availab.e data for model training may be limited for part of diseases, which would cause the widely adopted deep learning models not generalizing well. One alternative simple approach to small class prediction is the traditional k-nearest neighbor (kNN). However, due to the non-parametric characteristics of kNN, it is difficult to combine the kNN classification into the learning of feature extractor. This paper proposes an end-to-end learning strategy to unify the kNN classification and the feature extraction procedure. The basic idea is to enforce that each training sample and its K nearest neighbors belong to the same class during learning the feature extractor. Experiments on multiple small-class and class-imbalanced medical image datasets showed that the proposed deep kNN outperforms both kNN and other strong classifiers. © Springer Nature Switzerland AG 2020.

关键词： Nearest neighbor search

来源：评论

学校读者我要写书评

暂无评论

FSTR: Funds Skewness Aware Transaction Routing for Payment Channel Networks

FSTR: Funds Skewness Aware Transaction Routing for Payment C...

引用

International Conference on Dependable Systems and Networks (DSN)

作者： Siyi Lin Jingjing Zhang Weigang Wu School of Data and Computer Science Sun Yat-sen University Guangzhou China Guangdong Key Lab. of Big Data Analysis and Processing China MoE Key Lab. of Machine Intelligence and Advanced Computing China

ISBN: (数字)9781728158099

ISBN: (纸本)9781728158105

Payment channel is an effective and popular technique to improve the scalab.lity and throughput of blockchains by transferring transactions from on-chain to off-chain. Multiple payment channels can constitute a payment network and realize transaction execution via multi-hop paths. How to find a feasible and efficient transaction path, i.e., transaction routing, is a key issue in payment channel networks, and different solutions have been proposed. However, the problem of funds skewness, which may cause routing failures, has been largely ignored in existing routing algorithms. In this work, we design FSTR, a routing algorithm that attempts to route transactions using a funds skewness based path selection scheme so as to reduce funds skewness and increase transaction success probability. To evaluate the performance of FSTR, we conduct experiments using the real-world dataset of Ripple. The experiment results show that FSTR outperforms existing routing algorithms, in terms of success ratio, delay, and overhead.

关键词： Routing Delays Throughput Scalab.lity Spread spectrum communication Mice

来源：评论

学校读者我要写书评

暂无评论

Abnormality Detection in Chest X-Ray Images Using Uncertainty Prediction Autoencoders 23rd

Abnormality Detection in Chest X-Ray Images Using Uncertaint...

引用

23rd International Conference on Medical Image computing and Computer-Assisted Intervention, MICCAI 2020

作者： Mao, Yifan Xue, Fei-Fei Wang, Ruixuan Zhang, Jianguo Zheng, Wei-Shi Liu, Hongmei School of Data and Computer Science Sun Yat-sen University Guangzhou China Key Laboratory of Machine Intelligence and Advanced Computing MOE Guangzhou China Department of Computer Science and Engineering Southern University of Science and Technology Shenzhen China Pazhou Lab Guangzhou China Guangdong Province Key Laboratory of Information Security Technology Guangzhou China

ISBN: (纸本)9783030597245

Chest radiography is widely used in annual medical screening to check whether lungs are healthy or not. Therefore it would be desirable to develop an intelligent system to help clinicians automatically detect potential abnormalities in chest X-ray images. Here with only healthy X-ray images, we propose a new abnormality detection approach based on an autoencoder which outputs not only the reconstructed normal version of the input image but also a pixel-wise uncertainty prediction. Higher uncertainty often appears at normal region boundaries with relatively larger reconstruction errors, but not at potential abnormal regions in the lung area. Therefore the normalized reconstruction error by the uncertainty provides a natural measurement for abnormality detection in images. Experiments on two chest X-ray datasets show the state-of-the-art performance by the proposed approach. © 2020, Springer Nature Switzerland AG.

关键词： Forecasting

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：