检索结果-内蒙古大学图书馆

SpikingMiniLM: energy-efficient spiking transformer for natural language understanding

Science China(Information Sciences) 2024年第10期67卷 115-128页

作者： Jiayu ZHANG Jiangrong SHEN Zeke WANG Qinghai GUO Rui YAN Gang PAN Huajin TANG College of Computer Science and Technology Zhejiang University The State Key Lab of Brain-Machine Intelligence Zhejiang University Collaborative Innovation Center of Artificial Intelligence Zhejiang University Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd. College of Computer Science and Technology Zhejiang University of Technology MOE Frontier Science Center for Brain Science and Brain-Machine Integration Zhejiang University

In the era of large-scale pretrained models, artificial neural networks（ANNs） have excelled in natural language understanding（NLU） tasks. However, their success often necessitates substantial computational resources and energy consumption. To address this, we explore the potential of spiking neural networks（SNNs） in NLU——a promising avenue with demonstrated advantages, including reduced power consumption and improved efficiency due to their event-driven characteristics. We propose the SpikingMiniLM,a novel spiking Transformer model tailored for natural language understanding. We first introduce a multi-step encoding method to convert text embeddings into spike trains. Subsequently, we redesign the attention mechanism and residual connections to make our model operate on the pure spike-based paradigm without any normalization technique. To facilitate stable and fast convergence, we propose a general parameter initialization method grounded in the stable firing rate principle. Furthermore, we apply an ANN-to-SNN knowledge distillation to overcome the challenges of pretraining SNNs. Our approach achieves a macro-average score of 75.5 on the dev sets of the GLUE benchmark, retaining 98% of the performance exhibited by the teacher model MiniLMv2. Our smaller model also achieves similar performance to BERTMINIwith fewer parameters and much lower energy consumption, underscoring its competitiveness and resource efficiency in NLU tasks.

关键词： spiking neural networks natural language understanding spiking Transformer spike-based attention multi-step encoding ANN-to-SNN distillation

来源：评论

学校读者我要写书评

暂无评论

Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation

引用

IEEE Transactions on Multimedia 2024年 1-12页

作者： Tang, Yushun Chen, Shuoshuo Kan, Zhehan Zhang, Yi Guo, Qinghai He, Zhihai Department of Electrical and Electronic Engineering Southern University of Science and Technology Shenzhen China Advanced Computing and Storage Laboratory Huawei Technologies Co. LT China

Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. This work is based on the following interesting finding: in transformer-based image classification, the class token at the first transformer encoder layer can be learned to capture the domain-specific characteristics of target samples during test-time adaptation. This learned token, when combined with input image patch embeddings, is able to gradually remove the domain-specific information from the feature representations of input samples during the transformer encoding process, thereby significantly improving the test-time adaptation performance of the source model across different domains. We refer to this class token as visual conditioning token (VCT). To successfully learn the VCT, we propose a bi-level learning approach to capture the longterm variations of domain-specific characteristics while accommodating local variations of instance-specific characteristics. Experimental results on the benchmark datasets demonstrate that our proposed bi-level visual conditioning token learning method is able to achieve significantly improved test-time adaptation performance by up to 1.9% Authors

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

Thermal transport in lithium-ion battery: A micro perspective for thermal management

引用

Frontiers of physics 2022年第1期17卷 143-153页

作者： Changqing Xiang Cheng-Wei Wu Wu-Xing Zhou Guofeng Xie Gang Zhang College of Information Science and Engineering Jishou UniversityJishou416000China School of Materials Science and Engineering&Hunan Provincial Key Laboratory of Advanced Materials for New Energy Storage and Conversion Hunan University of Science and TechnologyXiangtan411201China Institute of High Performance Computing A*STAR Singapore138632Singapore

In recent years, lithium ion (Li-ion) batteries have served as significant power sources in portable electronic devices and electric vehicles because of their high energy density and rate capability. There are growing concerns towards the safety of Li-ion batteries, in which thermal conductivities of anodes, cathodes, electrolytes and separator play key roles for determining the thermal energy transport in Li-ion battery. In this review, we summarize the state-of-the-art studies on the thermal conductivities of commonly used anodes, cathodes, electrolytes and separator in Li-ion batteries, including both theoretical and experimental reports. First, the thermal conductivities of anodes and cathodes are discussed, and the effects of delithiation degree and temperature of materials are also discussed. Then, we review the thermal conductivities of commonly used electrolytes, especially on solid electrolytes. Finally, the basic concept of interfacial thermal conductance and simulation methods are presented, as well as the interfacial thermal conductance between separator and cathodes. This perspective review would provide atomic perspective knowledge to understand thermal transport in Li-ion battery, which will be beneficial to the thermal management and temperature control in electrochemical energy storage devices.

关键词： lithium ion batteries thermal management phonon transport interfacial thermal transport amorphous materials

来源：评论

学校读者我要写书评

暂无评论

Cooperative structure of Li/Ni mixing and stacking faults for achieving high-capacity Co-free Li-rich oxides

引用

Journal of Energy Chemistry 2024年第8期95卷 315-324,I0007页

作者： Zhen Wu Yu-Han Zhang Hao Wang Zewen Liu Xudong Zhang Xin Dai Kunyang Zou Xiaoming Lou Xuechen Hu Lijing Ma Yan Liu Yongning Liu State Key Laboratory for Mechanical Behavior of Materials Xi’an Jiaotong UniversityXi’an 710049ShaanxiChina Qingdao Industrial Energy Storage Research Institute Qingdao Institute of Bioenergy and Bioprocess TechnologyChinese Academy of SciencesQingdao 266101ShandongChina Hefei Advanced Computing Center Operation Management Corp Ltd Hefei 230088AnhuiChina Center for High Performance Computing Network Information CenterXi’an Jiaotong UniversityXi’an 710049ShaanxiChina Wuhan Dongfeng Motor Corporation Ltd Wuhan 430056HubeiChina School of Materials Science and Intelligent Engineering Nanjing UniversitySuzhou 215163JiangsuChina Key Laboratory of MEMS of the Ministry of Education Southeast UniversityNanjing 210096JiangsuChina International Research Center for Renewable Energy State Key Laboratory of Multiphase Flow in Power Engineering Xi’an Jiaotong UniversityXi’an 710049ShaanxiChina

Co-free Li-rich layered oxides(LLOs)are emerging as promising cathode materials for Li-ion batteries due to their low cost and high ***,they commonly face severe structural instability and poor electrochemical activity,leading to diminished capacity and voltage ***,we introduce a Co-free LLO,Li_(1.167)Ni_(0.222)Mn_(0.611)O_(2)(Cf-L1),which features a cooperative structure of Li/Ni mixing and stacking *** structure regulates the crystal and electronic structures,resulting in a higher discharge capacity of 300.6 mA h g^(-1)and enhanced rate capability compared to the typical Co-free LLO,Li_(1.2)Ni_(0.2)Mn_(0.6)O_(2)(Cf-Ls).Density functional theory(DFT)indicates that Li/Ni mixing in LLOs leads to increased Li-O-Li configurations and higher anionic redox activities,while stacking faults further optimize the electronic interactions of transition metal(TM)3d and non-bonding O 2p ***,stacking faults accommodate lattice strain,improving electrochemical reversibility during charge/discharge cycles,as demonstrated by the in situ XRD of Cf-L1 showing less lattice evolution than *** study offers a structured approach to developing Co-free LLOs with enhanced capacity,voltage,rate capability,and cyclability,significantly impacting the advancement of the next-generation Li-ion batteries.

关键词： Co-free Li-rich oxides Li/Ni mixing Stacking faults Electronic structure

来源：评论

学校读者我要写书评

暂无评论

Explicit Mutual Information Maximization for Self-Supervised Learning

Explicit Mutual Information Maximization for Self-Supervised...

引用

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Lele Chang Peilin Liu Qinghai Guo Fei Wen School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University Shanghai China Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd Shenzhen China

ISBN: (数字)9798350368741

ISBN: (纸本)9798350368758

Recently, self-supervised learning (SSL) has been extensively studied. Theoretically, mutual information maximization (MIM) is an optimal criterion for SSL, with a strong theoretical foundation in information theory. However, it is difficult to directly apply MIM in SSL since the data distribution is not analytically available in applications. In practice, many existing methods can be viewed as approximate implementations of the MIM criterion. This work shows that, based on the invariance property of MI, explicit MI maximization can be applied to SSL under a generic distribution assumption, i.e., a relaxed condition of the data distribution. We further illustrate this by analyzing the generalized Gaussian distribution. Based on this result, we derive a loss function based on the MIM criterion using only second-order statistics. We implement the new loss for SSL and demonstrate its effectiveness via extensive experiments.

关键词： Training Self-supervised learning Gaussian distribution Signal processing Benchmark testing Acoustics Speech processing Mutual information Optimization

来源：评论

学校读者我要写书评

暂无评论

Explicit Mutual Information Maximization for Self-Supervised Learning

arXiv

引用

arXiv 2024年

作者： Chang, Lele Liu, Peilin Guo, Qinghai Wen, Fei School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University Shanghai China Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd. Shenzhen China

关键词： Self-supervised learning

来源：评论

学校读者我要写书评

暂无评论

Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation

arXiv

引用

arXiv 2024年

作者： Tang, Yushun Chen, Shuoshuo Kan, Zhehan Zhang, Yi Guo, Qinghai He, Zhihai The Department of Electrical and Electronic Engineering Southern University of Science and Technology Shenzhen China The Pengcheng Lab Shenzhen China The Advanced Computing and Storage Laboratory Huawei Technologies Co. LTD China

Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. This work is based on the following interesting finding: in transformer-based image classification, the class token at the first transformer encoder layer can be learned to capture the domain-specific characteristics of target samples during test-time adaptation. This learned token, when combined with input image patch embeddings, is able to gradually remove the domain-specific information from the feature representations of input samples during the transformer encoding process, thereby significantly improving the test-time adaptation performance of the source model across different domains. We refer to this class token as visual conditioning token (VCT). To successfully learn the VCT, we propose a bi-level learning approach to capture the long-term variations of domain-specific characteristics while accommodating local variations of instance-specific characteristics. Experimental results on the benchmark datasets demonstrate that our proposed bi-level visual conditioning token learning method is able to achieve significantly improved test-time adaptation performance by up to 1.9%. Copyright © 2024, The Authors. All rights reserved.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation

arXiv

引用

arXiv 2023年

作者： Tang, Yushun Zhang, Ce Xu, Heng Chen, Shuoshuo Cheng, Jie Leng, Luziwei Guo, Qinghai He, Zhihai Department of Electronic and Electrical Engineering Southern University of Science and Technology Shenzhen China Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd. Shenzhen China

Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. We take inspiration from the biological plausibility learning where the neuron responses are tuned based on a local synapse-change procedure and activated by competitive lateral inhibition rules. Based on these feed-forward learning rules, we design a soft Hebbian learning process which provides an unsupervised and effective mechanism for online adaptation. We observe that the performance of this feed-forward Hebbian learning for fully test-time adaptation can be significantly improved by incorporating a feedback neuro-modulation layer. It is able to fine-tune the neuron responses based on the external feedback generated by the error back-propagation from the top inference layers. This leads to our proposed neuro-modulated Hebbian learning (NHL) method for fully test-time adaptation. With the unsupervised feed-forward soft Hebbian learning being combined with a learned neuro-modulator to capture feedback from external responses, the source model can be effectively adapted during the testing process. Experimental results on benchmark datasets demonstrate that our proposed method can significantly improve the adaptation performance of network models and outperforms existing state-of-the-art methods. Copyright © 2023, The Authors. All rights reserved.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

Cross-Inferential Networks for Source-Free Unsupervised Domain Adaptation

Cross-Inferential Networks for Source-Free Unsupervised Doma...

引用

IEEE International Conference on Image Processing

作者： Yushun Tang Qinghai Guo Zhihai He Department of Electronic and Electrical Engineering Southern University of Science and Technology Shenzhen China Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd. Shenzhen China Pengcheng Laboratory Shenzhen China

One central challenge in source-free unsupervised domain adaptation (UDA) is the lack of an effective approach to evaluate the prediction results of the adapted network model in the target domain. To address this challenge, we propose to explore a new method called cross-inferential networks (CIN). Our main idea is that, when we adapt the network model to predict the sample labels from encoded features, we use these prediction results to construct new training samples with derived labels to learn a new examiner network that performs a different but compatible task in the target domain. Specifically, in this work, the base network model is performing image classification while the examiner network is tasked to perform relative ordering of triplets of samples whose training labels are carefully constructed from the prediction results of the base network model. Two similarity measures, cross-network correlation matrix similarity and attention consistency, are then developed to provide important guidance for the UDA process. Our experimental results on benchmark datasets demonstrate that our proposed CIN approach can significantly improve the performance of source-free UDA.

关键词：

来源：评论

学校读者我要写书评

暂无评论

CROSS-INFERENTIAL NETWORKS FOR SOURCE-FREE UNSUPERVISED DOMAIN ADAPTATION

arXiv

引用

arXiv 2023年

作者： Tang, Yushun Guo, Qinghai He, Zhihai Department of Electronic and Electrical Engineering Southern University of Science and Technology Shenzhen China Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd. Shenzhen China Pengcheng Laboratory Shenzhen China

关键词： Forecasting

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：