Deep learning has become the cornerstone of artificial intelligence,playing an increasingly important role in human production and ***,as the complexity of problem-solving increases,deep learning models become increas...
详细信息
Deep learning has become the cornerstone of artificial intelligence,playing an increasingly important role in human production and ***,as the complexity of problem-solving increases,deep learning models become increasingly intricate,resulting in a proliferation of large language models with an astonishing number of *** model parallelism(PMP)has emerged as one of the mainstream approaches to addressing the significant challenge of training“big models”.This paper presents a comprehensive review of *** covers the basic concepts and main challenges of *** also comprehensively compares synchronous and asynchronous pipeline schedules for PMP approaches,and discusses the main techniques to achieve load balance for both intra-node and inter-node ***,the main techniques to optimize computation,storage,and communication are presented,with potential research directions being discussed.
Detecting rotated faces has always been a challenging task. Fixed convolutional kernels struggle to effectively match features after rotation, while the sampling point offsets of deformable convolutions are limited by...
详细信息
The rapid deployment and low-cost inference of controller area network (CAN) bus anomaly detection models on intelligent vehicles can drive the development of the Green Internet of Vehicles. Anomaly detection on intel...
详细信息
Faced with an escalating number of fingerprint images, most existing retrieval approachs suffer from a common problem: diminishing computational efficiency. This paper presents a hierarchical retrieval system tailored...
详细信息
The presence of fake tasks in mobile crowd sensing severely affects the normal operation of the platform. Due to the rapid development of deep learning, 'Adversarial Fake Tasks' now have a greater destructive ...
详细信息
This paper presents a novel algorithm for training robotic arms to manipulate cloth, by leveraging reinforcement learning and curriculum learning approaches. Traditional cloth manipulation algorithms rely heavily on p...
详细信息
With the development of deep learning in recent years, code representation learning techniques have become the foundation of many software engineering tasks such as program classification [1] and defect detection. Ear...
With the development of deep learning in recent years, code representation learning techniques have become the foundation of many software engineering tasks such as program classification [1] and defect detection. Earlier approaches treat the code as token sequences and use CNN, RNN, and the Transformer models to learn code representations.
Traditional techniques for network traffic classification are no longer effective in handling the complexities of dynamic network environments. Moreover, deep learning methods, while powerful, demand substantial spati...
详细信息
In the era of large-scale pretrained models, artificial neural networks（ANNs） have excelled in natural language understanding（NLU） tasks. However, their success often necessitates substantial computational resourc...
详细信息
In the era of large-scale pretrained models, artificial neural networks（ANNs） have excelled in natural language understanding（NLU） tasks. However, their success often necessitates substantial computational resources and energy consumption. To address this, we explore the potential of spiking neural networks（SNNs） in NLU——a promising avenue with demonstrated advantages, including reduced power consumption and improved efficiency due to their event-driven characteristics. We propose the SpikingMiniLM,a novel spiking Transformer model tailored for natural language understanding. We first introduce a multi-step encoding method to convert text embeddings into spike trains. Subsequently, we redesign the attention mechanism and residual connections to make our model operate on the pure spike-based paradigm without any normalization technique. To facilitate stable and fast convergence, we propose a general parameter initialization method grounded in the stable firing rate principle. Furthermore, we apply an ANN-to-SNN knowledge distillation to overcome the challenges of pretraining SNNs. Our approach achieves a macro-average score of 75.5 on the dev sets of the GLUE benchmark, retaining 98% of the performance exhibited by the teacher model MiniLMv2. Our smaller model also achieves similar performance to BERTMINIwith fewer parameters and much lower energy consumption, underscoring its competitiveness and resource efficiency in NLU tasks.
Owing to the extensive applications in many areas such as networked systems,formation flying of unmanned air vehicles,and coordinated manipulation of multiple robots,the distributed containment control for nonlinear m...
Owing to the extensive applications in many areas such as networked systems,formation flying of unmanned air vehicles,and coordinated manipulation of multiple robots,the distributed containment control for nonlinear multiagent systems (MASs) has received considerable attention,for example [1,2].Although the valued studies in [1,2] investigate containment control problems for MASs subject to nonlinearities,the proposed distributed nonlinear protocols only achieve the asymptotic *** a crucial performance indicator for distributed containment control of MASs,the fast convergence is conducive to achieving better control accuracy [3].The work in [4] first addresses the backstepping-based adaptive fuzzy fixed-time containment tracking problem for nonlinear high-order MASs with unknown external ***,the designed fixedtime control protocol [4] cannot escape the singularity problem in the backstepping-based adaptive control *** is well known,the singularity problem has become an inherent problem in the adaptive fixed-time control design,which may cause the unbounded control inputs and even the instability of controlled ***,how to solve the nonsingular fixed-time containment control problem for nonlinear MASs is still open and awaits breakthrough to the best of our knowledge.
暂无评论