检索结果-内蒙古大学图书馆

12th International Conference on Learning Representations, ICLR 2024

作者： Nahshan, Yury Kampeas, Joseph Haleva, Emir Distributed and Parallel Software Lab Huawei Technologies China

Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models. © 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Object-oriented run-time support for data-parallel applications 2nd

Object-oriented run-time support for data-parallel applicati...

引用

2nd International Symposium on Computing in Object-Oriented parallel Environments

作者： Bi, H Kessler, M Wilhelmi, M GMD Inst Comp Architecture & Software Technol Parallel & Distributed Syst Lab D-12489 Berlin Germany

ISBN: (纸本)3540653872

We present a C++ template run-time library, PROMOTER, and discuss run-time support for data-parallel applications. The PROMOTER run-time library provides a uniform framework for data-parallel applications, covering a broad spectrum of granularity, regularity and dynamicity. It supports user-defined data structures ranging from dense to sparse arrays, regular to irregular index structures and data distributions. The object-oriented design and implementation of the PROMOTER run-time library not only provides an easy data-parallel programming environment, but also leads to an efficient implementation of data-parallel applications through object reuse and object specialization.

关键词： C++ (programming language)

来源：评论

学校读者我要写书评

暂无评论

Modifying the softening process for knowledge distillation

引用

JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2022年第3期42卷 2247-2259页

作者： Tan, Chao Liu, Jie Natl Univ Def Technol Sci & Parallel & Distributed Proc Lab Changsha Peoples R China Natl Univ Def Technol Lab Software Engn Complex Syst Changsha Peoples R China

The prime focus of knowledge distillation (KD) seeks a light proxy termed student to mimic the outputs of its heavy neural networks termed teacher, and makes the student run real-time on the resource-limited devices. This paradigm requires aligning the soft logits of both teacher and student. However, few doubts whether the process of softening the logits truly give full play to the teacher-student paradigm. In this paper, we launch several analyses to delve into this issue from scratch. Subsequently, several simple yet effective functions are devised to replace the vanilla KD. The ultimate function can be an effective alternative to its original counterparts and work well with other skills like FitNets. To claim this point, we conduct several visual tasks on individual benchmarks, and experimental results verify the potential of our proposed function in terms of performance gains. For example, when the teacher and student networks are ShuffleNetV2-1.0 and ShuffleNetV2-0.5, our proposed method achieves 40.88% top-1 error rate on Tiny ImageNet.

关键词： Neural network compression knowledge distillation knowledge transfer

来源：评论

学校读者我要写书评

暂无评论

Online knowledge distillation with elastic peer

引用

INFORMATION SCIENCES 2022年 583卷 1-13页

作者： Tan, Chao Liu, Jie Natl Univ Def Technol Sci & Parallel & Distributed Proc Lab Changsha 410073 Peoples R China Natl Univ Def Technol Lab Software Engn Complex Syst Changsha 410073 Peoples R China

Knowledge distillation is a highly effective method for transferring knowledge from a cum-bersome teacher network to a lightweight student network. However, teacher networks are not always available. An alternative method called online knowledge distillation, which applies a group of peer networks to learn collaboratively with each other, has been pro -posed previously. In this study, we revisit online knowledge distillation and find that the existing training strategy limits the diversity among peer networks. Thus, online knowledge distillation cannot achieve its full potential. To address this, a novel online knowledge dis-tillation with elastic peer (KDEP) strategy is introduced here. The entire training process is divided into two phases by KDEP. In each phase, a specific training strategy is applied to adjust the diversity to an appropriate degree. Extensive experiments have been conducted on individual benchmarks, including CIFAR-100, CINIC-10, Tiny ImageNet, and Caltech-UCSD Birds. The results demonstrate the superiority of KDEP. For example, when the peer networks are ShuffleNetV2-1.0 and ShuffleNetV2-0.5, the target peer network ShuffleNetV2-0.5 achieves 57:00% top-1 accuracy on Tiny ImageNet via KDEP. (c) 2021 Elsevier Inc. All rights reserved.

关键词： Neural network compression Knowledge distillation Knowledge transfer

来源：评论

学校读者我要写书评

暂无评论

Improving Knowledge Distillation With a Customized Teacher

引用

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024年第2期35卷 2290-2299页

Knowledge distillation (KD) is a widely used approach to transfer knowledge from a cumbersome network (also known as a teacher) to a lightweight network (also known as a student). However, even though the accuracies of different teachers are similar, the fixed student's accuracies are significantly different. We find that teachers with more dispersed secondary soft probabilities are more qualified to play their roles. Therefore, an indicator, i.e., the standard deviation sigma of secondary soft probabilities, is introduced to choose the teacher. Moreover, to make a teacher's secondary soft probabilities more dispersed, a novel method, dubbed pretraining the teacher under dual supervision (PTDS), is proposed to pretrain a teacher under dual supervision. In addition, we put forward an asymmetrical transformation function (ATF) to further enhance the dispersion degree of the pretrained teachers' secondary soft probabilities. The combination of PTDS and ATF is termed knowledge distillation with a customized teacher (KDCT). Extensive empirical experiments and analyses are conducted on three computer vision tasks, including image classification, transfer learning, and semantic segmentation, to substantiate the effectiveness of KDCT.

关键词： Knowledge distillation (KD) knowledge transfer neural network acceleration neural network compression

来源：评论

学校读者我要写书评

暂无评论

SaSpGEMM: Sorting-Avoiding Sparse General Matrix-Matrix Multiplication on Multi-Core Processors 24

SaSpGEMM: Sorting-Avoiding Sparse General Matrix-Matrix Mult...

引用

53rd International Conference on parallel Processing (ICPP)

作者： Hong, Chuhe Wang, Qinglin Mao, Runzhang Liang, Yuechao Xia, Rui Liu, Jie Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Natl Key Lab Parallel & Distributed Comp Changsha Peoples R China

ISBN: (纸本)9798400717932

We propose the SaSpGEMM: a parallel sparse general matrix-matrix multiplication (SpGEMM) to avoid the overhead of sorting. The typical workflow of SpGEMM contains: size prediction, memory allocation, numeric calculation and sorting. However, sorting has always been overlooked as a bottleneck in the performance of SpGEMM. It constitutes an average of 30% in HASH and 55% in ESC during the calculation stage. The key idea behind SaSpGEMM is to leverage the compressed sparse row (CSR) storage format's feature of increasing index order of elements in the same row which is sorted during preprocessing, preserving intermediate products ordered consistently. To achieve this, we introduce a linked list-based accumulator (LLA) designed for batch insertion while maintaining order with a low time complexity. We provide a comprehensive empirical evidence showing that SaSpGEMM outperforms other methods based on time complexity analysis. Compared to three state-of-the-art methods ESC, SPA, and HASH on both x86 (Intel Xeon Gold 6348) and ARM (Phytium2000+) architectures, our method achieves an average speedup of 2.82x, 5.24x, 1.16x (with a maximum speedup of 34.4x, 195x, 23.3x) on the Intel Xeon Gold 6348. On the Phytium2000+, it achieves an average speedup of 2.21x, 4,65x, 1.05x (with a maximum speedup of 40.93x, 146.7x, 9.03x).

关键词： SpGEMM linked list-based accumulator sorting-avoiding multicore processors

来源：评论

学校读者我要写书评

暂无评论

Hard Contrastive Learning for Video Captioning 5

Hard Contrastive Learning for Video Captioning

引用

IEEE 5th International Conference on Electronics and Communication Engineering (ICECE)

作者： Wu, Lilei Liu, Jie Natl Univ Def Technol Sci & Technol Parallel & Distributed Proc Lab Lab Software Engn Complex Syst Changsha Peoples R China

ISBN: (纸本)9781665487894

Maximum likelihood estimation has been widely adopted along with the encoder-decoder framework for video captioning. However, it ignores the structure of sentences and restrains the diversity and distinction of generated captions. To address this issue, we propose a hard contrastive learning (HCL) method for video captioning. Specifically, built on the encoderdecoder framework, we introduce mismatched pairs to learn a reference distribution of video descriptions. The target model on the matched pairs is learned on top the reference model, which improves the distinctiveness of generated captions. In addition, we further boost the distinctiveness of the captions by developing a hard mining technique to select the hardest mismatched pairs within the contrastive learning framework. Finally, the relationships among multiple relevant captions for each video is consider to encourage the diversity of generated captions. The proposed method generates high quality captions which effectively capture the specialties in individual videos. Extensive experiments on two benchmark datasets, i.e., MSVD and MSR-VTT, show that our approach outperforms state-ofthe-art methods.

关键词： Hard Sample Mining Contrastive Learning

来源：评论

学校读者我要写书评

暂无评论

Hardware Support for Concurrent Detection of Multiple Concurrency Bugs on Fused CPU-GPU Architectures

引用

IEEE TRANSACTIONS ON COMPUTERS 2016年第10期65卷 3083-3095页

作者： Zhang, Weihua Yu, Shiqiang Wang, Haojun Dai, Zhuofang Chen, Haibo Fudan Univ Software Sch Shanghai Key Lab Data Sci Shanghai Peoples R China Shanghai Jiao Tong Univ Inst Parallel & Distributed Syst Shanghai 200030 Peoples R China

Detecting concurrency bugs, such as data race, atomicity violation and order violation, is a cumbersome task for programmers. This situation is further being exacerbated due to the increasing number of cores in a single machine and the prevalence of threaded programming models. Unfortunately, many existing software-based approaches usually incur high runtime overhead or accuracy loss, while most hardware-based proposals usually focus on a specific type of bugs and thus are inflexible to detect a variety of concurrency bugs. In this paper, we propose Hydra, an approach that leverages massive parallelism and programmability of fused CPU-GPU architectures to simultaneously detect multiple concurrency bugs in threaded software, including data race, atomicity violation and order violation. Hydra extends contemporary fused CPU and GPU by introducing two modules: 1) a trace collecting module (TCM) that instruments and collects program behavior on CPU;2) a trace preprocessing module (TPM) that processes and then transfers the traces to GPU for bug detection. Furthermore, Hydra exploits three optimizations to improve speed and accuracy, which includes: 1). using the bloom filter to filter out unnecessary traces;2). avoiding eviction of shared traces;3). comparing only last-write traces for shared data with the happens-before relation. Hydra incurs small hardware complexity and requires no changes to internal critical-path processor components such as cache and its coherence protocol, and is with about 1.1 percent hardware overhead under a 32-core configuration. Experimental results show that Hydra only introduces about 0.18 percent overhead on average for detecting one type of bugs and 0.46 percent overhead for simultaneously detecting multiple bugs, yet with the similar detectability of a heavyweight software bug detector (e.g., Helgrind).

关键词： Concurrency bug detection fused CPU-GPU architecture

来源：评论

学校读者我要写书评

暂无评论

How Do Developers Adapt Code Snippets to Their Contexts? An Empirical Study of Context-Based Code Snippet Adaptations

引用

IEEE TRANSACTIONS ON software ENGINEERING 2024年第11期50卷 2712-2731页

作者： Zhang, Tanghaoran Lu, Yao Yu, Yue Mao, Xinjun Zhang, Yang Zhao, Yuxin Natl Univ Def Technol Key Lab Software Engn Complex Syst Changsha 410073 Peoples R China Natl Univ Def Technol Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

Reusing code snippets from online programming Q&A communities has become a common development practice, in which developers often need to adapt code snippets to their code contexts to satisfy their own programming needs. However, how developers make these code adaptations based on contexts is still unclear. To bridge this gap, we first conduct a semi-structured interview of 21 developers to investigate their adaptation practices and perceived challenges during this process. The result suggests that code snippet adaptation is a challenging and exhausting task for developers, as they should tailor the snippets to guarantee their correctness and quality with laborious work. We also note that developers all resort to their intra-file context to complete adaptations, which motivates us to further study how developers performed context-based adaptations (CAs) in real scenarios. To this end, we conduct a quantitative study on an adaptation dataset comprising 300 code snippet reuse cases with 1,384 adaptations from Stack Overflow to GitHub. For each adaptation, we manually annotate its intention and relationship with the context. Based on our annotated data, we employ frequent itemset mining to obtain four CA patterns from our dataset, including Fortification, Code Wiring, Attribute-ization and Parameterization. Our main findings reveal that: (1) more than half of the code snippet reuse cases include CAs and 23.3% of the adaptations are CAs;(2) more than half of the CAs are corrective adaptations and variable is the primary adapted language construct;(3) attribute is the most frequently utilized context and 88% of the local contexts are within the nearest 10 LOCs;and (4) CAs towards different intentions are repetitive, which are useful for automatic adaptation. Overall, our study provides valuable insights into code snippet adaptation and has important implications for research, practice, and tool design.

关键词： Codes Interviews Programming Task analysis software development management Protocols Wiring Code reuse code snippet adaptation adaptation pattern semi-structured interview

来源：评论

学校读者我要写书评

暂无评论

Characterizing the Occurrence of Dockerfile Smells in Open-Source software: An Empirical Study

引用

IEEE ACCESS 2020年 8卷 34127-34139页

作者： Wu, Yiwen Zhang, Yang Wang, Tao Wang, Huaimin Natl Univ Def Technol Sci & Technol Parallel & Distributed Lab Changsha 410073 Hunan Peoples R China Natl Univ Def Technol Key Lab Software Engn Complex Syst Changsha 410073 Hunan Peoples R China

Dockerfile plays an important role in the Docker-based software development process, but many Dockerfile codes are infected with smells in practice. Understanding the occurrence of Dockerfile smells in open-source software can benefit the practice of Dockerfile and enhance project maintenance. In this paper, we perform an empirical study on a large dataset of 6,334 projects to help developers gain some insights into the occurrence of Dockerfile smells, including its coverage, distribution, co-occurrence, and correlation with project characteristics. Our results show that smells are very common in Dockerfile codes and there exists co-occurrence between different types of Dockerfile smells. Further, using linear regression analysis, when controlled for various variables, we statistically identify and quantify the relationships between Dockerfile smells occurrence and project characteristics. We also provide a rich resource of implications for software practitioners.

关键词： Containers Open source software Best practices Databases Buildings Licenses Docker Dockerfile smells Open-source software GitHub

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：