检索结果-内蒙古大学图书馆

12th International Conference on Learning Representations, ICLR 2024

作者： Nahshan, Yury Kampeas, Joseph Haleva, Emir Distributed and Parallel Software Lab Huawei Technologies China

Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models. © 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

3DMeshNet: A three-dimensional differential neural network for structured mesh generation

引用

GRAPHICAL MODELS 2025年 139卷

作者： Peng, Jiaming Chen, Xinhai Liu, Jie Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Changsha 410073 Peoples R China Natl Univ Def Technol Natl Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

Mesh generation is a crucial step in numerical simulations, significantly impacting simulation accuracy and efficiency. However, generating meshes remains time-consuming and requires expensive computational resources. In this paper, we propose a novel method, 3DMeshNet, for three-dimensional structured mesh generation. The method embeds the meshing-related differential equations into the loss function of neural networks, formulating the meshing task as an unsupervised optimization problem. It takes geometric points as input to learn the potential mapping between parametric and computational spaces. After suitable offline training, 3DMeshNet can efficiently output a three-dimensional structured mesh with a user-defined number of quadrilateral/hexahedral cells through the feed-forward neural prediction. To enhance training stability and accelerate convergence, we integrate loss function reweighting through weight adjustments and gradient projection alongside applying finite difference methods to streamline derivative computations in the loss. Experiments on different cases show that 3DMeshNet is robust and fast. It outperforms neural network- based methods and yields superior meshes compared to traditional mesh partitioning methods. 3DMeshNet significantly reduces training times by up to 85% compared to other neural network-based approaches and lowers meshing overhead by 4 to 8 times relative to traditional meshing methods.

关键词： Mesh generation Three-dimensional structured mesh Differential equations Neural network

来源：评论

学校读者我要写书评

暂无评论

Improving Knowledge Distillation With a Customized Teacher

引用

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024年第2期35卷 2290-2299页

作者： Tan, Chao Liu, Jie Natl Univ Def Technol Sci & Parallel & Distributed Proc Lab Changsha 410073 Peoples R China Natl Univ Def Technol Lab Software Engn Complex Syst Changsha 410073 Peoples R China

Knowledge distillation (KD) is a widely used approach to transfer knowledge from a cumbersome network (also known as a teacher) to a lightweight network (also known as a student). However, even though the accuracies of different teachers are similar, the fixed student's accuracies are significantly different. We find that teachers with more dispersed secondary soft probabilities are more qualified to play their roles. Therefore, an indicator, i.e., the standard deviation sigma of secondary soft probabilities, is introduced to choose the teacher. Moreover, to make a teacher's secondary soft probabilities more dispersed, a novel method, dubbed pretraining the teacher under dual supervision (PTDS), is proposed to pretrain a teacher under dual supervision. In addition, we put forward an asymmetrical transformation function (ATF) to further enhance the dispersion degree of the pretrained teachers' secondary soft probabilities. The combination of PTDS and ATF is termed knowledge distillation with a customized teacher (KDCT). Extensive empirical experiments and analyses are conducted on three computer vision tasks, including image classification, transfer learning, and semantic segmentation, to substantiate the effectiveness of KDCT.

关键词： Knowledge distillation (KD) knowledge transfer neural network acceleration neural network compression

来源：评论

学校读者我要写书评

暂无评论

SaSpGEMM: Sorting-Avoiding Sparse General Matrix-Matrix Multiplication on Multi-Core Processors 24

SaSpGEMM: Sorting-Avoiding Sparse General Matrix-Matrix Mult...

引用

53rd International Conference on parallel Processing (ICPP)

作者： Hong, Chuhe Wang, Qinglin Mao, Runzhang Liang, Yuechao Xia, Rui Liu, Jie Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Natl Key Lab Parallel & Distributed Comp Changsha Peoples R China

ISBN: (纸本)9798400717932

We propose the SaSpGEMM: a parallel sparse general matrix-matrix multiplication (SpGEMM) to avoid the overhead of sorting. The typical workflow of SpGEMM contains: size prediction, memory allocation, numeric calculation and sorting. However, sorting has always been overlooked as a bottleneck in the performance of SpGEMM. It constitutes an average of 30% in HASH and 55% in ESC during the calculation stage. The key idea behind SaSpGEMM is to leverage the compressed sparse row (CSR) storage format's feature of increasing index order of elements in the same row which is sorted during preprocessing, preserving intermediate products ordered consistently. To achieve this, we introduce a linked list-based accumulator (LLA) designed for batch insertion while maintaining order with a low time complexity. We provide a comprehensive empirical evidence showing that SaSpGEMM outperforms other methods based on time complexity analysis. Compared to three state-of-the-art methods ESC, SPA, and HASH on both x86 (Intel Xeon Gold 6348) and ARM (Phytium2000+) architectures, our method achieves an average speedup of 2.82x, 5.24x, 1.16x (with a maximum speedup of 34.4x, 195x, 23.3x) on the Intel Xeon Gold 6348. On the Phytium2000+, it achieves an average speedup of 2.21x, 4,65x, 1.05x (with a maximum speedup of 40.93x, 146.7x, 9.03x).

关键词： SpGEMM linked list-based accumulator sorting-avoiding multicore processors

来源：评论

学校读者我要写书评

暂无评论

An adaptive gradient correction method based on mesh skewness for finite volume fluid dynamics simulations

引用

PHYSICS OF FLUIDS 2025年第1期37卷

作者： Song, Min Li, Chao Guo, Xiaowei Liu, Jie Natl Univ Def Technol Sci & Technol Parallel & Distributed Proc Lab Changsha 410073 Hunan Peoples R China Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Changsha 410073 Hunan Peoples R China

In unstructured mesh solvers based on the finite volume method, accurate gradient calculation is crucial for determining the accuracy of equations. However, most mainstream computational fluid dynamics (CFD) software offer only uncorrected or globally corrected gradient options, which do not take into account local geometric characteristics. This limitation hampers computational efficiency and accuracy, especially in complex geometries, leading to poor convergence or even divergence, thus constraining its practical application in real-world engineering problems. To address these issues, this paper presents an adaptive gradient correction algorithm based on mesh skewness. By analyzing the geometric characteristics of mesh cells and dynamically applying a correction value, the algorithm modifies the gradient calculation formula adaptively to reduce numerical errors and improve accuracy, while ensuring stability. The algorithm has been successfully implemented in a general-purpose CFD software and validated through complex engineering cases such as sub-channel analysis, demonstrating its practical engineering value. Results indicate that for turbulent flow through a pipe, the proposed adaptive correction algorithm achieves an improved accuracy, demonstrating high precision and superiority in capturing intricate details of turbulence. In more complex scenarios like sub-channel analysis, the adaptive correction algorithm yields an error of 2.54% compared to 4.03% for the uncorrected approach. Unlike global correction methods that may lead to computational divergence within a few iterations, the adaptive correction method consistently maintains stability, accelerates convergence, and improves accuracy, offering significant theoretical and practical value.

关键词： Mesh generation

来源：评论

学校读者我要写书评

暂无评论

How Do Developers Adapt Code Snippets to Their Contexts? An Empirical Study of Context-Based Code Snippet Adaptations

引用

IEEE TRANSACTIONS ON software ENGINEERING 2024年第11期50卷 2712-2731页

作者： Zhang, Tanghaoran Lu, Yao Yu, Yue Mao, Xinjun Zhang, Yang Zhao, Yuxin Natl Univ Def Technol Key Lab Software Engn Complex Syst Changsha 410073 Peoples R China Natl Univ Def Technol Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

Reusing code snippets from online programming Q&A communities has become a common development practice, in which developers often need to adapt code snippets to their code contexts to satisfy their own programming needs. However, how developers make these code adaptations based on contexts is still unclear. To bridge this gap, we first conduct a semi-structured interview of 21 developers to investigate their adaptation practices and perceived challenges during this process. The result suggests that code snippet adaptation is a challenging and exhausting task for developers, as they should tailor the snippets to guarantee their correctness and quality with laborious work. We also note that developers all resort to their intra-file context to complete adaptations, which motivates us to further study how developers performed context-based adaptations (CAs) in real scenarios. To this end, we conduct a quantitative study on an adaptation dataset comprising 300 code snippet reuse cases with 1,384 adaptations from Stack Overflow to GitHub. For each adaptation, we manually annotate its intention and relationship with the context. Based on our annotated data, we employ frequent itemset mining to obtain four CA patterns from our dataset, including Fortification, Code Wiring, Attribute-ization and Parameterization. Our main findings reveal that: (1) more than half of the code snippet reuse cases include CAs and 23.3% of the adaptations are CAs;(2) more than half of the CAs are corrective adaptations and variable is the primary adapted language construct;(3) attribute is the most frequently utilized context and 88% of the local contexts are within the nearest 10 LOCs;and (4) CAs towards different intentions are repetitive, which are useful for automatic adaptation. Overall, our study provides valuable insights into code snippet adaptation and has important implications for research, practice, and tool design.

关键词： Codes Interviews Programming Task analysis software development management Protocols Wiring Code reuse code snippet adaptation adaptation pattern semi-structured interview

来源：评论

学校读者我要写书评

暂无评论

MH-DCNet: An improved flow field prediction framework coupling neural network with physics solver

引用

COMPUTERS & FLUIDS 2024年 284卷

作者： Xiao, Qisong Chen, Xinhai Liu, Jie Gong, Chunye Sun, Yufei Natl Univ Def Technol Sci & Technol Parallel & Distributed Proc Lab Changsha 410073 Peoples R China Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Changsha 410073 Peoples R China Nankai Univ Coll Software Tianjin 300350 Peoples R China

With the development of intelligent computing technology, deep learning methods have provided an efficient solution for rapid flow field prediction in computational fluid dynamics (CFD) problems. However, existing methods have limitations in handling interference among physical variables due to different data distributions, leading to a decline in prediction performance. In this paper, we propose MH-DCNet, an improved flow field prediction framework that couples a neural network with a physics solver. Specifically, to address the data distribution problem, we design a multi-head deep convolutional neural network that decouples the prediction of physical variables through multiple encoders and decoders. We also develop a hybrid loss function by introducing the mean structural similarity to better capture the complex spatial structures and distribution features of flow fields. We evaluate MH-DCNet with unseen geometries and various flow conditions. Experimental results show that MH-DCNet outperforms other advanced models in efficiency and generalization capability. It accelerates the prediction process by 2.35 times compared to the CFD method while meeting the convergence constraints.

关键词： Computational fluid dynamics Deep learning Flow field prediction Hybrid loss function

来源：评论

学校读者我要写书评

暂无评论

A survey and benchmark evaluation for neural-network-based lossless universal compressors toward multi-source data

引用

FRONTIERS OF COMPUTER SCIENCE 2025年第7期19卷 197360-197360页

作者： Sun, Hui Ma, Huidong Ling, Feng Xie, Haonan Sun, Yongxia Yi, Liping Yan, Meng Zhong, Cheng Liu, Xiaoguang Wang, Gang Nankai Univ Coll Comp Sci Nankai Baidu Joint Lab GTIISCParallel & Distributed Software Technol Lab Tianjin 300350 Peoples R China Nanyang Technol Univ Coll Comp & Data Sci Singapore 639798 Singapore Guangxi Univ Inst Artificial Intelligence Sch Elect Engn Nanning 53004 Peoples R China Guangxi Univ Guangxi Univ & Coll Sch Comp Key Lab Parallel Distributed & Intelligent Nanning 53004 Peoples R China

As various types of data grow explosively, large-scale data storage, backup, and transmission become challenging, which motivates many researchers to propose efficient universal compression algorithms for multi-source data. In recent years, due to the emergence of hardware acceleration devices such as GPUs, TPUs, DPUs, and FPGAs, the performance bottleneck of neural networks (NN) has been overcome, making NN-based compression algorithms increasingly practical and popular. However, the research survey for the NN-based universal lossless compressors has not been conducted yet, and there is also a lack of unified evaluation metrics. To address the above problems, in this paper, we present a holistic survey as well as benchmark evaluations. Specifically, i) we thoroughly investigate NN-based lossless universal compression algorithms toward multi-source data and classify them into 3 types: static pre-training, adaptive, and semi-adaptive. ii) We unify 19 evaluation metrics to comprehensively assess the compression effect, resource consumption, and model performance of compressors. iii) We conduct experiments more than 4600 CPU/GPU hours to evaluate 17 state-of-the-art compressors on 28 real-world datasets across data types of text, images, videos, audio, etc. iv) We also summarize the strengths and drawbacks of NN-based lossless data compressors and discuss promising research directions. We summarize the results as the NN-based Lossless Compressors Benchmark (NNLCB, See ***/NNLCB website), which will be updated and maintained continuously in the future.

关键词： lossless compression benchmark evaluation universal compressors neural networks deep learning

来源：评论

学校读者我要写书评

暂无评论

Cross-scale hierarchical spatio-temporal transformer for video enhancement

引用

KNOWLEDGE-BASED SYSTEMS 2025年 309卷

作者： Jiang, Qin Wang, Qinglin Chi, Lihua Liu, Jie Natl Univ Def Technol Changsha Peoples R China Lab Digitizing Software Frontier Equipment Changsha Peoples R China Sci & Technol Parallel & Distributed Proc Lab Changsha Peoples R China Hunan GuoKe Computil Technol Co Ltd Changsha Peoples R China

The diversity and complexity of degradations in low-quality videos pose non-trivial challenges on video enhancement to reconstruct the high-quality counterparts. Prevailing sliding window based methods represent poor performance due to the limitation of window size. Recurrent networks take advantage of long-term modeling to aggregate more information, resulting insignificant performance improvements. However, most of them are trained on simple degraded data and can only tackle specific degradation. To break through the limitation, we propose a progressive alignment network, namely Cross-scale Hierarchical Spatio-Temporal Transformer (CHSTT), which leverages cross-scale tokenization to generate multi-scale visual tokens in the entire sequence to capture extensive long-range temporal dependencies. To enhance the spatial and temporal interactions, we introduce an innovative hierarchical Transformer, facilitating the computation of mutual multi-head attention across both spatial and temporal dimensions. Quantitative and qualitative assessments substantiate the superior performance of CHSTT compared to several state-of-the-art benchmarks across three distinct video enhancement tasks, including video super-resolution, video denoising, and video deblurring.

关键词： Video super-resolution Denoising Deblurring Transformer Temporal

来源：评论

学校读者我要写书评

暂无评论

An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models

引用

ACM TRANSACTIONS ON INFORMATION SYSTEMS 2024年第5期42卷 1-28页

作者： Liu, Qi Guo, Gang Mao, Jiaxin Dou, Zhicheng Wen, Ji-Rong Jiang, Hao Zhang, Xinyu Cao, Zhao Renmin Univ China Gaoling Sch Artificial Intelligence Being Key Lab Data Management & Anal Methods 59 Zhongguancun St Beijing 100872 Peoples R China Distributed & Parallel Software Lab 360 Jiangshu Rd Hangzhou 310056 Zhejiang Peoples R China

With the development of pre-trained language models, the dense retrieval models have become promising alternatives to the traditional retrieval models that rely on exact match and sparse bag-of-words representations. Different from most dense retrieval models using a bi-encoder to encode each query or document into a dense vector, the recently proposed late-interaction multi-vector models (i.e., ColBERT and COIL) achieve state-of-the-art retrieval effectiveness by using all token embeddings to represent documents and queries andmodeling their relevance with a sum-of-max operation. However, these fine-grained representations may cause unacceptable storage overhead for practical search systems. In this study, we systematically analyze the matching mechanism of these late-interaction models and show that the sum-of-max operation heavily relies on the co-occurrence signals and some important words in the document. Based on these findings, we then propose several simple document pruning methods to reduce the storage overhead and compare the effectiveness of different pruning methods on different late-interaction models. We also leverage query pruning methods to further reduce the retrieval latency. We conduct extensive experiments on both in-domain and out-domain datasets and show that some of the used pruning methods can significantly improve the efficiency of these late-interaction models without substantially hurting their retrieval effectiveness.

关键词： Neural networks pre-trained language model late-interaction models token pruning efficiency optimization

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：