检索结果-内蒙古大学图书馆

ST-PINN: A Self-Training Physics-Informed Neural Network for Partial Differential Equations

学校读者我要写书评

暂无评论

ST-PINN: A Self-Training Physics-Informed Neural Network for...

International Joint Conference on Neural Networks (IJCNN)

作者： Junjun Yan Xinhai Chen Zhichao Wang Enqiang Zhoui Jie Liu Science and Technology on Parallel and Distributed Processing Laboratory National University of Defense Technology Changsha China Laboratory of Digitizing Software for Frontier Equipment National University of Defense Technology Changsha China

Partial differential equations (PDEs) are an essential computational kernel in physics and engineering. With the advance of deep learning, physics-informed neural networks (PINNs), as a mesh-free method, have shown great potential for fast PDE solving in various applications. To address the issue of low accuracy and convergence problems of existing PINNs, we propose a self-training physics-informed neural network, ST-PINN. Specifically, ST-PINN introduces a pseudo label based self-learning algorithm during training. It employs governing equation as the pseudo-labeled evaluation index and selects the highest confidence examples from the sample points to attach the pseudo labels. To our best knowledge, we are the first to incorporate a self-training mechanism into physics-informed learning. We conduct experiments on five PDE problems in different fields and scenarios. The results demonstrate that the proposed method allows the network to learn more physical information and benefit convergence. The ST-PINN outperforms existing physics-informed neural network methods and improves the accuracy by a factor of 1.33x-2.54x.

关键词：

parallel Implementation of SHA256 on Multizone Heterogeneous Systems

学校读者我要写书评

暂无评论

Parallel Implementation of SHA256 on Multizone Heterogeneous...

IEEE International Conference on Big Data and Cloud Computing (BdCloud)

作者： Yongtao Luo Jie Liu Tiaojie Xiao Chunye Gong Science and Technology on Parallel and Distributed Processing Laboratory Laboratory of Digitizing program for Frontier Equipment National University of Defense Technology Changsha China National Supercomputer Center in Tianjin Tianjin China

SHA-256 plays an important role in widely used applications, such as data security, data integrity, digital signatures, and cryptocurrencies. However, most of the current optimized implementations of SHA-256 are based on CPUs or dedicated hardware, such as ASICs and FPGAs. Consequently, there is a need to explore whether new heterogeneous parallel framework can improve the computational performance of the hash function. To address this issue, we conducted a study on the MT-3000 platform, which is a special architecture processor for the next-generation exascale prototype supercomputer. We proposed MT-SHA256, a heterogeneous multistage parallel implementation for hashing multiple messages on the MT-3000. Combining the architectural features of this processor, we developed an effective solution that significantly improved the computational performance of SHA-256. As a result, MT-SHA256 achieved a maximum throughput of 1045.68 MB/s on a single acceleration core of MT-3000. This is 9.84x higher than the C code implementation on one CPU core of MT-3000. We also performed a scalability test and found that MT-SHA256 achieved a throughput of 98.04 GB/s on a computing node, and extended to 512 nodes (2048 acceleration clusters) on this system with good scalability.

关键词：

XGrad: Boosting Gradient-Based Optimizers With Weight Prediction

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Guan, Lei Li, Dongsheng Shi, Yanqi Meng, Jian The Department of Mathematics National University of Defense Technology Changsha China The National Key Laboratory of Parallel and Distributed Computing National University of Defense Technology China

In this paper, we propose a general deep learning training framework XGrad which introduces weight prediction into the popular gradient-based optimizers to boost their convergence and generalization when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, the future weights are predicted according to the update rule of the used optimizer and are then applied to both the forward pass and backward propagation. In this way, during the whole training period, the optimizer always utilizes the gradients w.r.t. the future weights to update the DNN parameters, making the gradient-based optimizer achieve better convergence and generalization compared to the original optimizer without weight prediction. XGrad is rather straightforward to implement yet pretty effective in boosting the convergence of gradient-based optimizers and the accuracy of DNN models. Empirical results concerning five popular optimizers including SGD with momentum, Adam, AdamW, AdaBelief, and AdaM3 demonstrate the effectiveness of our proposal. The experimental results validate that XGrad can attain higher model accuracy than the baseline optimizers when training the DNN models. The code of XGrad will be available at: https://***/guanleics/XGrad. Copyright © 2023, The Authors. All rights reserved.

关键词： Forecasting

A Counterfactual Ultrasound Anti-Interference Self-Supervised Network for B-mode Ultrasound Tongue Extraction

学校读者我要写书评

暂无评论

A Counterfactual Ultrasound Anti-Interference Self-Supervise...

International Conference on Acoustics, Speech, and Signal processing (ICASSP)

作者： Yan Jia Yuqing Cheng Kele Xu Yong Dou Peng Qiao Zhouyu He National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology Changsha China College of Systems Engineering National University of Defense Technology Changsha China

ISBN: (数字)9798350368741

ISBN: (纸本)9798350368758

B-mode ultrasound tongue imaging is a non-invasive and real-time method for visualizing vocal tract deformation. However, accurately extracting the tongue’s surface contour remains a significant challenge due to the low signal-to-noise ratio (SNR) and prevalent speckle noise in ultrasound images. Traditional supervised learning models often require large labeled datasets, which are labor-intensive to produce and susceptible to noise interference. To address these limitations, we present a novel Counterfactual Ultrasound Anti-Interference Self-Supervised Network (CUAI-SSN), which integrates self-supervised learning (SSL) with counterfactual data augmentation, progressively disentangles confounding factors, ensuring that the model generalizes well across varied ultrasound conditions. Our approach leverages causal reasoning to decouple noise from relevant features, enabling the model to learn robust representations that focus on essential tongue structures. By generating counterfactual image-label pairs, our method introduces alternative, noise-independent scenarios that enhance model training. Furthermore, we introduce attention mechanisms to enhance the network’s ability to capture fine-grained details even in noisy conditions. Extensive experiments on real ultrasound tongue images demonstrate that CUAI-SSN outperforms existing methods, setting a new benchmark for automated contour extraction in ultrasound tongue imaging. Our code is publicly available at https://***/inexhaustible419/CounterfactualultrasoundAI.

关键词： Training Ultrasonic imaging Tongue Self-supervised learning Data augmentation Data models Cognition Data mining Noise measurement Signal to noise ratio

ACCURATE AND EFFICIENT FINE-TUNING OF QUANTIZED LARGE LANGUAGE MODELS THROUGH OPTIMAL BALANCE

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Shen, Ao Wang, Qiang Lai, Zhiquan Li, Xionglve Li, Dongsheng National Key Laboratory of Parallel and Distributed Computing National University of Defense Technology Hunan Changsha410073 China College of computer National University of Defense Technology Hunan Changsha410073 China

Large Language Models (LLMs) have demonstrated impressive performance across various domains. However, the enormous number of model parameters makes fine-tuning challenging, significantly limiting their application and deployment. Existing solutions combine parameter quantization with Low-Rank Adaptation (LoRA), greatly reducing memory usage but resulting in noticeable performance degradation. In this paper, we identify an imbalance in fine-tuning quantized pre-trained models: overly complex adapter inputs and outputs versus low effective trainability of the adaptation. We propose Quantized LLMs with Balanced-rank Adaptation (Q-BaRA), which simplifies the adapter inputs and outputs while increasing the adapter’s rank to achieve a more suitable balance for finetuning quantized LLMs. Additionally, for scenarios where fine-tuned LLMs need to be deployed as low-precision inference models, we introduce Quantization-Aware Fine-tuning with Higher Rank Adaptation (QA-HiRA), which simplifies the adapter inputs and outputs to align with the pre-trained model’s block-wise quantization while employing a single matrix to achieve a higher rank. Both Q-BaRA and QA-HiRA are easily implemented and offer the following optimizations: (i) Q-BaRA consistently achieves the highest accuracy compared to baselines and other variants, requiring the same number of trainable parameters and computational effort;(ii) QA-HiRA naturally merges adapter parameters into the block-wise quantized model after fine-tuning, achieving the highest accuracy compared to other methods. We apply our Q-BaRA and QA-HiRA to the LLaMA and LLaMA2 model families and validate their effectiveness across different fine-tuning datasets and downstream scenarios. Copyright © 2024, The Authors. All rights reserved.

关键词： Digital elevation model

Merak: An Efficient distributed DNN Training Framework with Automated 3D parallelism for Giant Foundation Models

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Lai, Zhiquan Li, Shengwei Tang, Xudong Ge, Keshi Liu, Weijie Duan, Yabo Qiao, Linbo Li, Dongsheng The National Laboratory for Parallel and Distributed Processing College of Computer National University of Defense Technology in Changsha Hunan China

Foundation models are in the process of becoming the dominant deep learning technology. Pretraining a foundation model is always time-consuming due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the pretraining process is extremely memory- and communication-intensive. These challenges make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism, and tensor model parallelism, to achieve high training efficiency. However, current 3D parallelism frameworks still encounter two issues: i) they are not transparent to model developers, requiring manual model modification to parallelize training, and ii) their utilization of computation resources, GPU memory, and network bandwidth is insufficient. We propose Merak, an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys 3D parallelism with an automatic model partitioner, which includes a graph-sharding algorithm and proxy node-based model graph. Merak also offers a non-intrusive API to scale out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine that employs several techniques to exploit available training resources, including a shifted critical path pipeline schedule that increases computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs demonstrate Merak's capability to speed up training performance over state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42, 1.39, 1.43, and 1.61×, respectively. The code for Merak has been open-sourced at https://***/hpdl-group/Merak. Copyright © 2022, The Authors. All rights reserved.

关键词： Pipelines

Simultaneously Learning Syntactic Dependency and Semantics Reasonability for Relation Extraction

学校读者我要写书评

暂无评论

Simultaneously Learning Syntactic Dependency and Semantics R...

International Conference on Image, Vision and Intelligent Systems, ICIVIS 2021

作者： Wang, Xin Yin, Nan Zhang, Xiang Bai, Xinyi Luo, Zhigang College of Computer National University of Defense Technology Changsha China Institute for Quantum and State Key Laboratory of High Performance Computing National University of Defense Technology Changsha China Science and Technology on Parallel and Distributed Laboratory National University of Defense Technology Changsha China

ISBN: (纸本)9789811669620

Relation extraction as an important Natural Language processing (NLP) task is to identify relations between named entities in text. Recently, graph convolutional networks over dependency trees have been widely used to capture syntactic features and achieved attractive performance. However, most existing dependency-based approaches ignore the positive influence of the words outside the dependency trees, sometimes conveying rich and useful information on relation extraction. In this paper, we propose a novel model, Entity-aware Self-attention Contextualized GCN (ESC-GCN), which efficiently incorporates syntactic structure of input sentences and semantic context of sequences. To be specific, relative position self-attention obtains the overall semantic pairwise correlation related to word position, and contextualized graph convolutional networks capture rich intra-sentence dependencies between words by adequately pruning operations. In this way, our proposed model not only reduces the noisy impact from dependency trees but also obtains easily-ignored entity-related semantic representation. Extensive experiments demonstrate that our model achieves encouraging performance. © 2022, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

关键词： Semantics

KC-GenRe: A Knowledge-constrained Generative Re-ranking Method Based on Large Language Models for Knowledge Graph Completion

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Yilin Hu, Minghao Huang, Zhen Li, Dongsheng Yang, Dong Lu, Xicheng Defense Innovation Institute Academy of Military Sciences China Information Research Center of Military Science China National Key Laboratory of Parallel and Distributed Computing China

The goal of knowledge graph completion (KGC) is to predict missing facts among entities. Previous methods for KGC re-ranking are mostly built on non-generative language models to obtain the probability of each candidate. Recently, generative large language models (LLMs) have shown outstanding performance on several tasks such as information extraction and dialog systems. Leveraging them for KGC re-ranking is beneficial for leveraging the extensive pre-trained knowledge and powerful generative capabilities. However, it may encounter new problems when accomplishing the task, namely mismatch, misordering and omission. To this end, we introduce KC-GenRe, a knowledge-constrained generative re-ranking method based on LLMs for KGC. To overcome the mismatch issue, we formulate the KGC re-ranking task as a candidate identifier sorting generation problem implemented by generative LLMs. To tackle the misordering issue, we develop a knowledge-guided interactive training method that enhances the identification and ranking of candidates. To address the omission issue, we design a knowledge-augmented constrained inference method that enables contextual prompting and controlled generation, so as to obtain valid rankings. Experimental results show that KG-GenRe achieves state-of-the-art performance on four datasets, with gains of up to 6.7% and 7.7% in the MRR and Hits@1 metric compared to previous methods, and 9.0% and 11.1% compared to that without re-ranking. Extensive analysis demonstrates the effectiveness of components in KG-GenRe. Copyright © 2024, The Authors. All rights reserved.

关键词： Knowledge graph

Graph Structure Learning via Transfer Entropy for Multivariate Time Series Anomaly Detection

学校读者我要写书评

暂无评论

Graph Structure Learning via Transfer Entropy for Multivaria...

International Conference on Acoustics, Speech, and Signal processing (ICASSP)

作者： Mingyu Liu Yijie Wang Xiaohui Zhou Yongjun Wang National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology Changsha China College of Computer Science and Technology National University of Defense Technology Changsha China

ISBN: (数字)9798350368741

ISBN: (纸本)9798350368758

Multivariate time series anomaly detection (MTAD) poses a challenge due to temporal and feature dependencies. The critical aspects of enhancing the detection performance lie in accurately capturing the dependencies between variables within the sliding window and effectively leveraging them. Existing studies rely on domain knowledge to pre-set the window size, and overlook the strength of dependencies while calculating direction based on variable similarity. This paper proposes GSLTE, a graph structure learning method for MTAD. GSLTE employs Fast Fourier Transform to conduct iterative segmentation of the whole series, selecting the dominant Fourier frequency as the window size for each subsequence within the minimum interval. GSLTE quantifies the direction and strength of the dependencies based on variable-lag transfer entropy which is achieved through Dynamic Time Warping method to learn asymmetric links between variables. Extensive experiments show that GNN-based MTAD methods applying GSLTE can further improve anomaly detection performance while outperforming state-of-the-art competitors.

关键词： Learning systems Time-frequency analysis Fast Fourier transforms Time series analysis Signal processing Feature extraction Entropy Iterative methods Speech processing Anomaly detection