检索结果-内蒙古大学图书馆

TOWARDS GENERALIZABLE GRAPH CONTRASTIVE LEARNING: AN INFORMATION THEORY PERSPECTIVE

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Yuan, Yige Xu, Bingbing Shen, Huawei Cao, Qi Cen, Keting Zheng, Wen Cheng, Xueqi Data Intelligence System Research Center Institute of Computing Technology Chinese Academy of Sciences Beijing China CAS Key Laboratory of Network Data Science and Technology Institute of Com- puting Technology Chinese Academy of Sciences Beijing China

Graph contrastive learning (GCL) emerges as the most representative approach for graph representation learning, which leverages the principle of maximizing mutual information (InfoMax) to learn node representations applied in downstream tasks. To explore better generalization from GCL to downstream tasks, previous methods heuristically define data augmentation or pretext tasks. However, the generalization ability of GCL and its theoretical principle are still less reported. In this paper, we first propose a metric named GCL-GE for GCL generalization ability. Considering the intractability of the metric due to the agnostic downstream task, we theoretically prove a mutual information upper bound for it from an information-theoretic perspective. Guided by the bound, we design a GCL framework named InfoAdv with enhanced generalization ability, which jointly optimizes the generalization metric and InfoMax to strike the right balance between pretext task fitting and the generalization ability on downstream tasks. We empirically validate our theoretical findings on a number of representative benchmarks, and experimental results demonstrate that our model achieves state-of-the-art performance. Copyright © 2022, The Authors. All rights reserved.

关键词： Information theory

Graph wavelet neural network

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Xu, Bingbing Shen, Huawei Cao, Qi Qiu, Yunqi Cheng, Xueqi CAS Key Laboratory of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences School of Computer and Control Engineering University of Chinese Academy of Sciences Beijing China

We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcomings of previous spectral graph CNN methods that depend on graph Fourier transform. Different from graph Fourier transform, graph wavelet transform can be obtained via a fast algorithm without requiring matrix eigendecomposition with high computational cost. Moreover, graph wavelets are sparse and localized in vertex domain, offering high efficiency and good interpretability for graph convolution. The proposed GWNN significantly outperforms previous spectral graph CNNs in the task of graph-based semi-supervised classification on three benchmark datasets: Cora, Citeseer and Pubmed. Copyright © 2019, The Authors. All rights reserved.

关键词： Wavelet transforms

MQGrad: Reinforcement learning of gradient quantization in parameter server

学校读者我要写书评

暂无评论

arXiv 2018年

作者： Cui, Guoxin Xu, Jun Zeng, Wei Lan, Yanyan Guo, Jiafeng Cheng, Xueqi CAS Key Laboratory of Network Data Science and Technology Institute of ComputingTechnology Chinese Academy of Sciences Beijing100190 China

One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effective approach to reducing the communication volume. One key issue in gradient quantization is setting the number of bits for quantizing the gradients. Small number of bits can significantly reduce the communication overhead while hurts the gradient accuracies, and vise versa. An ideal quantization method would dynamically balance the communication overhead and model accuracy, through adjusting the number bits according to the knowledge learned from the immediate past training iterations. Existing methods, however, quantize the gradients either with fixed number of bits, or with predefined heuristic rules. In this paper we propose a novel adaptive quantization method within the framework of reinforcement learning. The method, referred to as MQGrad, formalizes the selection of quantization bits as actions in a Markov decision process (MDP) where the MDP states records the information collected from the past optimization iterations (e.g., the sequence of the loss function values). During the training iterations of a machine learning algorithm, MQGrad continuously updates the MDP state according to the changes of the loss function. Based on the information, MDP learns to select the optimal actions (number of bits) to quantize the gradients. Experimental results based on a benchmark dataset showed that MQGrad can accelerate the learning of a large scale deep neural network while keeping its prediction accuracies. Copyright © 2018, The Authors. All rights reserved.

关键词： Markov processes

AH-TDMA: An Adaptive Heterogeneous Tridiagonal Matrix Algorithm on the New Sunway Supercomputer 29

学校读者我要写书评

暂无评论

AH-TDMA: An Adaptive Heterogeneous Tridiagonal Matrix Algori...

29th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2023

作者： Tian, Min Liu, Qi Li, Yanlong Du, Wei Ren, Zenghui Liu, Yue Qilu University of Technology Shandong Academy of Sciences Key Laboratory of Computing Power Network and Information Security Ministry of Education Shandong Computer Science Center National Supercomputer Center in Jinan China Shandong Fundamental Research Center for Computer Science Shandong Provincial Key Laboratory of Computer Networks China Jinan Institute of Supercomputing Technology Jinan Key Laboratory of High Performance Computing China

ISBN: (纸本)9798350330717

The solution of tridiagonal linear systems is used in in various fields and plays a crucial role in numerical simulations. However, there is few efficient solver for tridiagonal linear systems on the new Sunway supercomputer. Based on a three-dimensional heat conduction problem, we propose an adaptive heterogeneous tridiagonal matrix algorithm (AH-TDMA). Our major innovations include: (1) To address computational hotspots within AH-TDMA, a multi-level parallel approach involving MPI+Athread has been adopted. (2) An adaptive data partitioning scheme has been set up to achieve load balance. (3) Employing direct memory access and establishing shared space between registers and main memory, instead of employing discrete memory access, is done to enhance memory access efficiency. (4) The optimization of the loop structure has been made in adjusting the sequencing of the dual-layered loops to reduce communication overhead. The experimental results show that, with one core group, the AH-TDMA achieves a speedup of 99.6 times for hotspot and total time speedup up to 59.4 times compared to the Parallel and Scalable Library for Tridiagonal Matrix Algorithm (PaScal TDMA). The AH-TDMA is scalable up to 2048 core groups, with a parallel efficiency of 69.2%. © 2023 IEEE.

关键词： adaptive heterogeneous algorithm MPI+Athread Sunway supercomputer three-dimensional heat conduction problem tridiagonal linear systems

hcaPCG: A Heterogeneous and communication-avoid PCG with Jacobi preconditioner on SW26010-Pro architecture 29

学校读者我要写书评

暂无评论

hcaPCG: A Heterogeneous and communication-avoid PCG with Jac...

29th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2023

作者： Tian, Min Liu, Yue Wang, Li Chen, Yanyan Liu, Qi Pan, Jingshan Qilu University of Technology Shandong Academy of Sciences Key Laboratory of Computing Power Network and Information Security Ministry of Education Shandong Computer Science Center National Supercomputer Center in Jinan China Shandong Fundamental Research Center for Computer Science Shandong Provincial Key Laboratory of Computer Networks China Jinan Institute of Supercomputing Technology Jinan Key Laboratory of High Performance Computing China

ISBN: (纸本)9798350330717

Due to its efficiency and versatility, the preconditioned conjugate gradient algorithm has long been a staple in the realm of iterative linear system solvers. In this paper, we proposed an optimized preconditioned conjugate gradient algorithm tailored for the SW26010-Pro manycore processor, The main work includes: Optimizing data block sizes based on the processor's storage structure;combining thread-level and data-level parallelism, utilizing manual SIMD to improve efficiency;optimizing the memory access pattern by employing direct memory access and overlapping of computation and communication;employing a shared-memory approach to store long vectors across all cores. Furthermore, we design an accelerated algorithm for reduction operations, to avoid data communication. Experimental results show that the hcaPCG yields up to 28.1× speedups on average compared to the original implementation. © 2023 IEEE.

关键词： Communication-avoid Heterogeneous System Jacobi preconditioner PCG algorithm SW26010-Pro

Relative Importance Sampling for off-Policy Actor-Critic in Deep Reinforcement Learning

学校读者我要写书评

暂无评论

arXiv 2018年

作者： Humayoo, Mahammad Cheng, Xueqi CAS Key Laboratory of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences Beijing100190 China University of Chinese Academy of Sciences Beijing100190 China

Off-policy learning is more unstable compared to on-policy learning in reinforcement learning (RL). One of the reasons for instability of off-policy learning is a discrepancy between target (π) and behavior (b) policy distribution. The discrepancy between π and b distribution can be alleviated by employing the smooth variant of importance sampling (IS), such as relative importance sampling (RIS).The RIS has parameter β ∈ [0, 1] that controls the smoothness. To cope with the instability of off-policy learning, we present the first relative importance sampling-off-policy actor-critic (RIS-off-PAC) model-free algorithms in RL. In our method, A network yields the target policy (actor), the value function (critic) assessing the current policy (π) using samples drawn from the behavior policy. We use action values generated from the behavior policy in reward function to train our algorithms rather than from the target policy. We also use deep neural networks to train both the actor and critic. We evaluated our algorithms on a number of OpenAI Gym problems and demonstrated better or comparable performance to several state-of-the-art RL baselines. Copyright © 2018, The Authors. All rights reserved.

关键词： Reinforcement learning

Medical Entity Disambiguation with Medical Mention Relation and Fine-grained Entity Knowledge 30

学校读者我要写书评

暂无评论

Medical Entity Disambiguation with Medical Mention Relation ...

Joint 30th International Conference on Computational Linguistics and 14th International Conference on Language Resources and Evaluation, LREC-COLING 2024

作者： Lu, Wenpeng Zhang, Guobiao Peng, Xueping Guan, Hongjiao Wang, Shoujin Key Laboratory of Computing Power Network and Information Security Ministry of Education Shandong Computer Science Center National Supercomputer Center in Jinan Qilu University of Technology Shandong Academy of Sciences Jinan China Australian Artificial Intelligence Institute University of Technology Sydney Sydney Australia Data Science Institute University of Technology Sydney Sydney Australia Shandong Provincial Key Laboratory of Computer Networks Shandong Fundamental Research Center for Computer Science Jinan China

ISBN: (纸本)9782493814104

Medical entity disambiguation (MED) plays a crucial role in natural language processing and biomedical domains, which is the task of mapping ambiguous medical mentions to structured candidate medical entities from knowledge bases (KBs). However, existing methods for MED often fail to fully utilize the knowledge within medical KBs and overlook essential interactions between medical mentions and candidate entities, resulting in knowledge- and interaction-inefficient modeling and suboptimal disambiguation performance. To address these limitations, this paper proposes a novel approach, MED with Medical Mention Relation and Fine-grained Entity Knowledge (MMR-FEK). Specifically, MMR-FEK incorporates a mention relation fusion module and an entity knowledge fusion module, followed by an interaction module. The former employs a relation graph convolutional network to fuse mention relation information between medical mentions to enhance mention representations, while the latter leverages an attention mechanism to fuse synonym and type information of candidate entities to enhance entity representations. Afterwards, an interaction module is designed to employ a bidirectional attention mechanism to capture interactions between medical mentions and entities to generate the matching representation. Extensive experiments on two publicly available real-world datasets demonstrate MMR-FEK's superiority over state-of-the-art(SOTA) MED baselines across all metrics. Our source code is publicly available. © 2024 ELRA Language Resource Association: CC BY-NC 4.0.

关键词： Natural language processing systems

Label-consistency based graph neural networks for semi-supervised node classification

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Xu, Bingbing Huang, Junjie Hou, Liang Shen, Huawei Gao, Jinhua Cheng, Xueqi CAS Key Laboratory of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences China School of Computer and Control Engineering University of Chinese Academy of Sciences China

Graph neural networks (GNNs) achieve remarkable success in graph-based semi-supervised node classification, leveraging the information from neighboring nodes to improve the representation learning of target node. The success of GNNs at node classification depends on the assumption that connected nodes tend to have the same label. However, such an assumption does not always work, limiting the performance of GNNs at node classification. In this paper, we propose label-consistency based graph neural network (LC-GNN), leveraging node pairs unconnected but with the same labels to enlarge the receptive field of nodes in GNNs. Experiments on benchmark datasets demonstrate the proposed LC-GNN outperforms traditional GNNs in graph-based semi-supervised node classification. We further show the superiority of LC-GNN in sparse scenarios with only a handful of labeled nodes. Copyright © 2020, The Authors. All rights reserved.

关键词： Graph neural networks

Parameter estimation with the ordered 2 regularization via an alternating direction method of multipliers

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Humayoo, Mahammad Cheng, Xueqi CAS Key Laboratory of Network Data Science & Technology Institute of Computing Technology Chinese Academy of Sciences Beijing 100190 China University of Chinese Academy of Sciences Beijing 100049 China

Regularization is a popular technique in machine learning for model estimation and for avoiding overfitting. Prior studies have found that modern ordered regularization can be more effective in handling highly correlated, high-dimensional data than traditional regularization. The reason stems from the fact that the ordered regularization can reject irrelevant variables and yield an accurate estimation of the parameters. How to scale up the ordered regularization problems when facing large-scale training data remains an unanswered question. This paper explores the problem of parameter estimation with the ordered 2-regularization via Alternating Direction Method of Multipliers (ADMM), called ADMM-O2. The advantages of ADMM-O2 include (i) scaling up the ordered 2 to a large-scale dataset, (ii) predicting parameters correctly by excluding irrelevant variables automatically, and (iii) having a fast convergence rate. Experimental results on both synthetic data and real data indicate that ADMM-O2 can perform better than or comparable to several state-of-the-art baselines. Copyright © 2019, The Authors. All rights reserved.

关键词： Parameter estimation