检索结果-内蒙古大学图书馆

A Teacher-Free graph knowledge distillation Framework With Dual Self-distillation

IEEE TRANSACTIONS ON knowledge AND DATA ENGINEERING 2024年第9期36卷 4375-4385页

作者： Wu, Lirong Lin, Haitao Gao, Zhangyang Zhao, Guojiang Li, Stan Z. Westlake Univ Res Ctr Ind Future AI Lab Hangzhou 310000 Peoples R China

Recent years have witnessed great success in handling graph-related tasks with graph Neural Networks (GNNs). Despite their great academic success, Multi-Layer Perceptrons (MLPs) remain the primary workhorse for practical industrial applications. One reason for such an academic-industry gap is the neighborhood-fetching latency incurred by data dependency in GNNs. To reduce their gaps, graph knowledge distillation (GKD) is proposed, usually based on a standard teacher-student architecture, to distill knowledge from a large teacher GNN into a lightweight student GNN or MLP. However, we found in this paper that neither teachers nor GNNs are necessary for graph knowledge distillation. We propose a Teacher-Free graph Self-distillation (TGS) framework that does not require any teacher model or GNNs during both training and inference. More importantly, the proposed TGS framework is purely based on MLPs, where structural information is only implicitly used to guide dual knowledge self-distillation between the target node and its neighborhood. As a result, TGS enjoys the benefits of graph topology awareness in training but is free from data dependency in inference. Extensive experiments have shown that the performance of vanilla MLPs can be greatly improved with dual self-distillation, e.g., TGS improves over vanilla MLPs by 15.54% on average and outperforms state-of-the-art GKD algorithms on six real-world datasets. In terms of inference speed, TGS infers 75x-89x faster than existing GNNs and 16x-25x faster than classical inference acceleration methods.

关键词： graph neural networks Training Self-supervised learning Inference algorithms Task analysis Standards knowledge engineering graph knowledge distillation inference acceleration

来源：评论

学校读者我要写书评

暂无评论

Learning structure perception MLPs on graphs: a layer-wise graph knowledge distillation framework

引用

INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS 2024年第10期15卷 4357-4372页

作者： Du, Hangyuan Yu, Rong Bai, Liang Bai, Lu Wang, Wenjian Shanxi Univ Sch Comp & Informat Technol Taiyuan 030006 Shanxi Peoples R China Shanxi Univ Key Lab Computat Intelligence Chinese Informat Pro Minist Educ Taiyuan 030006 Shanxi Peoples R China Shanxi Univ Inst Intelligent Informat Proc Taiyuan 030006 Shanxi Peoples R China Beijing Normal Univ Sch Artificial Intelligence Beijing 100875 Peoples R China Cent Univ Finance & Econ Beijing 100875 Peoples R China

graph neural networks (GNNs) are expressive in dealing with graph data. Because of the large storage requirements and the high computational complexity, it is difficult to deploy these cumbersome models in resource-constrained environments. As a representative model compression strategy, knowledge distillation (KD) is introduced into graph analysis research to address this problem. However, there are some crucial challenges in existing graph knowledge distillation algorithms, such as knowledge transfer effectiveness and student model designation. To address these problems, a new graph distillation model is proposed in this paper. Specifically, a layer-wise mapping strategy is designed to distill knowledge for training the student model, in which staged knowledge learned by intermediate layers of teacher GNNs is captured to form supervision signals. And, an adaptive weight mechanism is developed to evaluate the importance of the distilled knowledge. On this basis, a structure perception MLPs is constructed as the student model, which can capture prior information of the input graph from the perspectives of node feature and topology structure. In this way, the proposed model shares the prediction advantage of GNNs and the latency advantage of MLPs. Node classification experiments on five benchmark datasets demonstrate the validity and superiority of our model over baseline algorithms.

关键词： graph knowledge distillation Supervision signal Layer-wise mapping Structure perception MLPs

来源：评论

学校读者我要写书评

暂无评论

Class-view graph knowledge distillation: A new idea for learning MLPs on graphs

引用

NEUROCOMPUTING 2025年 637卷

作者： Tian, Yingjie Xu, Shaokai Li, Muyang Univ Chinese Acad Sci Sch Econ & Management Beijing 100190 Peoples R China Univ Chinese Acad Sci Sch Comp Sci & Technol Beijing 100049 Peoples R China Chinese Acad Sci Res Ctr Fictitious Econ & Data Sci Beijing 100190 Peoples R China Univ Chinese Acad Sci Key Lab Big Data Min & Knowledge Management Beijing 100190 Peoples R China

graph Neural Networks (GNNs), while effective for processing non-Euclidean structured data, suffer from computationally intensive neighbor fetching, which hinders their deployment in low-latency applications. Cross-architecture graph knowledge distillation (KD), which trains high-performance Multi-layer Perceptrons (MLPs) to emulate teacher GNNs, offers a promising solution. However, existing GNN-MLP distillation methods rely on the sample-view KD paradigm, where student models directly mimic the teacher's parameter space. Given the fundamentally different architectures and parameter spaces of GNNs and MLPs, this direct mimicry approach limits effective knowledge transfer. Inspired by the inherent properties of MLPs, we propose a novel class-view KD paradigm for GNN-MLP distillation. Unlike sample-view KD, our method guides student MLPs to generate more discriminative parameter configurations within their own parameter space while preserving the similarity of prediction distributions with the teacher, rather than directly imitating the teacher's parameter configurations. Extensive experiments on public benchmark datasets demonstrate that class-view KD outperforms sample-view KD across various evaluation metrics and can be seamlessly integrated into existing GNN-MLP distillation methods to improve performance without additional computational cost. The code is available at https://***/xsk160/Class-View-graph-KD.

关键词： graph knowledge distillation Cross-architecture Class-view

来源：评论

学校读者我要写书评

暂无评论

Decoupled graph knowledge distillation: A general logits-based method for learning MLPs on graphs

引用

NEURAL NETWORKS 2024年 179卷 106567页

While graph Neural Networks (GNNs) have demonstrated their effectiveness in processing non-Euclidean structured data, the neighborhood fetching of GNNs is time-consuming and computationally intensive, making them difficult to deploy in low-latency industrial applications. To address the issue, a feasible solution is graph knowledge distillation (KD), which can learn high-performance student Multi-layer Perceptrons (MLPs) to replace GNNs by mimicking the superior output of teacher GNNs. However, state-of-the-art graph knowledge distillation methods are mainly based on distilling deep features from intermediate hidden layers, this leads to the significance of logit layer distillation being greatly overlooked. To provide a novel viewpoint for studying logits-based KD methods, we introduce the idea of decoupling into graph knowledge distillation. Specifically, we first reformulate the classical graph knowledge distillation loss into two parts, i.e., the target class graph distillation (TCGD) loss and the non-target class graph distillation (NCGD) loss. Next, we decouple the negative correlation between GNN's prediction confidence and NCGD loss, as well as eliminate the fixed weight between TCGD and NCGD. We named this logits-based method Decoupled graph knowledge distillation (DGKD). It can flexibly adjust the weights of TCGD and NCGD for different data samples, thereby improving the prediction accuracy of the student MLP. Extensive experiments conducted on public benchmark datasets show the effectiveness of our method. Additionally, DGKD can be incorporated into any existing graph knowledge distillation framework as a plug-and-play loss function, further improving distillation performance. The code is available at https://***/xsk160/DGKD.

关键词： graph knowledge distillation Decoupling graph neural networks Multi-layer perceptrons

来源：评论

学校读者我要写书评

暂无评论

Multi-Level knowledge distillation with Positional Encoding Enhancement

引用

PATTERN RECOGNITION 2025年 163卷

作者： Xu, Lixiang Wang, Zhiwen Bai, Lu Ji, Shengwei Ai, Bing Wang, Xiaofeng Yu, Philip S. Hefei Univ Sch Artificial Intelligence & Big Data Hefei Anhui Peoples R China Beijing Normal Univ Sch Artificial Intelligence Beijing Peoples R China Univ Illinois Dept Comp Sci Chicago IL USA

In recent years, graph Neural Networks (GNNs) have achieved substantial success in addressing graph-related tasks. knowledge distillation (KD) has increasingly been adopted in graph learning as a classical technique for model compression and acceleration, enabling the transfer of predictive power from trained GNN models to lightweight, easily deployable Multi-Layer Perceptron (MLP) models. However, this approach often neglects node positional features and relies solely on trained GNN-generated labels to train MLPs based on node content features. Moreover, it heavily depends on local information aggregation, making it challenging to capture global graph structure and thereby limiting performance in node classification tasks. To address this issue, we propose Multi-Level knowledge distillation with Positional Encoding Enhancement (MLKD-PE). Our method employs positional encoding technique to generate node positional features, which are then combined with node content features to enhance the MLP's ability to perceive node positions. Additionally, we introduce a multi-level KD technique that aligns the final output of the student model with the teacher model's output, facilitating detailed knowledge transfer by incorporating intermediate layer outputs from the teacher model. Experimental results demonstrate that our method significantly improves classification accuracy across multiple datasets compared to the baseline model, confirming its superiority in node classification tasks.

关键词： graph neural networks Multi-layer perceptrons graph knowledge distillation Positional encoding Efficient training and inference

来源：评论

学校读者我要写书评

暂无评论

PROSPECT: Learn MLPs on graphs Robust against Adversarial Structure Attacks 24

PROSPECT: Learn MLPs on Graphs Robust against Adversarial St...

引用

33rd ACM International Conference on Information and knowledge Management (CIKM)

作者： Deng, Bowen Chen, Jialong Hu, Yanming Xu, Zhiyong Chen, Chuan Zhang, Tao Sun Yat Sen Univ Guangzhou Peoples R China

ISBN: (纸本)9798400704369

Current adversarial defense methods for GNNs exhibit critical limitations obstructing real-world application: 1) inadequate adaptability to graph heterophily, 2) absent generalizability to early GNNs like graphSAGE used downstream, and 3) low inference scalability unacceptable for resource-constrained scenarios. To simultaneously address these challenges, we propose the first online GNN-MLP distillation framework PROSPECT, which merges the complementary knowledge of MLP and GNN and can thus learn GNN and MLP robust against adversarial structure attacks on both homophilic and heterophilic graphs. PROSPECT integrates seamlessly into graphSAGE and achieves inference scalability exponentially higher than conventional GNNs. To mitigate potential convergence failure caused by inductive bias conflicts between the heterogeneous MLP and GNN, we propose the Quasi-Alternating Cosine Annealing (QACA) learning rate scheduler, inspired by our convergence analysis of the involved MLP. Experiments on homophilic and heterophilic graphs demonstrate the advantages of PROSPECT over current defenses and offline GNN-MLP distillation methods in terms of adversarial robustness and clean accuracy, the inference scalability of PROSPECT orders of magnitude higher than existing defenses, and the effectiveness of QACA.

关键词： Adversarial machine learning graph neural network graph knowledge distillation graph heterophily

来源：评论

学校读者我要写书评

暂无评论

Distill graph Structure knowledge from Masked graph Autoencoders into MLP 27

Distill Graph Structure Knowledge from Masked Graph Autoenco...

引用

27th International Conference on Computer Supported Cooperative Work in Design (CSCWD)

作者： Zhang, Xiong Xie, Cheng Yunnan Univ Software Coll Kunming Yunnan Peoples R China

ISBN: (纸本)9798350349184;9798350349191

In recent years, graph neural networks(GNNs) have been increasingly used in collaborative computing applications such as recommender systems and social networks. GNNs perform message passing, aggregating local neighbors to update node representations, which leads to scalability issues in applications, and they are difficult to deploy in real-world applications. To address this problem, recent approaches use knowledge distillation(KD) to learn to compute efficient multilayer perceptron(MLP) by mimicking the output of GNN. However, existing GNN knowledge distillation methods require an input of specific node location features to capture the structural information of the graph, which limits the knowledge migration from GNN to MLP. Here, we propose a novel framework, GMAE-MLP, that uses representational similarity to bridge the powerful graph representation space of GNN and MLP. We employ the graph mask autoencoder (GMAE) as a structure-aware graph encoder, which is able to represent the local structure of reconstructed mask nodes as meaningful structural features. We then propose a feature similarity-based distillation approach to transfer structural knowledge from GNNs to MLPs adequately. Extensive experiments and analyses demonstrate the superiority of GMAE-MLP, and We achieved state-of-the-art performance for GNN-MLP distillation in four of the five graphical datasets in the transductive setup. We show that GMAE-MLP improves GNNs by an average of 2.95%, MLPs by 26.03%, and the state-of-the-art NOSMOG by 0.95% across 5 datasets in the transductive setting.

关键词： graph knowledge distillation Collaborative Computing graph Representation Learning Node Classification

来源：评论

学校读者我要写书评

暂无评论

FleX: Interpreting graph Neural Networks with Subgraph Extraction and Flexible Objective Estimation 30th

FleX: Interpreting Graph Neural Networks with Subgraph Extr...

引用

30th International Conference on Cooperative Information Systems, CoopIS 2024

作者： Nguyen, Duy Le, Thanh Le, Bac Faculty of Information Technology University of Science Ho Chi Minh City Viet Nam Vietnam National University Ho Chi Minh City Viet Nam

ISBN: (纸本)9783031813740

graph Neural Networks (GNNs) have shown remarkable results in graph-related tasks, yet interpreting their decision-making process remains challenging. Most existing methods for interpreting GNNs focus on finding a subgraph that preserves the model’s predictions. However, removing edges alters the original structure, making the optimization process heavily dependent on the loss function. In this paper, we propose FleX, a novel approach that transcends these limitations by using a distillation model to estimate prediction values after subgraph extraction. Our method combines implicit and explicit edge masking techniques to identify the most relevant subgraph. We introduce a flexible loss estimation strategy that allows for a more robust optimization process. Experimental results demonstrate that FleX outperforms most existing graph XAI models across various benchmark datasets, achieving superior performance in interpreting GNNs. This approach enhances the interpretability of GNNs while maintaining high accuracy, contributing to more trustworthy and explainable graph-based machine learning models. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

关键词： graph explainability graph knowledge distillation graph neural network Subgraph extraction

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：