检索结果-内蒙古大学图书馆

29th International Conference on database Systems for Advanced Applications (DASFAA)

作者： Zeng, Xinfeng Liu, Tao Zeng, Ming Wu, Qingqiang Wang, Meihong Xiamen Univ Sch Informat Xiamen Peoples R China Xiamen Univ Sch Film Xiamen Peoples R China

ISBN: (纸本)9789819757787;9789819757794

data-free knowledge distillation (KD) circumvents the limitation of knowledge extration from original training data by utilizing generated data. data-free KD has made good progress in models for processing grid data. However, for Graph Neural Networks (GNN) that process non-grid data, existing related methods are primarily designed for graph classification tasks involving small-sized graphs. Research on node classification tasks for larger graphs remains unexplored. To address this issue, we propose the first data-free knowledge distillation framework for Node Classification (DFKD-NC). DFKD-NC obtains training data by generating target node and its neighbor information. For data generation, we predefine a full tree pseudo-subgraph template for each target node and use a generator component to generate node features. We train student model and generator using adversarial training. Moreover, we introduce contrastive learning to diversify the generated data and regularize the generated node features to facilitate student model convergence. Extensive experiments on six benchmark datasets have demonstrated that DFKD-NC achieves state-of-the-art results in node classification tasks. Code is available at https://***/zengxinfeng/DFKD-NC.

关键词： Graph Neural Networks data-free knowledge distillation Node Classification

来源：评论

学校读者我要写书评

暂无评论

Effective and efficient conditional contrast for data-free knowledge distillation with low memory

引用

JOURNAL OF SUPERCOMPUTING 2025年第4期81卷 1-21页

作者： Jiang, Chenyang Li, Zhendong Yang, Jun Wu, Yiqiang Li, Shuai Ningxia Univ Sch Informat Engn 489 Helan Mt West Rd Yinchuan 750021 Ningxia Peoples R China Key Lab Internet Water & Digital Water Governance Yinchuan 750021 Ningxia Peoples R China

data-free knowledge distillation has recently gained significant attention in the field of model compression, as it enables knowledge transfer from a trained teacher model to a smaller student model without requiring original training data. Current methods often utilize generative adversarial networks (GANs) to synthesize fake samples, but this approach introduces two main issues. First, mode collapse leads to instances lacking diversity for downstream tasks. Second, inefficient instance synthesis makes existing methods too time-consuming and thus difficult to adapt to large-scale datasets. Finally, the increased memory footprint makes deployment difficult. In this paper, we propose a novel paradigm called conditional contrast for data-free knowledge distillation (CC-DFKD), which integrates conditional generative adversarial network (CGAN) and contrastive learning. CGAN synthesizes class-specific diverse images to address the diversity challenge, while contrastive learning enriches the student model's feature representations to tackle the reality challenge. Additionally, compared with the recent work, simplification of the distillation loss reduces instance generation time and memory usage during operation, achieving significant speed improvements (half an hour to nine hours reduction) and lower GPU memory usage (2000-5000 MB reduction). Empirical results across multiple datasets validate CC-DFKD's effectiveness and efficiency under low-memory conditions. Code is available at: https://***/jcynxu/CC-DFKD.

关键词： Model compression data-free knowledge distillation Contrastive learning Conditional adversarial Generative networks

来源：评论

学校读者我要写书评

暂无评论

data-free knowledge distillation via Contrastive Inversion and Cluster Alignment

Data-Free Knowledge Distillation via Contrastive Inversion a...

引用

2025 International Conference on Electrical Automation and Artificial Intelligence, ICEAAI 2025

作者： Zheng, Zhaoming Du, Zhibin School of Artificial Intelligence South China Normal University Foshan China

ISBN: (纸本)9798331506797

data-free knowledge distillation (DFKD) aims to transfer the knowledge from a large teacher network to a lightweight student network without accessing the original training data. However, existing DFKD methods face the issue of distribution shift during data synthesis, which leads to a decline in knowledge distillation. To address this issue, we propose a data-free knowledge distillation method (CCD) based on contrastive inversion (CI) and cluster alignment (CA). Specifically, CI enables the generative network to better capture relevant features and efficiently synthesize images that conforms to the original training data. However, using CI alone can severely reduce the diversity of synthesis images, which may lead to poor performance of the student network. Thus, we propose CA to assure the generative network can learn not only intra-class feature but also the inter-class feature to alleviate the diversity issue. The experiments on various dataset show that our proposed method outperform other baseline methods. © 2025 IEEE.

关键词： Clustering learning Contrast learning data-free knowledge distillation Image classification

来源：评论

学校读者我要写书评

暂无评论

Unpacking the Gap Box Against data-free knowledge distillation

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024年第9期46卷 6280-6291页

作者： Wang, Yang Qian, Biao Liu, Haipeng Rui, Yong Wang, Meng Hefei Univ Technol Sch Comp Sci & Informat Engn Hefei 230002 Anhui Peoples R China Lenovo Res Beijing 100094 Peoples R China

data-free knowledge distillation (DFKD) improves the student model (S) by mimicking the class probability from apre-trained teacher model (T) without training data. Under such setting, an ideal scenario is that T can help generate "good" samples from a generator (G) to maximally benefit S. However, existing arts suffer from the non-ideal generated samples under the disturbance of the gap (i.e., either too large or small) between the class probabilities of T and S;for example, the generated samples with too large gap may exhibit excessive information for S, while too small gap leads to the limited knowledge in the samples, resulting into the poor generalization. Meanwhile, they fail to judge the "goodness" of the generated samples for S since the fixed T is not necessarily ideal. In this paper, we aim to answer what is inside the gap box;together with how to yield "good" generated samples for DFKD? To this end, we propose a Gap-Sensitive Sample Generation (GapSSG)approach, by revisiting the empirical distilled risk from a data-free perspective, which confirms the existence of an ideal teacher (T & lowast;),while theoretically implying: (1) the gap disturbance originates from the mismatch between T and T & lowast;, hence the class probabilities of T enable the approximation to those of T & lowast;and (2) "good" samples should maximally benefit S via T's class probabilities, owing to unknown T & lowast;. To this end, we unpack the gap box between T and S as two findings: in heren t gap to perceive T and T & lowast;derived gap to monitor S and T & lowast;. Benefiting from the derived gap that focuses on the adaptability of generated sample to S, we attempt to track student's training route (a series of training epochs) to capture the category distribution of S;upon which, a regulatory factor is fur-the r devised to approximate T & lowast;over in here n t gap, so as to generate "good" samples to S. Furthermore, during the distillation process, a sample-balanced strategy

关键词： Training Art data models Analytical models knowledge engineering Generators Three-dimensional displays data-free knowledge distillation derived gap empirical distilled risk generative model inherent gap

来源：评论

学校读者我要写书评

暂无评论

FedAlign: Federated Model Alignment via data-free knowledge distillation for Machine Fault Diagnosis

引用

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 2024年 73卷 1页

作者： Sun, Wenjun Yan, Ruqiang Jin, Ruibing Zhao, Rui Chen, Zhenghua Southeast Univ Sch Instrument Sci & Engn Nanjing 210096 Jiangsu Peoples R China Xi An Jiao Tong Univ Sch Mech Engn Xian 710049 Shaanxi Peoples R China ASTAR Inst Infocomm Res Singapore 138632 Singapore Pluang Tech Singapore 049145 Singapore

Due to privacy issues, the data island problem of machine fault diagnosis widely exists in real industry. Federated learning (FL) has received much attention as a decentralized machine-learning paradigm that learns a global model in the server by iteratively aggregating the local model parameters in a privacy-preserving scheme to address the data island problem. However, the fault data in industry is inevitably scarce and class imbalanced, so the fault data in different clients are heterogeneous from each other in FL. The existing federated machine fault diagnosis methods are mostly based on the federated averaging (FedAvg) algorithm, which ignores the data heterogeneity issue for machine fault diagnosis. To address this issue for machine fault diagnosis, we propose a new model alignment method (called FedAlign) via data-free knowledge distillation, where a compact generator is trained in the server in a data-free manner to estimate the data feature distribution in a global view without access to the local data. Then, the generator can produce the pseudo features to convey the distribution knowledge to both the server and the client sides. Our proposed FedAlign utilizes the pseudo features to align the global model and the local models to finally address the inherent heterogeneity of local data for fault diagnosis in FL. Experiments performed on fault diagnosis datasets of nonidentically and independently (non-iid) settings indicate that our proposed method facilitates FL for machine fault diagnosis with favorable effectiveness and achieves significant performance gains compared with state-of-the-art methods.

关键词： Fault diagnosis data models Servers Federated learning Generators Industries Companies data heterogeneity data-free knowledge distillation fault diagnosis federated learning (FL) generator

来源：评论

学校读者我要写书评

暂无评论

Variational data-free knowledge distillation for Continual Learning

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023年第10期45卷 12618-12634页

作者： Li, Xiaorong Wang, Shipeng Sun, Jian Xu, Zongben Xi An Jiao Tong Univ Sch Math & Stat Xian 710049 Shaanxi Peoples R China

Deep neural networks suffer from catastrophic forgetting when trained on sequential tasks in continual learning. Various methods rely on storing data of previous tasks to mitigate catastrophic forgetting, which is prohibited in real-world applications considering privacy and security issues. In this paper, we consider a realistic setting of continual learning, where training data of previous tasks are unavailable and memory resources are limited. We contribute a novel knowledge distillation-based method in an information-theoretic framework by maximizing mutual information between outputs of previously learned and current networks. Due to the intractability of computation of mutual information, we instead maximize its variational lower bound, where the covariance of variational distribution is modeled by a graph convolutional network. The inaccessibility of data of previous tasks is tackled by Taylor expansion, yielding a novel regularizer in network training loss for continual learning. The regularizer relies on compressed gradients of network parameters. It avoids storing previous task data and previously learned networks. Additionally, we employ self-supervised learning technique for learning effective features, which improves the performance of continual learning. We conduct extensive experiments including image classification and semantic segmentation, and the results show that our method achieves state-of-the-art performance on continual learning benchmarks.

关键词： Catastrophic forgetting continual learning data-free knowledge distillation mutual information

来源：评论

学校读者我要写书评

暂无评论

data-free knowledge distillation via generator-free data generation for Non-IID federated learning

引用

NEURAL NETWORKS 2024年 179卷 106627页

作者： Zhao, Siran Liao, Tianchi Fu, Lele Chen, Chuan Bian, Jing Zheng, Zibin Sun Yat Sen Univ Sch Comp Sci & Engn Guangzhou Peoples R China Sun Yat Sen Univ Sch Software Engn Zhuhai Peoples R China Sun Yat Sen Univ Sch Syst Sci & Engn Guangzhou Peoples R China

data heterogeneity (Non-IID) on Federated Learning (FL) is currently a widely publicized problem, which leads to local model drift and performance degradation. Because of the advantage of knowledge distillation, it has been explored in some recent work to refine global models. However, these approaches rely on a proxy dataset or a data generator. First, in many FL scenarios, proxy dataset do not necessarily exist on the server. Second, the quality of data generated by the generator is unstable and the generator depends on the computing resources of the server. In this work, we propose a novel data-free knowledge distillation approach via generator-free data Generation for Non-IID FL, dubbed as FedF2DG. 2 DG. Specifically, FedF2DG 2 DG requires only local models to generate pseudo datasets for each client, and can generate hard samples by adding an additional regularization term that exploit disagreements between local model and global model. Meanwhile, FedF2DG 2 DG enables flexible utilization of computational resources by generating pseudo dataset locally or on the server. And to address the label distribution shift in Non-IID FL, we propose a data Generation Principle that can adaptively control the label distribution and number of pseudo dataset based on client current state, and this allows for the extraction of more client knowledge. Then knowledge distillation is performed to transfer the knowledge in local models to the global model. Extensive experiments demonstrate that our proposed method significantly outperforms the state-of-the-art FL methods and can serve as plugin for existing Federated Learning methds such as FedAvg, FedProx, etc, and improve their performance.

关键词： Non-IID federated learning data heterogeneity data generation data-free knowledge distillation

来源：评论

学校读者我要写书评

暂无评论

data-free knowledge distillation in neural networks for regression

引用

EXPERT SYSTEMS WITH APPLICATIONS 2021年 175卷 114813-114813页

作者： Kang, Myeonginn Kang, Seokho Sungkyunkwan Univ Dept Ind Engn 2066 Seobu Ro Suwon 16419 South Korea

knowledge distillation has been used successfully to compress a large neural network (teacher) into a smaller neural network (student) by transferring the knowledge of the teacher network with its original training dataset. However, the original training dataset is not reusable in many real-world applications. To address this issue, data-free knowledge distillation, which is knowledge distillation in the absence of the original training datasets, has been studied. However, existing methods are limited to classification problems and cannot be directly applied to regression problems. In this study, we propose a novel data-free knowledge distillation method that is applicable to regression problems. Given a teacher network, we adopt a generator network to transfer the knowledge in the teacher network to a student network. We simultaneously train the generator and student networks in an adversarial manner. The generator network is trained to create synthetic data on which the teacher and student networks make different predictions, with the student network being trained to mimic the teacher network's predictions. We demonstrate the effectiveness of the proposed method on benchmark datasets. Our results show that the student network emulates the prediction ability of the teacher network with little performance loss.

关键词： Neural network knowledge distillation data-free knowledge distillation Zero-shot knowledge distillation Regression

来源：评论

学校读者我要写书评

暂无评论

De-confounded data-free knowledge distillation for Handling Distribution Shifts

De-confounded Data-free Knowledge Distillation for Handling ...

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Wang, Yuzheng Yang, Dingkang Chen, Zhaoyu Liu, Yang Liu, Siao Zhang, Wenqiang Zhang, Lihua Qi, Lizhe Fudan Univ Acad Engn & Technol Shanghai Engn Res Ctr AI & Robot Shanghai Peoples R China Fudan Univ Acad Engn & Technol Engn Res Ctr AI & Robot Minist Educ Shanghai Peoples R China Green Ecol Smart Technol Sch Enterprise Joint Res Shanghai Peoples R China

ISBN: (纸本)9798350353006

data-free knowledge distillation (DFKD) is a promising task to train high-performance small models to enhance actual deployment without relying on the original training data. Existing methods commonly avoid relying on private data by utilizing synthetic or sampled data. However, a long-overlooked issue is that the severe distribution shifts between their substitution and original data, which manifests as huge differences in the quality of images and class proportions. The harmful shifts are essentially the confounder that significantly causes performance bottlenecks. To tackle the issue, this paper proposes a novel perspective with causal inference to disentangle the student models from the impact of such shifts. By designing a customized causal graph, we first reveal the causalities among the variables in the DFKD task. Subsequently, we propose a knowledge distillation Causal Intervention ( KDCI) framework based on the backdoor adjustment to de-confound the confounder. KDCI can be flexibly combined with most existing state-of-the-art baselines. Experiments in combination with six representative DFKD methods demonstrate the effectiveness of our KDCI, which can obviously help existing methods under almost all settings, e.g., improving the baseline by up to 15.54% accuracy on the CIFAR-100 dataset.

关键词： Causal Inference data-free knowledge distillation

来源：评论

学校读者我要写书评

暂无评论

Towards Effective data-free knowledge distillation via Diverse Diffusion Augmentation 24

Towards Effective Data-Free Knowledge Distillation via Diver...

引用

32nd ACM International Conference on Multimedia, MM 2024

作者： Li, Muquan Zhang, Dongyang He, Tao Xie, Xiurui Li, Yuan-Fang Qin, Ke University of Electronic Science and Technology of China Sichuan Chengdu China Monash University Melbourne Australia

ISBN: (纸本)9798400706868

data-free knowledge distillation (DFKD) has emerged as a pivotal technique in the domain of model compression, substantially reducing the dependency on the original training data. Nonetheless, conventional DFKD methods that employ synthesized training data are prone to the limitations of inadequate diversity and discrepancies in distribution between the synthesized and original datasets. To address these challenges, this paper introduces an innovative approach to DFKD through diverse diffusion augmentation (DDA). Specifically, we revise the paradigm of common data synthesis in DFKD to a composite process through leveraging diffusion models subsequent to data synthesis for self-supervised augmentation, which generates a spectrum of data samples with similar distributions while retaining controlled variations. Furthermore, to mitigate excessive deviation in the embedding space, we introduce an image filtering technique grounded in cosine similarity to maintain fidelity during the knowledge distillation process. Comprehensive experiments conducted on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets showcase the superior performance of our method across various teacher-student network configurations, outperforming the contemporary state-of-the-art DFKD methods. Code will be available at: https://***/SLGSP/DDA. © 2024 ACM.

关键词： data-free knowledge distillation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：