检索结果-内蒙古大学图书馆

Post-Training Quantization for Diffusion Transformer via Hierarchical Timestep Grouping

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Ding, Ning Han, Jing Tian, Yuchuan Xu, Chao Han, Kai Tang, Yehui State Key Lab of General AI School of Intelligence Science and Technology Peking University China School of Artificial Intelligence Beijing University of Posts and Telecommunications China Huawei Noah’s Ark Lab Canada

Diffusion Transformer (DiT) has now become the preferred choice for building image generation models due to its great generation capability. Unlike previous convolution-based UNet models, DiT is purely composed of a stack of transformer blocks, which renders DiT excellent in scalability like large language models. However, the growing model size and multi-step sampling paradigm bring about considerable pressure on deployment and inference. In this work, we propose a post-training quantization framework tailored for Diffusion Transforms to tackle these challenges. We firstly locate that the quantization difficulty of DiT mainly originates from the time-dependent channel-specific outliers. We propose a timestep-aware shift-and-scale strategy to smooth the activation distribution to reduce the quantization error. Secondly, based on the observation that activations of adjacent timesteps have similar distributions, we utilize a hierarchical clustering scheme to divide the denoising timesteps into multiple groups. We further design a re-parameterization scheme which absorbs the quantization parameters into nearby module to avoid redundant computations. Comprehensive experiments demonstrate that out PTQ method successfully quantize the Diffusion Transformer into 8-bit weight and 8-bit activation (W8A8) with state-of-the-art FiD score. And our method can further quantize DiT model into 4-bit weight and 8-bit activation (W4A8) without sacrificing generation quality. © 2025, CC BY-NC-SA.

关键词： Hierarchical clustering

Symmetry Discovery for Different Data Types

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hu, Lexiang Li, Yikang Lin, Zhouchen State Key Lab of General AI School of Intelligence Science and Technology Peking University China Institute for Artificial Intelligence Peking University China Guangdong Guangzhou China

Equivariant neural networks incorporate symmetries into their architecture, achieving higher generalization performance. However, constructing equivariant neural networks typically requires prior knowledge of data types and symmetries, which is difficult to achieve in most tasks. In this paper, we propose LieSD, a method for discovering symmetries via trained neural networks which approximate the input-output mappings of the tasks. It characterizes equivariance and invariance (a special case of equivariance) of continuous groups using Lie algebra and directly solves the Lie algebra space through the inputs, outputs, and gradients of the trained neural network. Then, we extend the method to make it applicable to multi-channel data and tensor data, respectively. We validate the performance of LieSD on tasks with symmetries such as the two-body problem, the moment of inertia matrix prediction, and top quark tagging. Compared with the baseline, LieSD can accurately determine the number of Lie algebra bases without the need for expensive group sampling. Furthermore, LieSD can perform well on nonuniform datasets, whereas methods based on GANs fail. Copyright © 2024, The Authors. All rights reserved.

关键词： Neural networks

HYPNOS: Interactive Data Lineage Tracing for Data Transformation Scripts

学校读者我要写书评

暂无评论

IEEE Transactions on Visualization and Computer Graphics 2025年 PP卷 PP页

作者： Cai, Xiwen Ge, Xiaodong Xiong, Kai Ye, Shuainan Weng, Di Xu, Ke Wei, Datong Long, Jiang Wu, Yingcai Zhejiang University State Key Lab of CADCG Hangzhou China Zhejiang Lab Hangzhou China Zhejiang University School of Software Technology Hangzhou China Nanjing University School of Intelligence Science and Technology Nanjing China HUAWEI Cloud Data Intelligence Innovation Lab Hangzhou China Zhejiang University Hangzhou China

In a formal data analysis workflow, data validation is a necessary step that helps data analysts verify the quality of the data and ensure the reliability of the results. Data analysts usually need to validate the result when encountering an unexpected result, such as an abnormal record in a table. In order to understand how a specific record is derived, they would backtrace it in the pipeline step by step via checking the code lines, exposing the intermediate tables, and finding the data records from which it is derived. However, manually reviewing code and backtracing data requires certain expertise, while inspecting the traced records in multiple tables and interpreting their relationships is tedious. In this work, we propose HYPNOS, a visualization system that supports interactive data lineage tracing for data transformation scripts. HYPNOS uses a lineage module for parsing and adapting code to capture both schema-level and instance-level data lineage from data transformation scripts. Then, it provides users with a lineage view for obtaining an overview of the data transformation process and a detail view for tracing instance-level data lineage and inspecting details. HYPNOS reveals different levels of data relationships and helps users with data lineage tracing. We demonstrate the usability and effectiveness of HYPNOS through a use case, interviews of four expert users, and a user study. © 1995-2012 IEEE.

关键词： Data assimilation

Incorporating Arbitrary Matrix Group Equivariance into KANs

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hu, Lexiang Wang, Yisen Lin, Zhouchen State Key Lab of General AI School of Intelligence Science and Technology Peking University China Institute for Artificial Intelligence Peking University China Guangdong Guangzhou China

Kolmogorov-Arnold Networks (KANs) have seen great success in scientific domains thanks to spline activation functions, becoming an alternative to Multi-Layer Perceptrons (MLPs). However, spline functions may not respect symmetry in tasks, which is crucial prior knowledge in machine learning. In this paper, we propose Equivariant Kolmogorov-Arnold Networks (EKAN), a method for incorporating arbitrary matrix group equivariance into KANs, aiming to broaden their applicability to more fields. We first construct gated spline basis functions, which form the EKAN layer together with equivariant linear weights, and then define a lift layer to align the input space of EKAN with the feature space of the dataset, thereby building the entire EKAN architecture. Compared with baseline models, EKAN achieves higher accuracy with smaller datasets or fewer parameters on symmetry-related tasks, such as particle scattering and the three-body problem, often reducing test MSE by several orders of magnitude. Even in non-symbolic formula scenarios, such as top quark tagging with three jet constituents, EKAN achieves comparable results with state-of-the-art equivariant architectures using fewer than 40% of the parameters, while KANs do not outperform MLPs as expected. Copyright © 2024, The Authors. All rights reserved.

关键词： Splines

EMG Based Rehabilitation Gesture Recognition Using DAE-CNN-LSTM Hybrid Model

学校读者我要写书评

暂无评论

EMG Based Rehabilitation Gesture Recognition Using DAE-CNN-L...

2024 International Convention on Rehabilitation Engineering and Assistive technology and World Rehabilitation Robot Convention, i-CREATE and WRRC 2024

作者： Cao, Wujing Guo, Xinqiang Zou, Yupeng Zhang, Shuo Luo, Mingxiang Kobsiriphat, Worawarit Wu, Xinyu Yin, Meng Shenzhen Institute of Advanced Technology Chinese Academy of Sciences Guangdong Provincial Key Lab of Robotics and Intelligent System Shenzhen518005 China SIAT Branch Shenzhen Institute of Artificial Intelligence and Robotics for Society SIAT-CUHK Joint Laboratory of Robotics and Intelligent Systems China School of Mechanical Engineering China University of Petroleum China National Metal and Materials Technology Center Thailand

ISBN: (纸本)9798350355123

Stroke, as one of the leading causes of long-term disability globally, often results in motor impairments, particularly in the hands, significantly affecting patients' daily activities and causing profound psychological trauma. Rehabilitation gesture recognition, as one of the key means of active rehabilitation medicine, holds significant potential in stroke rehabilitation, providing real-time feedback on patient progress and enabling personalized interventions to meet individual needs. Traditional gesture recognition methods face numerous challenges when dealing with stroke patient data, such as noise, motion blur, and individual differences. To address these challenges, this paper proposes a novel approach for stroke patient gesture recognition using deep learning models. We adopt a strategy combining denoising autoencoder (DAE), convolutional neural network (CNN), and long short-term memory (LSTM) to enhance the accurate recognition of hand movements in stroke patients. Specifically, DAE is utilized for denoising EMG signals, extracting features, and reducing noise to improve the signal's noise resistance. CNN is employed for spatial feature extraction, while LSTM captures temporal dependencies in gesture sequences. By integrating these three deep learning models, our aim is to enhance the accuracy and robustness of rehabilitation gesture recognition. We validate the proposed method's effectiveness on an EMG dataset from seven subjects through experiments and compare it with traditional machine learning and individual CNN and LSTM algorithms. Experimental results demonstrate significant performance improvements of our hybrid model in rehabilitation gesture recognition tasks, indicating promising application prospects and practical significance. © 2024 IEEE.

关键词： Convolutional neural networks

Accelerated gradient tracking over time-varying graphs for decentralized optimization

学校读者我要写书评

暂无评论

The Journal of Machine Learning Research

The Journal of Machine Learning Research 2024年第1期25卷 13187-13238页

作者： Huan Li Zhouchen Lin Institute of Robotics and Automatic Information Systems College of Artificial Intelligence Nankai University Tianjin China State Key Lab of General AI School of Intelligence Science and Technology Institute for Artificial Intelligence Peking University Beijing China and Pazhou Laboratory (Huangpu) Guangzhou China

Decentralized optimization over time-varying graphs has been increasingly common in modern machine learning with massive data stored on millions of mobile devices, such as in federated learning. This paper revisits the widely used accelerated gradient tracking and extends it to time-varying graphs. We prove that the practical single loop accelerated gradient tracking needs $O((\frac{\gamma}{1-\sigma_{\gamma}})^2\sqrt{\frac{L}{\epsilon}})$ and $O((\frac{\gamma}{1-\sigma_{\gamma}})^{1.5}\sqrt{\frac{L}{\mu}} log{\frac{1}{\epsilon}})$ iterations to reach an ε-optimal solution over time-varying graphs when the problems are nonstrongly convex and strongly convex, respectively, where γ and σγ are two common constants charactering the network connectivity, L and µ are the smoothness and strong convexity constants, respectively, and one iteration corresponds to one gradient oracle call and one communication round. Our convergence rates improve significantly over the ones of $O(\frac{1}{\epsilon^{5/7}})$ and $O((\frac{L}{\mu})^{5/7}\frac{1}{(1-\sigma)^{1.5}}\log\frac{1}{\epsilon})$, respectively, which were proved in the original literature of accelerated gradient tracking only for static graphs, where $\frac{\gamma}{1-\sigma_{\gamma}}$ equals 1/1-σ when the network is time-invariant. When combining with a multiple consensus subroutine, the dependence on the network connectivity constants can be further improved to O(1) and $O(\frac{\gamma}{1-\sigma_{\gamma}})$ for the gradient oracle and communication round complexities, respectively. When the network is static, by employing the Chebyshev acceleration, our complexities exactly match the lower bounds without hiding any poly-logarithmic factor for both nonstrongly convex and strongly convex problems.

关键词： decentralized optimization accelerated gradient tracking time-varying graphs

Mixture of Lookup Experts

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Jie, Shibo Tang, Yehui Han, Kai Li, Yitong Tang, Duyu Deng, Zhi-Hong Wang, Yunhe State Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University China Huawei Noah’s Ark Lab Canada Consumer Business Group Huawei

Mixture-of-Experts (MoE) activates only a subset of experts during inference, allowing the model to maintain low inference FLOPs and latency even as the parameter count scales up. However, since MoE dynamically selects the experts, all the experts need to be loaded into VRAM. Their large parameter size still limits deployment, and offloading, which load experts into VRAM only when needed, significantly increase inference latency. To address this, we propose Mixture of Lookup Experts (MoLE), a new MoE architecture that is efficient in both communication and VRAM usage. In MoLE, the experts are Feed-Forward Networks (FFNs) during training, taking the output of the embedding layer as input. Before inference, these experts can be re-parameterized as lookup tables (LUTs) that retrieves expert outputs based on input ids, and offloaded to storage devices. Therefore, we do not need to perform expert computations during inference. Instead, we directly retrieve the expert’s computation results based on input ids and load them into VRAM, and thus the resulting communication overhead is negligible. Experiments show that, with the same FLOPs and VRAM usage, MoLE achieves inference speeds comparable to dense models and significantly faster than MoE with experts offloading, while maintaining performance on par with MoE. Code: https://***/JieShibo/MoLE © 2025, CC0.

关键词： HTTP

Dispersion Staining Method for Extracting Image Characteristics of Suspended Fibers in Transformer Oil

学校读者我要写书评

暂无评论

Dispersion Staining Method for Extracting Image Characterist...

2023 IEEE Sustainable Power and Energy Conference, iSPEC 2023

作者： Zheng, Zhibo Lu, Bowen Gan, Wenyan Huang, Zhengyong Wang, Qian Wang, Feipeng Li, Jian State Key Lab of Power Transmission Equipment & System Security and New Technology School of Electrical Engineering Chongqing University Chongqing China State Grid Chongqing Electric Power Company Research Institute State Grid Chongqing Electric Power CO. LTD Chongqing China

ISBN: (纸本)9798350322699

In order to extract the new aging characteristics of suspended fibers in insulating oil and solve the defects in the application of current aging characteristics of transformer paper insulation, the accelerated thermal aging experiments of transformer oil-paper insulation and the experiment of image characteristics acquisition based on dispersive staining method were carried out at 90℃, 110℃ and 130℃, respectively. The suspended fibers in insulating oil were obtained, and the dispersion staining images of suspended fibers under different aging degrees were observed. Through image processing technology, the extraction of aging characteristics of dispersion staining images were acquired with the optimal expansion radius and optimal corrosion radius. The results showed that: As aging progressed, the dispersion staining color of the suspended fibers in insulating oil gradually changed from the initial blue to red and orange, the proportion of blue of the three primary colors gradually decreased, while the proportion of red gradually increased, and the color inhomogeneity increased at first and then decreased. The extracted three primary color proportion and color inhomogeneity can be used as reliable predictors of the aging degree of oil-paper insulation, showing its application potential in the aging life evaluation and condition diagnosis of transformers. © 2023 IEEE.

关键词： dispersion staining image processing technology oil-paper insulation the suspended fibers in insulating oil

ViT-based Terrain Recognition system for wearable soft exosuit

学校读者我要写书评

暂无评论

Biomimetic intelligence & Robotics 2023年第1期3卷 57-64页

作者： Fangliang Yang Chunjie Chen Zhuo Wang Hui Chen Yao Liu Gang Li Xinyu Wua Shenzhen Institute of Advanced Technology Chinese Academy of SciencesShenzhen 518055China Shenzhen College of Advanced Technology University of Chinese Academy of SciencesShenzhen 518055China Guangdong Provincial Key Lab of Robotics and Intelligent System Shenzhen Institute of Advanced TechnologyChinese Academy of SciencesShenzhen 518055China Guangdong-Hong Kong-Macao Joint Laboratory of Human–Machine Intelligence-Synergy Systems Shenzhen 518055China SIASUN Robot and Automation CO. LtdShenyang 110169China

Terrain classification and force assistance strategies in complex environments have always piqued the interest of many *** wearable soft exosuits,inaccurate terrain recognition can easily introduce undesired assist forces that can easily injure the *** of these problems,we introduced a depth camera into the exosuit system,perform classification of terrain based on a Vision Transformer(ViT),and optimized the control algorithm,which is known as a ViT-Based Terrain Recognition system(TTRS).First,we used the Transformer algorithm to achieve a considerable classification effect in terrain *** also introduced terrain recognition as prior knowledge into the force assistance strategy of the exosuit,providing different force assistance to the exosuit in different ***,we performed human experiments with seven able-bodied people(six males and one female).The promising results demonstrate that our classification accuracy can reach 99.2%under six different terrains and that it can smoothly switch the force–assist curve in different terrains to better adapt to the complex terrain and improve the walking *** aforementioned terrain recognition algorithms and force–assist strategies may positively influence the study of soft exosuit,powered prostheses,and orthotics.

关键词： Vision Transformer(ViT) Depth camera Wearable soft exosuit Force assistance strategy Terrain recognition