检索结果-内蒙古大学图书馆

ISONet: Reforming 1DCNN for aero-engine system inter-shaft bearing fault diagnosis via input spatial over-parameterization

引用

EXPERT SYSTEMS WITH APPLICATIONS 2025年 277卷

作者： Xiang, Qian Wang, Xiaodan Song, Yafei Lei, Lei PLA Rocket Force Univ Engn Lab Intelligent Control Xian 710025 Peoples R China Air Force Engn Univ Coll Air & Missile Def Xian 710051 Peoples R China Air Force Engn Univ Coll Informat & Nav Xian 710077 Peoples R China

Data-driven neural networks have risen as avant-garde approaches to fault diagnosis. However, recent studies have indicated that traditional one-dimensional convolutional neural networks (1DCNNs) exhibit inadequate performance in diagnosing faults within the aero-engine system's inter-shaft bearings. Addressing this, we introduces a novel 1DCNN variant, termed ISONet, which incorporates over-parameterization in the input spatial dimension to enhance the training landscape of 1DCNNs. Theoretically, we prove that this over- parameterization is akin to implicitly integrating a specific preconditioning scheme from a dynamic differential perspective. This preconditioning facilitates optimization movement along previously traversed directions, effectively acting as an acceleration procedure that amalgamates momentum with adaptive learning rates. Empirical validation confirms the irreplaceability of this implicit acceleration mechanism, demonstrating its capacity to further augment the convergence speed and stability of training, even when advanced gradient descent optimizers with explicit momentum and adaptive learning rates are employed. During the training phase, ISONet does not significantly increase training time, and it can be folded into a standard 1DCNN during the testing phase. The efficacy of the proposed ISONet architecture is substantiated through empirical testing utilizing real-world vibration data from an aero-engine test rig. Comparative analyses indicate that ISONet surpasses current state-of-the-art deep learning models, particularly under various limited-sample conditions.

关键词： Aero-engine system Fault diagnosis One-dimensional convolutional neural networks over-parameterization Preconditioning scheme

来源：评论

学校读者我要写书评

暂无评论

Convergence analysis of deep Ritz method with over-parameterization

引用

NEURAL NETWORKS 2025年 184卷 107110页

作者： Ding, Zhao Jiao, Yuling Lu, Xiliang Wu, Peiying Yang, Jerry Zhijian Wuhan Univ Natl Ctr Appl Math Hubei Wuhan 430072 Peoples R China Wuhan Univ Wuhan Inst Math & AI Wuhan 430072 Peoples R China Wuhan Univ Sch Math & Stat Wuhan 430072 Peoples R China Wuhan Univ Hubei Key Lab Computat Sci Wuhan 430072 Peoples R China Wuhan Univ Sch Artificial Intelligence Wuhan 430072 Peoples R China

The deep Ritz method (DRM) has recently been shown to be a simple and effective method for solving PDEs. However, the numerical analysis of DRM is still incomplete, especially why over-parameterized DRM works remains unknown. This paper presents the first convergence analysis of the over-parameterized DRM for second-order elliptic equations with Robin boundary conditions. We demonstrate that the convergence rate can be controlled by the weight norm, regardless of the number of parameters in the network. To this end, we establish novel approximation results in Sobolev spaces with norm constraints, which have independent significance.

关键词： Deep Ritz method over-parameterization Convergence rate Deep approximation with norm control

来源：评论

学校读者我要写书评

暂无评论

over-parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron 36

Over-Parameterization Exponentially Slows Down Gradient Desc...

引用

36th Annual Conference on Learning Theory (COLT)

作者： Xu, Weihang Du, Simon S. Tsinghua Univ Beijing Peoples R China Univ Washington Seattle WA 98195 USA

We revisit the canonical problem of learning a single neuron with ReLU activation under Gaussian input with square loss. We particularly focus on the over-parameterization setting where the student network has n >= 2 neurons. We prove the global convergence of randomly initialized gradient descent with a O (T-3) rate. This is the first global convergence result for this problem beyond the exact-parameterization setting (n = 1) in which the gradient descent enjoys an exp(-Omega(T)) rate. Perhaps surprisingly, we further present an Omega(T-3) lower bound for randomly initialized gradient flow in the over-parameterization setting. These two bounds jointly give an exact characterization of the convergence rate and imply, for the first time, that over-parameterization can exponentially slow down the convergence rate. To prove the global convergence, we need to tackle the interactions among student neurons in the gradient descent dynamics, which are not present in the exact-parameterization case. We use a three-phase structure to analyze GD's dynamics. Along the way, we prove gradient descent automatically balances student neurons, and use this property to deal with the non-smoothness of the objective function. To prove the convergence rate lower bound, we construct a novel potential function that characterizes the pairwise distances between the student neurons (which cannot be done in the exact-parameterization case). We show this potential function converges slowly, which implies the slow convergence rate of the loss function.

关键词： over-parameterization global convergence non-convex optimization

来源：评论

学校读者我要写书评

暂无评论

Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and over-parameterization

引用

JOURNAL OF MACHINE LEARNING RESEARCH 2023年第1期24卷 1-84页

作者： Ma, Jianhao Fattahi, Salar Univ Michigan Dept Ind & Operat Engn Ann Arbor MI 48105 USA Univ Michigan Dept Ind & Operat Engn Ann Arbor MI 48105 USA

In this work, we study the performance of sub-gradient method (SubGM) on a natural nonconvex and nonsmooth formulation of low-rank matrix recovery with t1-loss, where the goal is to recover a low-rank matrix from a limited number of measurements, a subset of which may be grossly corrupted with noise. We study a scenario where the rank of the true solution is unknown and over-estimated instead. The over-estimation of the rank gives rise to an over-parameterized model in which there are more degrees of freedom than needed. Such over-parameterization may lead to overfitting, or adversely affect the performance of the algorithm. We prove that a simple SubGM with small initialization is agnostic to both over-parameterization and noise in the measurements. In particular, we show that small initialization nullifies the effect of over-parameterization on the performance of SubGM, leading to an exponential improvement in its convergence rate. Moreover, we provide the first unifying framework for analyzing the behavior of SubGM under both outlier and Gaussian noise models, showing that SubGM converges to the true solution, even under arbitrarily large and arbitrarily dense noise values, and, perhaps surprisingly, even if the globally optimal solutions do not correspond to the ground truth. At the core of our results is a robust variant of restricted isometry property, called Sign-RIP, which controls the deviation of the sub-differential of the t1-loss from that of an ideal, expected loss. As a byproduct of our results, we consider a subclass of robust low-rank matrix recovery with Gaussian measurements, and show that the number of required samples to guarantee the global convergence of SubGM is independent of the over-parameterized rank.1

关键词： low-rank matrix recovery nonconvex optimization nonsmooth optimization sub-gradient method over-parameterization

来源：评论

学校读者我要写书评

暂无评论

Decomposition-based over-parameterization forgetting factor stochastic gradient algorithm for Hammerstein-Wiener nonlinear systems with non-uniform sampling

引用

INTERNATIONAL JOURNAL OF ROBUST AND NONLINEAR CONTROL 2021年第12期31卷 6007-6024页

作者： Liu, Qilin Xiao, Yongsong Ding, Feng Hayat, Tasawar Jiangnan Univ Sch Internet Things Engn Key Lab Adv Proc Control Light Ind Minist Educ Wuxi Jiangsu Peoples R China Qingdao Univ Sci & Technol Coll Automat & Elect Engn Qingdao Peoples R China King Abdulaziz Univ Dept Math Jeddah Saudi Arabia

This article investigates the parameter estimation problems of Hammerstein-Wiener nonlinear systems with non-uniform sampling. The over-parameterization identification model for the Hammerstein-Wiener nonlinear systems is established from the non-uniformly sampled input-output data. By applying the gradient search principle, we derive an over-parameterization forgetting factor stochastic gradient algorithm for identifying the nonlinear systems. In order to improve the parameter estimation accuracy, a decomposition-based over-parameterization forgetting factor stochastic gradient algorithm is presented by using the decomposition technique. The key is to transform the original system into two subsystems and to estimate the parameters of each subsystem, respectively. The simulation results indicate that the proposed algorithms are effective.

关键词： decomposition technique Hammerstein-Wiener model nonlinear system over-parameterization parameter estimation

来源：评论

学校读者我要写书评

暂无评论

Iterative Identification Algorithms for Bilinear-in-parameter Systems by Using the over-parameterization Model and the Decomposition

引用

INTERNATIONAL JOURNAL OF CONTROL AUTOMATION AND SYSTEMS 2018年第6期16卷 2634-2643页

作者： Chen, Mengting Ding, Feng Alsaedi, Ahmed Hayat, Tasawar Jiangnan Univ Minist Educ Sch Internet Things Engn Key Lab Adv Proc Control Light Ind Wuxi 214122 Peoples R China Qingdao Univ Sci & Technol Coll Automat & Elect Engn Qingdao 266061 Peoples R China King Abdulaziz Univ Fac Sci Dept Math NAAM Res Grp Jeddah Saudi Arabia Quaid I Azam Univ Dept Math Islamabad Pakistan

This paper focuses on the identification problem for a class of bilinear-in-parameter systems with an additive noise modeled by an autoregressive moving average process. By using the over-parameterization model, the special form of the bilinear term can be obtained by the model equivalent transformation. Then, we use a decomposition of the model into two synthetic models in order to separate the effect of the two sets of parameters, i.e., the coefficients of the nonlinear basis functions from the parameters of the colored noise. Moreover, two decomposition based iterative algorithms are proposed to identify the unknown parameters. A numerical example is presented to confirm the effectiveness of the proposed methods.

关键词： Bilinear-in-parameter system decomposition iterative identification over-parameterization parameter estimation

来源：评论

学校读者我要写书评

暂无评论

TENSOR-ON-TENSOR REGRESSION: RIEMANNIAN OPTIMIZATION, over-parameterization, STATISTICAL-COMPUTATIONAL GAP AND THEIR INTERPLAY

引用

ANNALS OF STATISTICS 2024年第6期52卷 2583-2612页

作者： Luo, Yuetian Zhang, Anru r. Univ Chicago Data Sci Inst Chicago IL 60605 USA Duke Univ Dept Biostat & Bioinformat Durham NC USA Duke Univ Dept Comp Sci Durham NC USA

We study the tensor-on-tensor regression, where the goal is to connect tensor responses to tensor covariates with a low Tucker rank parameter tensor/matrix without prior knowledge of its intrinsic rank. We propose the Riemethods and cope with the challenge of unknown rank by studying the effect of rank over-parameterization. We provide the first convergence guarantee for the general tensor-on-tensor regression by showing that RGD and RGN respectively converge linearly and quadratically to a statistically optimal estimate in both rank correctly-parameterized and over-parameterized settings. Our theory reveals an intriguing phenomenon: Riemannian optimization methods naturally adapt to over-parameterization without modifications to their implementation. We also prove the statistical-computational gap in scalar-on-tensor regression by a direct low-degree polynomial argument. Our theory demonstrates a "blessing of statistical-computational gap" phenomenon: in a wide range of scenarios in tensor-on-tensor regression for tensors of order three or higher, the computationally required sample size matches what is needed by moderate rank over-parameterization when considering computationally feasible estimators, while there are no such benefits in the matrix settings. This shows moderate rank over-parameterization is essentially "cost-free" in terms of sample size in tensor-on-tensor regression of order three or higher. Finally, we conduct simulation studies to show the advantages of our proposed methods and to corroborate our theoretical findings.

关键词： Riemannian optimization statistical-computational gaps low-degree polynomials Tensor-on-tensor regression over-parameterization

来源：评论

学校读者我要写书评

暂无评论

COLTR: Semi-Supervised Learning to Rank With Co-Training and over-parameterization for Web Search

引用

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2023年第12期35卷 12542-12555页

作者： Li, Yuchen Xiong, Haoyi Wang, Qingzhong Kong, Linghe Liu, Hao Li, Haifang Bian, Jiang Wang, Shuaiqiang Chen, Guihai Dou, Dejing Yin, Dawei Shanghai Jiao Tong Univ Dept Comp Sci & Engn Shanghai 200240 Peoples R China Baidu Inc Beijing 100085 Peoples R China Hong Kong Univ Sci & Technol Guangzhou Thrust Artificial Intelligence Guangzhou 510000 Guangdong Peoples R China Hong Kong Univ Sci & Technol Dept Comp Sci & Engn Hong Kong 999077 Peoples R China

learning to rank (LTR) has been widely used in web search to prioritize most relevant webpages among the retrieved contents subject to the input queries, the traditional LTR models fail to deliver decent performance due to two main reasons: 1) the lack of well-annotated query-webpage pairs with ranking scores to cover search queries of various popularity, and 2) ill trained models based on a limited number of training samples with poor generalization performance. To improve the performance of LTR models, tremendous efforts have been done from above two aspects, such as enlarging training sets with pseudo-labels of ranking scores by self-training, or refining the features used for LTR through feature extraction and dimension reduction. Though LTR performance has been marginally increased, we still believe these methods could be further improved in the newly-fashioned "interpolating regime". Specifically, instead of lowering the number of features used for LTR models, our work proposes to transform original data with random Fourier feature, so as to over-parameterize the downstream LTR models (e.g., GBRank or LightGBM) with features in ultra-high dimensionality and achieve superb generalization performance. Furthermore, rather than self-training with pseudo-labels produced by the same LTR model in a "self-tuned" fashion, the proposed method incorporates the diversity of prediction results between the listwise and pointwise LTR models while co-training both models with a cyclic labeling-prediction pipeline in a "ping-pong" manner. We deploy the proposed Co-trained and over-parameterized LTR system COLTR at Baidu search and evaluate COLTR with a large number of baseline methods. The results show that COLTR could achieve Delta NDCG(4) = 3.64%similar to 4.92%, compared to baselines, under various ratios of labeled samples. We also conduct a 7-day A/B Test using the realistic web traffics of Baidu Search, where we can still observe significant performance improvement aroun

关键词： Learning to rank semi-supervised learning over-parameterization

来源：评论

学校读者我要写书评

暂无评论

Expansion-Squeeze-Block: Linear over-parameterization with Shortcut Connections to Train Compact Convolutional Networks 8

Expansion-Squeeze-Block: Linear Over-parameterization with S...

引用

8th International Conference on Big Data Computing and Communications (BigCom)

作者： Yang, Linzhuo Zhang, Lan Univ Sci & Technol China Dept Comp Sci Hefei Peoples R China

ISBN: (纸本)9781665473842

We propose a new structure called ExpansionSqueeze-Block by leveraging over-parameterization to train given compact neural networks. The structure expands the width of convolutional layers and adds shortcut connections for better performance without adding any nonlinearity. The expanded networks can be contracted back to the original format algebraically at inference time without loss of information. In addition, we introduce a new initialization method utilizing the weights of the pretrained original networks, which can further improve the accuracy. We evaluate our methods on CIFAR-100 and Tiny-ImageNet datasets to show their effectiveness. As evidenced by our experiments, Expansion-Squeeze-Block outperforms the baselines in most cases with average 0.81% improvements and 1.15% combined with our initialization method on CIFAR-100 and Tiny-ImageNet datasets. We also demonstrate the effectiveness of the partial expansion strategy by choosing the layers of the highest importance in VGG11-BN 0.5x and ResNet20. We can save on average 58.82% #MACs and 38.82% #Params than expanding all layers.

关键词： Deep Learning CNN Convolution over-parameterization

来源：评论

学校读者我要写书评

暂无评论

BOOSTING PRUNED NETWORKS WITH LINEAR over-parameterization 49

BOOSTING PRUNED NETWORKS WITH LINEAR OVER-PARAMETERIZATION

引用

49th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Qian, Yu Li, Xiaoshuang Cao, Jian Zhang, Jie Li, Hufei Chen, Jue Peking Univ Beijing Peoples R China Shanghai Jiao Tong Univ Shanghai Peoples R China Zhejiang Univ Hangzhou Peoples R China

ISBN: (纸本)9798350344868;9798350344851

Structured pruning is a popular technique for reducing the computational cost and memory footprint of neural networks by removing channels. It often leads to a decrease in network accuracy, which can be restored through fine-tuning. However, as the pruning ratio increases, it becomes progressively more difficult to restore the accuracy of the trimmed network. To overcome this challenge, we propose a method that linearly over-parameterizes the compact layers, increasing the number of learnable parameters, and re-parameterizes them back to the original layers after fine-tuning, therefore contributing to better accuracy restoration. Similarity-preserving knowledge distillation is exploited to maintain the feature extraction ability of the expanded layers. Our method can be easily integrated without requiring additional technology, ensuring compatibility and can effectively enhance the pruning process for existing pruning techniques. Our approach outperforms fine-tuning on CIFAR-10 and ImageNet under different pruning strategies, particularly when dealing with large pruning ratios.

关键词： structured pruning fine-tuning over-parameterization knowledge distillation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：