Fully connected deep neural networks (DNN) often include redundant weights leading to overfitting and high memory requirements. Additionally, in tabular data classification, DNNs are challenged by the often superior p...
详细信息
Fully connected deep neural networks (DNN) often include redundant weights leading to overfitting and high memory requirements. Additionally, in tabular data classification, DNNs are challenged by the often superior performance of traditional machine learning models. This paper proposes periodic perturbations (prune and regrow) of DNN weights, especially at the self-supervised pretraining stage of deep autoencoders. The proposed weight perturbation strategy outperforms dropout learning or weight regularization (L1 or L2) for four out of six tabular data sets in downstream classification tasks. Unlike dropout learning, the proposed weight perturbation routine additionally achieves 15% to 40% sparsity across six tabular data sets, resulting in compressed pretrained models. The proposed pretrained model compression improves the accuracy of downstream classification, unlike traditional weight pruning methods that trade off performance for model compression. Our experiments reveal that a pretrained deep autoencoder with weight perturbation can outperform traditional machine learning in tabular data classification, whereas baseline fully-connected DNNs yield the worst classification accuracy. However, traditional machine learning models are superior to any deep model when a tabular data set contains uncorrelated variables. Therefore, the performance of deep models with tabular data is contingent on the types and statistics of constituent variables.(c) 2022 Elsevier Ltd. All rights reserved.
The self-regulated recognition of human activities from time-series smartphone sensor data is a growing research area in smart and intelligent health care. Deep learning (DL) approaches have exhibited improvements ove...
详细信息
The self-regulated recognition of human activities from time-series smartphone sensor data is a growing research area in smart and intelligent health care. Deep learning (DL) approaches have exhibited improvements over traditional machine learning (ML) models in various domains, including human activity recognition (HAR). Several issues are involved with traditional ML approaches;these include handcrafted feature extraction, which is a tedious and complex task involving expert domain knowledge, and the use of a separate dimensionality reduction module to overcome overfitting problems and hence provide model generalization. In this article, we propose a DL-based approach for activity recognition with smartphone sensor data, i.e., accelerometer and gyroscope data. Convolutional neural networks (CNNs), autoencoders (AEs), and long short-term memory (LSTM) possess complementary modeling capabilities, as CNNs are good at automatic feature extraction, AEs are used for dimensionality reduction and LSTMs are adept at temporal modeling. In this study, we take advantage of the complementarity of CNNs, AEs, and LSTMs by combining them into a unified architecture. We explore the proposed architecture, namely, "ConvAE-LSTM", on four different standard public datasets (WISDM, UCI, PAMAP2, and OPPORTUNITY). The experimental results indicate that our novel approach is practical and provides relative smartphone-based HAR solution performance improvements in terms of computational time, accuracy, F1-score, precision, and recall over existing state-of-the-art methods.
This paper proposes a new image caption generative model for Memes called GUMI-AE. Meme denotes a humorous short sentence suitable for the given image in this paper. An Image caption generative model usually consists ...
详细信息
ISBN:
(纸本)9789819970186;9789819970193
This paper proposes a new image caption generative model for Memes called GUMI-AE. Meme denotes a humorous short sentence suitable for the given image in this paper. An Image caption generative model usually consists of an image encoder and a sentence decoder. Furthermore, most conventional models use a pre-trained neural network model for the image encoder, e.g., ResNet152 trained using ImageNet. However, pre-trained ResNet152 may not be effective as an encoder for extracting features from arbitrary images. Because the training samples for the meme generative model can be obtained from the website "Bokete" (in Japanese) which is a website that provides a system for people to post images and humorous short sentences associated with these images. Images posted on Bokete include a wide variety of images such as illustrations and text-only images which may be outside of the training images of ImageNet. This paper proposes an image caption generative model incorporating autoencoder (AE) as the image encoder. AE can be trained with the training samples obtained from Bokete without the image annotation. This enables the proposed method to generate short sentences with humor for memes. Finally, the proposed model is compared with the conventional one, and the evaluation of the proposed GUMI-AE will be discussed.
autoencoder (AE) has been widely used in multivariate statistical process monitoring (MSPM) and various AE-based methods have been applied in fault detection. Process data is often contaminated by outliers and measure...
详细信息
autoencoder (AE) has been widely used in multivariate statistical process monitoring (MSPM) and various AE-based methods have been applied in fault detection. Process data is often contaminated by outliers and measurement noise which may lead to the overfitting problem for AE-based methods. In this paper, a novel feature extraction method called low-rank reconstruction-based autoencoder (LRAE) is proposed for robust fault detection. LRAE decomposes the input into a combination of a low-rank data matrix and a noise matrix. By penalizing the rank of the data matrix, LRAE separates the low-rank clean data from the contaminated process data. Instead of directly reconstructing the loss between the input data and the output data, we design a low-rank reconstruction strategy, i.e. reconstruct the loss between the low-rank clean data and the output of the AE. The proposed LRAE can be trained end-to-end by jointly optimizing an AE and a low-rank approximation. LRAE is a nonlinear method which can tackle the complicated process data better than the linear methods such as principal component analysis (PCA). Moreover, the optimization of the low-rank approximation provides the robustness of LRAE to reconstruct the clean data in the output layer when the input process data is contaminated. After training, the features of the hidden layer can be computed for further fault detection. Extensive experiments demonstrate that LRAE outperforms traditional fault detection methods, including PCA, robust principal component analysis (RPCA), kernel principal component analysis (KPCA), AE, and denoising autoencoder (DAE). Especially, LRAE provides more robust results when the process data suffer from outliers and measurement noise.
Cognitive network management is becoming quintessential to realize autonomic networking. However, the wide spread adoption of the Internet of Things (IoT) devices, increases the risk of cyber attacks. Adversaries can ...
详细信息
Cognitive network management is becoming quintessential to realize autonomic networking. However, the wide spread adoption of the Internet of Things (IoT) devices, increases the risk of cyber attacks. Adversaries can exploit vulnerabilities in IoT devices, which can be harnessed to launch massive Distributed Denial of Service (DDoS) attacks. Therefore, intelligent security mechanisms are needed to harden network security against these threats. In this paper, we propose Chronos, a novel time-based anomaly detection system. The anomaly detector, primarily an autoencoder, leverages time-based features over multiple time windows to efficiently detect anomalous DDoS traffic. We develop a threshold selection heuristic that maximizes the F1-score across various DDoS attacks. Further, we compare the performance of Chronos against state-of-the-art approaches. We show that Chronos marginally outperforms another time-based system using a less complex anomaly detection pipeline, while out classing flow-based approaches with superior precision. In addition, we showcase the robustness of Chronos in the face of zero-day attacks, noise in training data, and a small number of training packets, asserting its suitability for online deployment.
In hyperspectral image (HSI) analysis, high-dimensional data may contain noisy, irrelevant and redundant information. To mitigate the negative effect from these information, feature selection is one of the useful solu...
详细信息
In hyperspectral image (HSI) analysis, high-dimensional data may contain noisy, irrelevant and redundant information. To mitigate the negative effect from these information, feature selection is one of the useful solutions. Unsupervised feature selection is a data preprocessing technique for dimensionality reduction, which selects a subset of informative features without using any label information. Different from the linear models, the autoencoder is formulated to nonlinearly select informative features. The adjacency matrix of HSI can be constructed to extract the underlying relationship between each data point, where the latent representation of original data can be obtained via matrix factorization. Besides, a new feature representation can be also learnt from the autoencoder. For a same data matrix, different feature representations should consistently share the potential information. Motivated by these, in this paper, we propose a latent representation learning based autoencoder feature selection (LRLAFS) model, where the latent representation learning is used to steer feature selection for the autoencoder. To solve the proposed model, we advance an alternative optimization algorithm. Experimental results on three HSI datasets confirm the effectiveness of the proposed model.
Unsupervised feature selection is one of the efficient approaches to reduce the dimension of unlabeled high-dimensional data. We present a novel adaptive autoencoder with redundancy control (AARC) as an unsupervised f...
详细信息
Unsupervised feature selection is one of the efficient approaches to reduce the dimension of unlabeled high-dimensional data. We present a novel adaptive autoencoder with redundancy control (AARC) as an unsupervised feature selector. By adding two Group Lasso penalties to the objective function, AARC integrates unsupervised feature selection and determination of a compact network structure into a single framework. Besides, a penalty based on a measure of dependency between features (such as Pearson correlation, mutual information) is added to the objective function for controlling the level of redundancy in the selected features. To realize the desired effects of different regularizers in different phases of the training, we introduce adaptive parameters which change with iterations. In addition, a smoothing function is utilized to approximate the three penalties since they are not differentiable at the origin. An ablation study is carried out to validate the capabilities of redundancy control and structure optimization of AARC. Subsequently, comparisons with nine state-of-the-art methods illustrate the efficiency of AARC for unsupervised feature selection. (c) 2022 Elsevier Ltd. All rights reserved.
Dynamic information is a non-negligible part of time-correlated process data, and its full utilization can improve the performance of fault detection. Traditional dynamic methods concatenate the current process data w...
详细信息
Dynamic information is a non-negligible part of time-correlated process data, and its full utilization can improve the performance of fault detection. Traditional dynamic methods concatenate the current process data with a certain number of previous process data into an extended vector before performing feature extraction. However, this simple way of using dynamic information inevitably increases the input dimensionality and it is inappropriate to treat previous process data as equally important. To address these problems, this paper proposes a novel nonlinear dynamic method, called graph dynamic autoencoder (GDAE), for fault detection. GDAE utilizes a graph structure to model the dynamic information between different data points. GDAE firstly embeds the current data point and previous data points as the features of the central node and its neighbors, respectively, then convolves the feature of the central node with the features of its neighbors to derive the updated feature for the central node, and finally, an encoder-decoder structure is adopted to extract the key low-dimensional feature. Due to the utilization of the graph structure, the extended high-dimensional vectors utilized by traditional dynamic fault detection methods are avoided in GDAE. Furthermore, with the dynamically constructed graph, GDAE is able to adaptively assign different weights to its neighbors by updating the adjacency matrix of the graph. Experimental results obtained from a numerical simulation and the Tennessee Eastman process illustrate the superiority of GDAE in terms of missed detection rate (MDR) and false alarm rate (FAR). The source code of GDAE can be found in https://***/luliu-fighting/Graph-Dynamic-autoencoder. (c) 2022 Elsevier Ltd. All rights reserved.
Document digitization has an important role in helping the company's activities be more efficient, such as detecting text in invoice document images using optical character recognition (OCR). However, writing in i...
详细信息
ISBN:
(纸本)9798350334449
Document digitization has an important role in helping the company's activities be more efficient, such as detecting text in invoice document images using optical character recognition (OCR). However, writing in images has many problems, especially tediously saved documents that can cause noise or interference in the picture, resulting in difficultly recognized writing. Our research aims to build an autoencoder for denoising text images and evaluate the OCR's performance in converting the denoised image into text. The first step in the research is to test the OCR characteristics on the original text image and the text image given Gaussian noise. The next step is to build the optimal autoencoder model for denoising by studying the effect of dataset size and optimizer type. The last step is to test the OCR performance on the denoised text image produced by the optimum autoencoder model. The test results show that datasetsize affects denoising performance and OCR performance. From several autoencoder models compared, the autoencoder with dataset size = 40 has the optimum performance, where the MSE values of the model for train and validation are 1277 and 1385, respectively. With images denoised from the optimum model, the OCR performance in converting images into text is 100%.
In today's industrial processes, data-driven soft sensors are a frequently used tool for predicting quality variables. autoencoder (AE) is an unsupervised algorithm which can extract latent features from initial d...
详细信息
ISBN:
(纸本)9798350321050
In today's industrial processes, data-driven soft sensors are a frequently used tool for predicting quality variables. autoencoder (AE) is an unsupervised algorithm which can extract latent features from initial data. However, during the feature extraction process, the traditional autoencoder does not consider the correlation between modeling input variables and quality variables to be predicted. To solve this issue, a novel autoencoder based on variable correlation analysis (VCA-AE) is proposed. In VCA-AE, the correlation of modeling input variables and quality variables to be predicted is performed by correlation analysis, and input variables are divided into two parts, which are input to the sub-autoencoder to extract latent features, respectively. In each sub-autoencoder, input variables and quality variables have the same correlation. Next, a feedforward neural network Extreme Learning Machine (ELM) is used to develop soft sensor model based on the extracted latent feature variables and quality variables. Finally, the effectiveness of the proposed soft sensor model combining VCA-AE and ELM is illustrated by an experiment of the industrial PTA process.
暂无评论