检索结果-内蒙古大学图书馆

transformer encoder Model for Sequential Prediction of Student Performance Based on Their Log Activities

IEEE ACCESS 2023年 11卷 18960-18971页

作者： Kusumawardani, Sri Suning Alfarozi, Syukron Abu Ishaq Univ Gadjah Mada Dept Elect & Informat Engn Yogyakarta 55281 Indonesia

Learning management systems (LMSs) have been used massively due to the growing utilization of distance learning. This advancement has led to increased educational data that can be analyzed to improve the quality of the learning process. Learning analytics (LA) is one of the most important methods that can be used to analyze student performance. In this paper, we proposed an LA method based on deep learning, i.e., transformer encoder, to sequentially predict the student's final performance based on log activities provided by an LMS. The objective is to predict at-risk students of failing so that they can be mitigated as soon as possible. The proposed model was evaluated on the Open University LA Dataset (OULAD) for daily or weekly prediction. The results show that the model could predict at the early stage with an accuracy of 83.17% on withdrawn versus pass-distinction classes. Meanwhile, for the other tasks, i.e., withdrawn-fail versus pass-distinction and fail versus pass-distinction tasks, the accuracy was at least 76% at the early stage. The proposed model was compared to the LSTM model. We found that the transformer encoder performed better than the LSTM, with the average difference values from 1% to 3% in terms of accuracy and from 3% to 7% in terms of F1-score for all tasks, based on the statistical testing. Furthermore, the ablation study using positional encoding, different feature aggregation methods, and weighted loss function for the imbalanced class problem was conducted. In OULAD, we found that model without positional encoding was better in all cases. Furthermore, the weekly feature aggregation and the use of a weighted loss function performed better in some cases.

关键词： Predictive models transformers Learning systems Data models Feature extraction Encoding Electronic learning Sequential analysis Learning analytics transformer encoder student at-risk prediction massive open online courses sequential model imbalanced dataset

来源：评论

学校读者我要写书评

暂无评论

Research on Ship Anomaly Detection Algorithm Based on transformer-GSA encoder

引用

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 2025年

作者： Liu, Hongdan Wu, Congjie Li, Bing Zong, Zhengcong Shu, Yaqing Harbin Engn Univ Coll Intelligent Sci & Engn Harbin 150001 Peoples R China Wuhan Univ Technol Hubei Key Lab Inland Shipping Technol Sch Nav Wuhan 430063 Peoples R China

As global economies continue to expand, maritime trade is projected to double by the year 2050. In this context, the development of a new generation of maritime intelligent supervision systems, which rely on precise information perception and machine learning-based technologies, is crucial for ensuring the safety of maritime navigation. Traditional methods for detecting abnormal ship behavior often suffer from inadequate feature extraction capabilities, leading to ineffective and inaccurate anomaly detection. To address this issue, this paper introduces a transformer-GSA encoder detection model. This model enhances ship information data through BCE-GAN, extracts multi-layer convolutional features, and utilizes the transformer-GSA module to capture the dependencies between trajectory features and time-series data, thereby effectively identifying abnormal ship behaviors. An evaluation conducted using AIS data from the Yantai-Qingdao route in January 2021 demonstrates that the model achieves an accuracy of 96.26%, with a reduced parameter count of 12.11M. These results confirm the model's efficacy and superiority in detecting abnormal ship trajectories, contributing to the reduction of maritime accidents and illegal activities, alleviating the management burden on duty personnel, and enhancing the quality of intelligent ship traffic services and port operation efficiency under the maritime Internet of Things architecture.

关键词： Marine vehicles Data models transformers Trajectory Feature extraction Anomaly detection Biological system modeling Artificial intelligence Accuracy Navigation Ship anomaly detection AIS transformer encoder global attention lightweight

来源：评论

学校读者我要写书评

暂无评论

Elimination of Random Mixed Noise in ECG Using Convolutional Denoising Autoencoder With transformer encoder

引用

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 2024年第4期28卷 1993-2004页

作者： Chen, Meng Li, Yongjian Zhang, Liting Liu, Lei Han, Baokun Shi, Wenzhuo Wei, Shoushui Shandong Univ Sch Control Sci & Engn Jinan 250061 Peoples R China Shandong Univ Shandong Prov Hosp Dept Cardiol Jinan 250061 Peoples R China Shandong Univ Inst Marine Sci & Technol Qingdao 266237 Peoples R China

Electrocardiogram (ECG) signals frequently encounter diverse types of noise, such as baseline wander (BW), electrode motion (EM) artifacts, muscle artifact (MA), and others. These noises often occur in combination during the actual data acquisition process, resulting in erroneous or perplexing interpretations for cardiologists. To suppress random mixed noise (RMN) in ECG with less distortion, we propose a transformer-based Convolutional Denoising Autoencoder model (TCDAE) in this study. The encoder of TCDAE is composed of three stacked gated convolutional layers and a transformer encoder block with a point-wise multi-head self-attention module. To obtain minimal distortion in both time and frequency domains, we also propose a frequency weighted Huber loss function in training phase to better approximate the original signals. The TCDAE model is trained and tested on the QT Database (QTDB) and MIT-BIH Noise Stress Test Database (NSTDB), with the training data and testing data coming from different records. All the metrics perform the most robust in overall noise and separate noise intervals for RMN removal compared with the baseline methods. We also conduct generalization tests on the Icentia11k database where the TCDAE outperforms the state-of-the-art models, with a 55% reduction of the false positives in R peak detection after denoising. The TCDAE model approximates the short-term and long-term characteristics of ECG signals and has higher stability even under extreme RMN corruption. The memory consumption and inference speed of TCDAE are also feasible for its deployment in clinical applications.

关键词： Electrocardiography Convolution transformers Feature extraction Logic gates Adaptive filters Wiener filters Denoising autoencoder ECG denoising multi-head self-attention point-wise convolution random mixed noise transformer encoder

来源：评论

学校读者我要写书评

暂无评论

Multi-level network based on transformer encoder for fine-grained image-text matching

引用

MULTIMEDIA SYSTEMS 2023年第4期29卷 1981-1994页

作者： Yang, Lei Feng, Yong Zhou, Mingliang Xiong, Xiancai Wang, Yongheng Qiang, Baohua Chongqing Univ Coll Comp Sci Chongqing 400044 Peoples R China Minist Nat Resources Key Lab Monitoring Evaluat & Early Warning Terr Sp Chongqing 401147 Peoples R China Chongqing Inst Planning & Nat Resources Invest & M Chongqing 401121 Peoples R China Zhejiang Lab Hangzhou 311121 Peoples R China Guilin Univ Elect Technol Guangxi Key Lab Trusted Software Guilin 541004 Peoples R China

Enabling image-text matching is important to understand both vision and language. Existing methods utilize the cross attention mechanism to explore deep semantic information. However, the majority of these methods need to perform two types of alignment, which is extremely time-consuming. In addition, current methods do not consider the digital information within the image or text, which may lead to a reduction in retrieval performance. In this paper, we propose a multi-level network, which is based on the transformer encoder for fine-grained, image-text matching. First, we use the transformer encoder to extract intra-modality relations within the image and text and perform the alignment through an efficient aggregating method, rendering the alignment more efficient and the intra-modality information fully utilized. Second, we capture the discriminative digital information within the image and text to make the representation more distinguishable. Finally, we utilize the global information of the image and text as complementary information to enhance the representation. According to our experimental results, significant improvements in terms of retrieval tasks and runtime estimation can be achieved compared with state-of-the-art algorithms. The source code is available at https://***/CQULab/MNTE.

关键词： Image-text matching Multi-level network transformer encoder Fine-grained information

来源：评论

学校读者我要写书评

暂无评论

Improving policy training for autonomous driving through randomized ensembled double Q-learning with transformer encoder feature evaluation

引用

APPLIED SOFT COMPUTING 2024年 167卷

作者： Fan, Jie Zhang, Xudong Zou, Yuan Li, Yuanyuan Liu, Yingqun Sun, Wenjing Beijing Inst Technol Sch Mech Engn Natl Engn Res Ctr Elect Vehicles 5 South Zhongguancun St Beijing 100081 Peoples R China

In the burgeoning field of autonomous driving, reinforcement learning (RL) has gained prominence for its adaptability and intelligent decision-making. However, conventional RL methods face challenges inefficiently extracting relevant features from high-dimensional inputs and maximizing the use of environment-agent interaction data. To surmount these obstacles, this paper introduces a novel RL-based approach that integrates randomized ensembled double Q-Learning (REDQ) with a transformer encoder. The transformer encoder's attention mechanism is utilized to dynamically evaluate features according to their relevance indifferent driving scenarios. Simultaneously, the implementation of REDQ, characterized by a high update-to-data (UTD) ratio, enhances the utilization of interaction data during policy training. Especially, the incorporation of ensemble strategy and in-target minimization in REDQ significantly improves training stability, especially under high UTD conditions. Ablation studies indicate that the transformer encoder exhibits significantly enhanced feature extraction capabilities compared to conventional network architectures, resulting in a 13.6% to 21.4% increase in success rate for the MetaDrive autonomous driving task. Additionally, when compared to standard RL methodologies, the proposed approach demonstrates a faster rate of reward acquisition and achieves a 67.5% to 69% improvement in success rate.

关键词： Autonomous driving Reinforcement learning transformer encoder

来源：评论

学校读者我要写书评

暂无评论

Defocus Blur detection via transformer encoder and edge guidance

引用

APPLIED INTELLIGENCE 2022年第12期52卷 14426-14439页

作者： Zhao, Zijian Yang, Hang Luo, Huiyuan Chinese Acad Sci Changchun Inst Opt Fine Mech & Phys Changchun 130033 Peoples R China Univ Chinese Acad Sci Beijing 100049 Peoples R China

Defocus blur detection (DBD) aims to separate blurred and unblurred regions for a given image. Benefiting from the powerful extraction capabilities of convolutional neural networks (CNNs), deep learning based defocus blur detection has achieved a remarkable progress compared with traditional methods. However, due to the limited local receptive field of CNNs, it is difficult to achieve satisfactory results in the detection of the low-contrast focal regions. Besides, the output maps of the most of previous works have coarse object boundaries and background clutter. In this paper, we propose a hybrid CNN-transformer architecture with an edge guidance aggregation module (EGAM) and a feature fusion module (FFM) for DBD. In our knowledge, this is the first study to utilize a transformer encoder for DBD to capture the global context information. Additionally, an edge extraction network (EENet) is adopted to obtain local edge information of in-focus objects. To effectively aggregate local edge information and global semantic features, three EGAMs are integrated into an edge guidance fusion network (EGFNet). Benefiting from the rich edge information, the fused features can generate more accurate boundaries. Finally, three FFMs are cascaded as a hierarchical feature aggregation network (HFANet) to hierarchically decode and refine the feature maps. Experimental results on three widely used DBD datasets demonstrate that the proposed model outperforms the state-of-the-art approaches.

关键词： Defocus blur detection Edge guidance aggregation transformer encoder Low-contrast focal regions

来源：评论

学校读者我要写书评

暂无评论

Anomalous Indoor Human Trajectory Detection Based on the transformer encoder and Self-Organizing Map

引用

IEEE ACCESS 2023年 11卷 131848-131865页

作者： Lan, Doi Thi Yoon, Seokhoon Univ Ulsan Dept Elect Elect & Comp Engn Ulsan 44610 South Korea

Anomalous human trajectory detection is a critical task in security surveillance in working areas. To identify anomalous human trajectories, understanding features of their movement plays an important role. Therefore, in this work, a transformer encoder and self-organizing map-based model called TENSO is proposed to learn trajectory characteristics for detecting anomalies. In particular, the proposed model learns the internal characteristics of normal trajectories and clusters of normal trajectory representations in a latent space. To learn the internal characteristics of normal trajectories, the encoder of transformer with a self-attention mechanism first encodes trajectories into sequences of embedding vectors of trajectory points in the latent space. Then, a decoder reconstructs the trajectories from the latent space. In addition, to learn clusters of normal trajectory representations in the latent space, the self-organizing map (SOM) layer is used, which gets its input as the output of the transformer encoder. In the training phase, the TENSO model is trained using a total loss of trajectory reconstruction and SOM losses. In the anomaly detection phase, a test trajectory is evaluated to determine whether it is an anomaly based on trajectory reconstruction errors and the quantization error on the SOM. In this phase, a new metric is proposed which, namely WS, is the weighted sum of recall and precision to choose the appropriate threshold for detecting anomalies. The TENSO model-based framework is evaluated using two real trajectory datasets: MIT Badge and sCREEN. Experimental results show that the proposed framework identifies anomalies effectively and outperforms the baselines.

关键词： Trajectory transformers Self-organizing feature maps Prototypes Anomaly detection Training Measurement Human activity recognition Anomalous trajectory detection indoor human trajectory transformer encoder self-organizing map

来源：评论

学校读者我要写书评

暂无评论

A personalized paper recommendation method based on knowledge graph and transformer encoder with a self-attention mechanism

引用

APPLIED INTELLIGENCE 2023年第24期53卷 29991-30008页

作者： Gao, Li Lan, Yu Yu, Zhen Zhu, Jian-min Univ Shanghai Sci & Technol Lib & Dept Comp Sci & Engn Shanghai 200093 Peoples R China Univ Shanghai Sci & Technol Dept Comp Sci & Engn Shanghai 200093 Peoples R China Shanghai Datong High Sch Shanghai 200011 Peoples R China Univ Shanghai Sci & Technol Sch Mech Engn Shanghai 200093 Peoples R China

Paper recommendation with personalized methods helps researchers to track the latest academic trends and master cutting-edge academic trends efficiently. Meanwhile, the methods of previous paper recommendation suffer from three problems: data sparsity of content-based and collaborative filtering methods;Graph-based recommendations do not fully consider the personalized information of authors and their articles;Cold start based on deep learning. To overcome those difficulties, we propose a personalized paper recommendation method based on a knowledge graph and transformer encoder (KGTE) with a self-attention mechanism. Firstly, we add auxiliary information (article title, publication year, citation times, and abstract) as attributes to the nodes of knowledge graph(KG), which contain author, digital object unique identifier(DOI) and keywords. Secondly, BERT is used to represent the semantic information features of the article and transformer is introduced to fully integrate the feature context. After that, by using RippleNet, we traverse the knowledge graph, filter the user preference distribution and form a set of pre recommended nodes with multi_hop nodes. Finally, the prediction layer sorts the set and gets a Top_n paper recommendation. In the experiments on the DBLP and Aminer datasets, the precision value of KGTE improved by an average of 2.59% over the existing baseline methods DER and 4.23% improvement in NDCG.

关键词： Knowledge graph Self-attention mechanism BERT pre-training transformer encoder Paper recommendation

来源：评论

学校读者我要写书评

暂无评论

Remaining useful life estimation via transformer encoder enhanced by a gated convolutional unit

引用

JOURNAL OF INTELLIGENT MANUFACTURING 2021年第7期32卷 1997-2006页

作者： Mo, Yu Wu, Qianhui Li, Xiu Huang, Biqing Tsinghua Univ Dept Automat Beijing Natl Res Ctr Informat Sci & Technol Beijing 100084 Peoples R China Tsinghua Shenzhen Int Grad Sch Div Informat Sci Shenzhen 518055 Peoples R China

Remaining Useful Life (RUL) estimation is a fundamental task in the prognostic and health management (PHM) of industrial equipment and systems. To this end, we propose a novel approach for RUL estimation in this paper, based on deep neural architecture due to its great success in sequence learning. Specifically, we take the transformer encoder as the backbone of our model to capture short- and long-term dependencies in a time sequence. Compared with convolutional neural network based methods, there is no limitation from the kernel size for a complete receptive field of all time steps. While compared with recurrent neural network based methods, we develop our model based on dot-product self-attention, enabling it to fully exploit parallel computation. Moreover, we further propose a gated convolutional unit to facilitate the model's ability of incorporating local contexts at each time step, for the attention mechanism used in the transformer encoder makes the output high-level features insensitive to local contexts. We conduct experiments on the C-MAPSS datasets and show that, the performance of our model is superior or comparable to those of other existing methods. We also carry out ablation studies and demonstrate the necessity and effectiveness of each component used in the proposed model.

关键词： Remaining useful life estimation transformer encoder Gated convolutional unit Turbofan engine

来源：评论

学校读者我要写书评

暂无评论

Tran-DSR: A hybrid model for dysarthric speech recognition using transformer encoder and ensemble learning

引用

APPLIED ACOUSTICS 2024年 222卷

作者： Mahum, Rabbia El-Sherbeeny, Ahmed M. Alkhaledi, Khaled Hassan, Haseeb Univ Engn & Technol Taxila Dept Comp Sci Taxila 47050 Pakistan King Saud Univ Ind Engn Dept Coll Engn Ind Engn Dept POB 800 Riyadh 11421 Saudi Arabia King Salman Ctr Disabil Res POB 94682 Riyadh 11614 Saudi Arabia Kuwait Univ Coll Engn & Petr Ind & Management Syst Engn Dept POB 5969 Safat 13060 Kuwait Shenzhen Technol Univ Coll Hlth Sci & Environm Engn Shenzhen 518118 Peoples R China

Over the last decade, there has been a notable increase in the pervasiveness of neurological diseases due to population growth and aging. Among individuals with conditions like stroke, Parkinson's disease, cerebral palsy, and other neurological indications, dysarthria commonly manifests. Dysarthria can have a detrimental impact on functional communication, often leading to significantly reduced quality of life;also, ineffective communication has been behind significant health and safety incidents and interferes with people's productivity. Timely detecting and treating dysarthria in these patients is crucial to effectively manage their disease progression. Failure to do so can lead to challenges in disease management and may have adverse effects on the patient's psychological and physiological well-being as symptoms worsen. Many previous studies focused on dysarthria speech detection by employing several machine learning or deep learning techniques as classification tools. In this work, we propose a hybrid model, namely Tran-DSR, in which the strengths of ensemble deep networks and the transformer encoder scheme are combined. Ensemble learning as a backbone plays a crucial role in extracting powerful features from the mel-spectrograms. Two scenarios are considered: ensemble 1 (E1), which includes VGG16, DenseNet201, and GoogleNet, and ensemble 2 (E2), comprising InceptionResNetV2, DenseNet201, and Xception. On the other hand, the transformer encoder is constructed utilizing the self-attention approach, which allows the network to focus on relevant information, along with a multilayer perceptron for precise speech recognition. By leveraging this hybrid approach, accurate and efficient disease identification can be achieved. Experimental outcomes demonstrate that the Tran-DSR model achieves the highest accuracy of 99.18%, surpassing the performance of other research models.

关键词： Aging Ensemble learning Dysarthria speech recognition transformer encoder Deep learning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：