检索结果-内蒙古大学图书馆

Enhancing multivariate time-series anomaly detection with positional encoding mechanisms in transformers

JOURNAL OF SUPERCOMPUTING 2025年第1期81卷 1-27页

作者： Alioghli, Abdul Amir Okay, Feyza Yildirim Gazi Univ Dept Comp Sci Ankara Turkiye Gazi Univ Dept Comp Engn Ankara Turkiye

The surge in automation driven by IoT devices has generated extensive time-series data with highly variable features, posing challenges in anomaly detection. DL, particularly Transformer networks, has shown promise in addressing these issues. However, Transformer networks struggle with accurately determining the position of data points and maintaining the order of data in sequences, leading to the development of positional encoding (PE). Initially, Absolute PE was introduced, but newer methods like Relative PE and Rotary PE have been adopted in natural language processing tasks to improve performance. This study evaluates the potential of PEs including Absolute PE, Rotary PE, and two modifications of Relative PE methods (Representative attention and Global attention), for multivariate time-series anomaly detection problems. The experimental results indicate that Absolute PE, with a 98% accuracy score, performs well across different window sizes. Representative attention, with a 98% F1-score, performs best for short sequences (8, 16, and 32);whereas, Global attention, with a 97% F1-score, is more effective for longer sequences (64 and 128). Additionally, Absolute PE has the shortest training times, starting at 25 for sequence length 8 and increasing to 192 for length 128. Rotary PE also has slightly longer training times compared to Absolute PE. On the other hand, Representative attention consistently has the longest times, starting at 48 for length 8 and reaching 366 for length 128. Overall, Absolute PE and Global attention are the most time-efficient;while, Representative attention has significantly higher training times, particularly for long sequences.

关键词： positional encoding Transformer Internet of things Multivariate time-series Anomaly detection

来源：评论

学校读者我要写书评

暂无评论

F-StrIPE: Fast Structure-Informed positional encoding for Symbolic Music Generation

F-StrIPE: Fast Structure-Informed Positional Encoding for Sy...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Agarwal, Manvi Wang, Changhong Richard, Gaël LTCI Télécom Paris Institut Polytechnique de Paris France

ISBN: (纸本)9798350368741

While music remains a challenging domain for generative models like Transformers, recent progress has been made by exploiting suitable musically-informed priors. One technique to leverage information about musical structure in Transformers is inserting such knowledge into the positional encoding (PE) module. However, Transformers carry a quadratic cost in sequence length. In this paper, we propose F-StrIPE, a structure-informed PE scheme that works in linear complexity. Using existing kernel approximation techniques based on random features, we show that F-StrIPE is a generalization of Stochastic positional encoding (SPE). We illustrate the empirical merits of F-StrIPE using melody harmonization for symbolic music. © 2025 IEEE.

关键词： kernels music generation positional encoding symbolic music transformers

来源：评论

学校读者我要写书评

暂无评论

Memory positional encoding for image captioning

引用

SIGNAL PROCESSING-IMAGE COMMUNICATION 2025年 130卷

作者： Yang, Xiaobao He, Shuai Zhang, Jie Ma, Sugang Hou, Zhiqiang Sun, Wei Xian Univ Posts & Telecommun Xian Peoples R China Northwestern Polytech Univ Xian Peoples R China Changzhou Sanxin Power Supply Serv Co Ltd Changzhou Peoples R China

Transformer-based architectures represent the state-of-the-art in image captioning. Due to its natural parallel internal structure, it cannot be aware of the order of inputting tokens, so the positional encoding becomes an indispensable component of Transformer-based models. However, most of the existing absolute positional encodings (APE) have certain limitations for image captioning. Their spatial positional features are predefined and cannot been well generalized to other forms of data, such as visual data. Meanwhile, each positional features are decoupled from each other and lack internal correlation, therefore which affects the accuracy of spatial position context representation of visual or text semantic to a certain extent. Therefore, we propose a memory positional encoding (MPE), which has generalization ability that can be applied to both the visual encoder and the sequence decoder of the image captioning models. In MPE, each positional feature is recursively generated by the learnable network with memory function, making the current generated positional features effectively inherit the genetic information of the previous n positions. In addition, existing positional encodings provide positional features with fixed value and scale, that means, they provide the same positional encoding for different inputs, which is unreasonable. Thus, to address the previous issues of scale and value of current positional encoding methods in practical applications, we further explore dynamic memory positional encoding (DMPE) based on MPE. DMPE dynamically adjusts and generates positional features based on different input to provide them with unique positional representation. Extensive experiments on the MSCOCO validate the effectiveness of MPE and DMPE.

关键词： positional encoding Image captioning Transformer LSTM

来源：评论

学校读者我要写书评

暂无评论

Multi-Level Knowledge Distillation with positional encoding Enhancement

引用

PATTERN RECOGNITION 2025年 163卷

作者： Xu, Lixiang Wang, Zhiwen Bai, Lu Ji, Shengwei Ai, Bing Wang, Xiaofeng Yu, Philip S. Hefei Univ Sch Artificial Intelligence & Big Data Hefei Anhui Peoples R China Beijing Normal Univ Sch Artificial Intelligence Beijing Peoples R China Univ Illinois Dept Comp Sci Chicago IL USA

In recent years, Graph Neural Networks (GNNs) have achieved substantial success in addressing graph-related tasks. Knowledge Distillation (KD) has increasingly been adopted in graph learning as a classical technique for model compression and acceleration, enabling the transfer of predictive power from trained GNN models to lightweight, easily deployable Multi-Layer Perceptron (MLP) models. However, this approach often neglects node positional features and relies solely on trained GNN-generated labels to train MLPs based on node content features. Moreover, it heavily depends on local information aggregation, making it challenging to capture global graph structure and thereby limiting performance in node classification tasks. To address this issue, we propose Multi-Level Knowledge Distillation with positional encoding Enhancement (MLKD-PE). Our method employs positional encoding technique to generate node positional features, which are then combined with node content features to enhance the MLP's ability to perceive node positions. Additionally, we introduce a multi-level KD technique that aligns the final output of the student model with the teacher model's output, facilitating detailed knowledge transfer by incorporating intermediate layer outputs from the teacher model. Experimental results demonstrate that our method significantly improves classification accuracy across multiple datasets compared to the baseline model, confirming its superiority in node classification tasks.

关键词： Graph neural networks Multi-layer perceptrons Graph knowledge distillation positional encoding Efficient training and inference

来源：评论

学校读者我要写书评

暂无评论

positional encoding-based Resident Identification in Multi-resident Smart Homes

引用

ACM TRANSACTIONS ON INTERNET TECHNOLOGY 2024年第1期24卷 1-27页

作者： Song, Zhiyi Chaki, Dipankar Lakhdari, Abdallah Bouguettaya, Athman Univ Sydney Sch Comp Sci Sydney NSW 2006 Australia

We propose a novel resident identification framework to identify residents in a multi-occupant smart environment. The proposed framework employs a feature extraction model based on the concepts of positional encoding. The feature extraction model considers the locations of homes as a graph. We design a novel algorithm to build such graphs from layout maps of smart environments. The Node2Vec algorithm is used to transform the graph into high-dimensional node embeddings. A Long Short-Term Memory model is introduced to predict the identities of residents using temporal sequences of sensor events with the node embeddings. Extensive experiments show that our proposed scheme effectively identifies residents in a multi-occupant environment. Evaluation results on two real-world datasets demonstrate that our proposed approach achieves 94.5% and 87.9% accuracy, respectively.

关键词： Resident Identification Smart Homes positional encoding Node2Vec Node Embeddings LSTM

来源：评论

学校读者我要写书评

暂无评论

A hybrid MLP-CNN model based on positional encoding for daytime radiative cooler

引用

OPTICS COMMUNICATIONS 2024年 560卷

作者： Wang, Xueyu Chen, Shuo Chen, Lei Zhu, Danfeng Liu, Yumin Wu, Tiesheng Beijing Univ Posts & Telecommun State Key Lab Informat Photon & Opt Commun 10 Xitucheng Rd Beijing 100876 Peoples R China Jiaying Univ Sch Phys & Elect Engn Meizhou 514015 Peoples R China Guilin Univ Elect Technol Coll Informat & Commun Engn 1 Jinji Rd Guilin 541004 Peoples R China

Radiative cooling is a groundbreaking technology for sustainable temperature regulation to combat global warming and the urban heat island effect. Traditional optimization algorithms for cooling system structures are often brute-force and even limited to local optimal solutions. Here, we propose a joint simulation method based on a multilayer perceptron (MLP) and a convolutional neural network (CNN) with position encoding (PE), namely the hybrid MLP-CNN model with PE. This method can significantly reduce simulation time, avoid getting trapped in local optima, and accurately predict the optical response and radiative cooling power of the structure. Applying PE enables the MLP model to learn the relationship between structural parameters and optical response more quickly and effectively, reducing loss during training and more minor root mean square error (RMSE) and mean relative error (MRE) during testing. In addition, the residual network can reduce the problem of overfitting in the network. Compared to using only an MLP, the hybrid MLP-CNN with PE model can effectively reduce loss and achieve more accurate prediction outputs. The hybrid model is much faster than traditional electromagnetic simulations, taking only 1% of the time, and the model dramatically reduces the time needed to predict the optical response of structures and effectively decreases the cost of designing optical devices. Utilizing this hybrid model not only avoids the potential drawbacks of being limited to suboptimal solutions and significantly reduces the simulation time, but also effectively predicts the optical response of the device. This approach can demonstrate remarkable adaptability in optimizing geometric parameters of various optical apparatuses, including polarization converters, beam splitters, metalens, and other two-dimensional nanostructures.

关键词： positional encoding Radiative cooling Multilayer perceptron Convolutional neural network

来源：评论

学校读者我要写书评

暂无评论

HDTSLR: A Framework Based on Hierarchical Dynamic positional encoding for Sign Language Recognition

引用

IEEE TRANSACTIONS ON MOBILE COMPUTING 2024年第5期23卷 5631-5643页

作者： Zhang, Jiangtao Wang, Qingshan Wang, Qi Hefei Univ Technol Sch Math Hefei 230601 Anhui Peoples R China

Sign language is the basic way for people with hearing impairment to communicate, and sign language recognition (SLR) could effectively help in this regard. Mainstream Transformer-based SLR requires positional encoding to sense the positional information of the data. However, existing PE methods globally encode the sign data result in weaken or even ignoring the sequence variation within the gestures. This article proposes HDTSLR: A Transformer-based SLR framework built on hierarchical dynamic positional encoding (HDPE) enhances individual gesture sequence features while preserving the sign overall temporal features. HDPE designs semantic positional encoding utilizing predefined scale functions with trainable biases to emphasize sign semantic relationships. The t-distribution is used by the designed lexical positional encoding to explore the unique variation of gestures. Before the HDPE operation, the sign language data is split into equal-length feature clips while feature extraction and chunking are performed by the autoencoder. The feature clips with significant changes in gesture chunk are further selected and aggregated with the remaining ones by deforming Gram matrix. In addition, HDTSLR is evaluated on the one-handed and two-handed datasets, achieving word error rates of 16.59% and 21.67%, respectively. Comparison experiments show that it outperforms known SLR methods in both accuracy and robustness.

关键词： positional encoding sign language recognition t-distribution transformer wearable computing

来源：评论

学校读者我要写书评

暂无评论

An attention mechanism model based on positional encoding for the prediction of ship maneuvering motion in real sea state

引用

JOURNAL OF MARINE SCIENCE AND TECHNOLOGY 2024年第1期29卷 136-152页

作者： Dong, Lei Wang, Hongdong Lou, Jiankun Shanghai Jiao Tong Univ MOE Key Lab Marine Intelligent Equipment & Syst Shanghai 200240 Peoples R China

This paper proposes an positional encoding-based attention mechanism model which can quantify the temporal correlation of ship maneuvering motion to predict the future ship motion in real sea state. To represent the temporal information of the sequential motion status, the positional encoding consisted by sine and cosine functions of different frequencies is chosen as the input of the model. First, the reasonableness of the improved architecture of the model is validated on the standard turning test datasets of an unmanned surface vehicle. Then, the absolute positional encoding based-scaled-dot product attention mechanism model is compared with other two attention mechanism models with different positional encoding and attention calculation methods and its superiority is verified. As demonstrated by exhaustive experiments, the model has the highest prediction accuracy when the input sequence length equals the output sequence length and the accuracy defined in this paper of the model will drop to less than 90% when the predicted length exceeds 45. Finally, the attention mechanism model is compared with the LSTM model with different lengths of input sequences to demonstrate that the attention mechanism model has a faster training speed when dealing with long sequences.

关键词： Maneuvering motion prediction Neural network Attention mechanism positional encoding

来源：评论

学校读者我要写书评

暂无评论

A novel hybrid model based on STL decomposition and one-dimensional convolutional neural networks with positional encoding for significant wave height forecast

引用

RENEWABLE ENERGY 2021年 173卷 531-543页

作者： Yang, Shaobo Deng, Zegui Li, Xingfei Zheng, Chongwei Xi, Lintong Zhuang, Jucheng Zhang, Zhenquan Zhang, Zhiyou Tianjin Univ State Key Lab Precis Measuring Technol & Instrume Tianjin 300072 Peoples R China Dalian Naval Acad Dalian 116018 Peoples R China Chinese Acad Sci Inst Atmospher Phys State Key Lab Numer Modeling Atmospher Sci & Geop Beijing 100029 Peoples R China Ocean Univ China Shandong Prov Key Lab Ocean Engn Qingdao 266100 Peoples R China Marine Resources & Environm Res Grp Maritime Silk Rd Dalian 116018 Peoples R China

Reducing the dependence on fossil fuels and utilizing the renewable energy have become essential due to the global resource exhaustion and unfriendly environmental impact from coal, petroleum and natural gas. Therefore, the rising attention has been paid to wave energy characterized by sustainability, clean, high energy density and extensive distribution. As one of the most important parameters of wave energy, significant wave height (SWH) is difficult to forecast accurately due to the complex marine condition and ubiquitous presence of chaos in nature. In this research, a novel hybrid model called STL-CNN-PE which combines seasonal-trend decomposition procedure based on loess (STL) and one-dimensional convolutional neural networks (CNN) with positional encoding (PE) was proposed to forecast SWH effi-ciently and accurately. To evaluate the proposed model comprehensively, the hourly standard meteorology data at station 44007, 46087 and 51000 from NOAA's National Data Buoy Center were selected for model training and testing. The experimental results indicated that STL-CNN-PE provided more reliable forecasting values than the single model. Meanwhile, STL-CNN-PE had enormous advantage on speed and similar precision compared with EMD-LSTM. Finally, the experimental results revealed that the models provided better forecasting metrics at deeper waters. (c) 2021 Elsevier Ltd. All rights reserved.

关键词： Forecast STL Convolutional neural networks positional encoding

来源：评论

学校读者我要写书评

暂无评论

Structural positional encoding for knowledge integration in transformer-based medical process monitoring and trace classification

引用

PROGRESS IN ARTIFICIAL INTELLIGENCE 2024年 1-13页

作者： Irwin, Christopher Dossena, Marco Leonardi, Giorgio Montani, Stefania Univ Piemonte Orientale DISIT Comp Sci Inst Alessandria Italy Azienda Osped SS Antonio & Biagio & Cesare Arrigo Lab Integrato Intelligenza Artificiale & Informat Alessandria Italy

Predictive process monitoring is a process mining task aimed at forecasting information about a running process trace, such as the most correct next activity to be executed. In medical domains, predictive process monitoring can provide valuable decision support in atypical and nontrivial situations. On the other hand, process trace classification exploits the logged activity sequences to classify traces on the basis of some properties;in medical applications, this information can be fruitfully adopted for quality assessment and process/resource optimization. Decision support and quality assessment in medicine cannot ignore domain knowledge, in order to be grounded on all the available information (which is not limited to data) and to be really acceptable by end users. In this paper, we propose a predictive process monitoring approach relying on the use of a transformer, a deep learning architecture based on the attention mechanism. A major contribution of our work lies in the incorporation of ontological domain-specific knowledge, carried out through a graph positional encoding technique. Interestingly, the transformer-based knowledge-enhanced architecture has been adopted to cover the trace classification task as well, by incorporating the class label as the last trace activity to predict. The paper presents and discusses the encouraging experimental result we are collecting in the domain of stroke management.

关键词： Medical process monitoring Medical trace classification Domain knowledge Transformers Graphs positional encoding

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：