检索结果-内蒙古大学图书馆

Enhancing multivariate time-series anomaly detection with positional encoding mechanisms in transformers

JOURNAL OF SUPERCOMPUTING 2025年第1期81卷 1-27页

作者： Alioghli, Abdul Amir Okay, Feyza Yildirim Gazi Univ Dept Comp Sci Ankara Turkiye Gazi Univ Dept Comp Engn Ankara Turkiye

The surge in automation driven by IoT devices has generated extensive time-series data with highly variable features, posing challenges in anomaly detection. DL, particularly Transformer networks, has shown promise in addressing these issues. However, Transformer networks struggle with accurately determining the position of data points and maintaining the order of data in sequences, leading to the development of positional encoding (PE). Initially, Absolute PE was introduced, but newer methods like Relative PE and Rotary PE have been adopted in natural language processing tasks to improve performance. This study evaluates the potential of PEs including Absolute PE, Rotary PE, and two modifications of Relative PE methods (Representative attention and Global attention), for multivariate time-series anomaly detection problems. The experimental results indicate that Absolute PE, with a 98% accuracy score, performs well across different window sizes. Representative attention, with a 98% F1-score, performs best for short sequences (8, 16, and 32);whereas, Global attention, with a 97% F1-score, is more effective for longer sequences (64 and 128). Additionally, Absolute PE has the shortest training times, starting at 25 for sequence length 8 and increasing to 192 for length 128. Rotary PE also has slightly longer training times compared to Absolute PE. On the other hand, Representative attention consistently has the longest times, starting at 48 for length 8 and reaching 366 for length 128. Overall, Absolute PE and Global attention are the most time-efficient;while, Representative attention has significantly higher training times, particularly for long sequences.

关键词： positional encoding Transformer Internet of things Multivariate time-series Anomaly detection

来源：评论

学校读者我要写书评

暂无评论

F-StrIPE: Fast Structure-Informed positional encoding for Symbolic Music Generation

F-StrIPE: Fast Structure-Informed Positional Encoding for Sy...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Agarwal, Manvi Wang, Changhong Richard, Gaël LTCI Télécom Paris Institut Polytechnique de Paris France

ISBN: (纸本)9798350368741

While music remains a challenging domain for generative models like Transformers, recent progress has been made by exploiting suitable musically-informed priors. One technique to leverage information about musical structure in Transformers is inserting such knowledge into the positional encoding (PE) module. However, Transformers carry a quadratic cost in sequence length. In this paper, we propose F-StrIPE, a structure-informed PE scheme that works in linear complexity. Using existing kernel approximation techniques based on random features, we show that F-StrIPE is a generalization of Stochastic positional encoding (SPE). We illustrate the empirical merits of F-StrIPE using melody harmonization for symbolic music. © 2025 IEEE.

关键词： kernels music generation positional encoding symbolic music transformers

来源：评论

学校读者我要写书评

暂无评论

Memory positional encoding for image captioning

引用

SIGNAL PROCESSING-IMAGE COMMUNICATION 2025年 130卷

作者： Yang, Xiaobao He, Shuai Zhang, Jie Ma, Sugang Hou, Zhiqiang Sun, Wei Xian Univ Posts & Telecommun Xian Peoples R China Northwestern Polytech Univ Xian Peoples R China Changzhou Sanxin Power Supply Serv Co Ltd Changzhou Peoples R China

Transformer-based architectures represent the state-of-the-art in image captioning. Due to its natural parallel internal structure, it cannot be aware of the order of inputting tokens, so the positional encoding becomes an indispensable component of Transformer-based models. However, most of the existing absolute positional encodings (APE) have certain limitations for image captioning. Their spatial positional features are predefined and cannot been well generalized to other forms of data, such as visual data. Meanwhile, each positional features are decoupled from each other and lack internal correlation, therefore which affects the accuracy of spatial position context representation of visual or text semantic to a certain extent. Therefore, we propose a memory positional encoding (MPE), which has generalization ability that can be applied to both the visual encoder and the sequence decoder of the image captioning models. In MPE, each positional feature is recursively generated by the learnable network with memory function, making the current generated positional features effectively inherit the genetic information of the previous n positions. In addition, existing positional encodings provide positional features with fixed value and scale, that means, they provide the same positional encoding for different inputs, which is unreasonable. Thus, to address the previous issues of scale and value of current positional encoding methods in practical applications, we further explore dynamic memory positional encoding (DMPE) based on MPE. DMPE dynamically adjusts and generates positional features based on different input to provide them with unique positional representation. Extensive experiments on the MSCOCO validate the effectiveness of MPE and DMPE.

关键词： positional encoding Image captioning Transformer LSTM

来源：评论

学校读者我要写书评

暂无评论

Multi-Level Knowledge Distillation with positional encoding Enhancement

引用

PATTERN RECOGNITION 2025年 163卷

作者： Xu, Lixiang Wang, Zhiwen Bai, Lu Ji, Shengwei Ai, Bing Wang, Xiaofeng Yu, Philip S. Hefei Univ Sch Artificial Intelligence & Big Data Hefei Anhui Peoples R China Beijing Normal Univ Sch Artificial Intelligence Beijing Peoples R China Univ Illinois Dept Comp Sci Chicago IL USA

In recent years, Graph Neural Networks (GNNs) have achieved substantial success in addressing graph-related tasks. Knowledge Distillation (KD) has increasingly been adopted in graph learning as a classical technique for model compression and acceleration, enabling the transfer of predictive power from trained GNN models to lightweight, easily deployable Multi-Layer Perceptron (MLP) models. However, this approach often neglects node positional features and relies solely on trained GNN-generated labels to train MLPs based on node content features. Moreover, it heavily depends on local information aggregation, making it challenging to capture global graph structure and thereby limiting performance in node classification tasks. To address this issue, we propose Multi-Level Knowledge Distillation with positional encoding Enhancement (MLKD-PE). Our method employs positional encoding technique to generate node positional features, which are then combined with node content features to enhance the MLP's ability to perceive node positions. Additionally, we introduce a multi-level KD technique that aligns the final output of the student model with the teacher model's output, facilitating detailed knowledge transfer by incorporating intermediate layer outputs from the teacher model. Experimental results demonstrate that our method significantly improves classification accuracy across multiple datasets compared to the baseline model, confirming its superiority in node classification tasks.

关键词： Graph neural networks Multi-layer perceptrons Graph knowledge distillation positional encoding Efficient training and inference

来源：评论

学校读者我要写书评

暂无评论

PIONet: a positional encoding integrated onehot feature based RNA-binding protein classification using deep neural network

引用

IEEE Access 2025年 13卷 87220-87228页

作者： Rashid, Mahmood A. Chaturvedi, Mayank Paliwal, Kuldip K. Griffith University Signal Processing Laboratory School of Engineering and Built Environment Nathan QLD Australia Victoria University Institute for Sustainable Industries and Liveable Cities Melbourne VIC Australia

RNA-binding proteins (RBPs) are pivotal in regulating various biological processes, including RNA splicing and protein translation. Accurate classification of RBPs is crucial for understanding their functional roles in cellular mechanisms. Here we present PIONet, a deep learning method based on a convolutional neural network (CNN) that accurately classifies RBPs. Traditional one-hot encoding methods fail to capture positional relationships within RNA sequences. The PIONet addresses this limitation and enriches sequence representations with spatial information by integrating positional encoding with one-hot encoding. The CNN model processes these combined features to extract local patterns and motifs critical for RNA-protein interactions. The positional encoding enables the model to effectively learn inter-nucleotide dependencies within the sequence, making it highly suitable for capturing the intricate patterns associated with RBPs. Our model has been evaluated on 24 benchmark datasets, collectively referred to as RBP-24, demonstrating significant improvements in classification accuracy and robustness compared to the state-of-the-art methods. Additionally, we applied transfer learning to enhance the model’s performance, which led to improvements on 23 out of 24 datasets. This framework presents a powerful tool for advancing RBP classification tasks and offers potential applications in understanding post-transcriptional gene regulation. © 2013 IEEE.

关键词： Convolutional neural network (CNN) deep Learning positional encoding RNA-binding proteins transfer learning

来源：评论

学校读者我要写书评

暂无评论

2D-TPE: Two-Dimensional positional encoding Enhances Table Understanding for Large Language Models 34

2D-TPE: Two-Dimensional Positional Encoding Enhances Table U...

引用

34th ACM Web Conference, WWW 2025

作者： Li, Jia-Nan Guan, Jian Wu, Wei Yu, Zhengtao Yan, Rui Renmin University of China Beijing China Ant Group Beijing China Kunming University of Science and Technology Kunming China Wuhan University Wuhan China

ISBN: (纸本)9798400712746

Tables are ubiquitous across various domains for concisely representing structured information. Empowering large language models (LLMs) to reason over tabular data represents an actively explored direction. However, since typical LLMs only support one-dimensional (1D) inputs, existing methods often flatten the two-dimensional (2D) table structure into a sequence of tokens, which can severely disrupt the spatial relationships and result in an inevitable loss of vital contextual information. In this paper, we first empirically demonstrate the detrimental impact of such flattening operations on the performance of LLMs in capturing the spatial information of tables through two elaborate proxy tasks. Subsequently, we introduce a simple yet effective positional encoding method, termed “2D-TPE” (Two-Dimensional Table positional encoding), to address this challenge. 2D-TPE enables each attention head to dynamically select a permutation order of tokens within the context for attending to them, where each permutation represents a distinct traversal mode for the table, such as column-wise or row-wise traversal. 2D-TPE effectively mitigates the risk of losing essential spatial information while preserving computational efficiency, thus better preserving the table structure. Extensive experiments across five benchmarks demonstrate that 2D-TPE outperforms strong baselines, underscoring the importance of preserving the table structure for accurate table comprehension. Comprehensive analysis further reveals the substantially better scalability of 2D-TPE to large tables than baselines. © 2025 Copyright held by the owner/author(s).

关键词： Large Language Model positional encoding Table Understanding

来源：评论

学校读者我要写书评

暂无评论

positional encoding-based Resident Identification in Multi-resident Smart Homes

引用

ACM TRANSACTIONS ON INTERNET TECHNOLOGY 2024年第1期24卷 1-27页

作者： Song, Zhiyi Chaki, Dipankar Lakhdari, Abdallah Bouguettaya, Athman Univ Sydney Sch Comp Sci Sydney NSW 2006 Australia

We propose a novel resident identification framework to identify residents in a multi-occupant smart environment. The proposed framework employs a feature extraction model based on the concepts of positional encoding. The feature extraction model considers the locations of homes as a graph. We design a novel algorithm to build such graphs from layout maps of smart environments. The Node2Vec algorithm is used to transform the graph into high-dimensional node embeddings. A Long Short-Term Memory model is introduced to predict the identities of residents using temporal sequences of sensor events with the node embeddings. Extensive experiments show that our proposed scheme effectively identifies residents in a multi-occupant environment. Evaluation results on two real-world datasets demonstrate that our proposed approach achieves 94.5% and 87.9% accuracy, respectively.

关键词： Resident Identification Smart Homes positional encoding Node2Vec Node Embeddings LSTM

来源：评论

学校读者我要写书评

暂无评论

positional trace encoding for next activity prediction in event logs

引用

KNOWLEDGE-BASED SYSTEMS 2025年 319卷

作者： Pellicani, Antonio Ceci, Michelangelo Univ Bari Dept Comp Sci Via Orabona 4 I-70125 Bari Italy Natl Interuniv Consortium Informat CINI Data Sci Lab Via Volturno 58 Roma TX 00185 USA Jozef Stefan Inst Jamova 39 Ljubljana 1000 Slovenia

The analysis of log data, generated by running processes in many application domains, enables organizations identify opportunities for operational improvements. For instance, in healthcare, analyzing patient treatment logs can optimize care pathways;in manufacturing, production line logs can reveal bottlenecks;and in customer service, ticket resolution logs can streamline response protocols. One key analytical task is predicting next activity in a process, which supports operational decision-making through better resource allocation proactive response to customer needs. In this paper, we solve the next activity prediction task by exploiting a novel positional encoding approach that is based on sliding windows. This approach allows us to both a way to adapt to changes in the data distribution, and exploit positional information of the activities in the traces. The method proposed in this paper, called OREO, takes into account these aspects through positional encoding tightly coupled with specific types of deep neural network architectures. The results eight real-world process logs show the superiority of the models exploiting OREO encoding over state-of-the-art approaches, confirming our initial intuition of benefits gained by combining a time-window based model positional information.

关键词： Process mining Next activity prediction positional encoding

来源：评论

学校读者我要写书评

暂无评论

An attention mechanism model based on positional encoding for the prediction of ship maneuvering motion in real sea state

引用

JOURNAL OF MARINE SCIENCE AND TECHNOLOGY 2024年第1期29卷 136-152页

作者： Dong, Lei Wang, Hongdong Lou, Jiankun Shanghai Jiao Tong Univ MOE Key Lab Marine Intelligent Equipment & Syst Shanghai 200240 Peoples R China

This paper proposes an positional encoding-based attention mechanism model which can quantify the temporal correlation of ship maneuvering motion to predict the future ship motion in real sea state. To represent the temporal information of the sequential motion status, the positional encoding consisted by sine and cosine functions of different frequencies is chosen as the input of the model. First, the reasonableness of the improved architecture of the model is validated on the standard turning test datasets of an unmanned surface vehicle. Then, the absolute positional encoding based-scaled-dot product attention mechanism model is compared with other two attention mechanism models with different positional encoding and attention calculation methods and its superiority is verified. As demonstrated by exhaustive experiments, the model has the highest prediction accuracy when the input sequence length equals the output sequence length and the accuracy defined in this paper of the model will drop to less than 90% when the predicted length exceeds 45. Finally, the attention mechanism model is compared with the LSTM model with different lengths of input sequences to demonstrate that the attention mechanism model has a faster training speed when dealing with long sequences.

关键词： Maneuvering motion prediction Neural network Attention mechanism positional encoding

来源：评论

学校读者我要写书评

暂无评论

A hybrid MLP-CNN model based on positional encoding for daytime radiative cooler

引用

OPTICS COMMUNICATIONS 2024年 560卷

作者： Wang, Xueyu Chen, Shuo Chen, Lei Zhu, Danfeng Liu, Yumin Wu, Tiesheng Beijing Univ Posts & Telecommun State Key Lab Informat Photon & Opt Commun 10 Xitucheng Rd Beijing 100876 Peoples R China Jiaying Univ Sch Phys & Elect Engn Meizhou 514015 Peoples R China Guilin Univ Elect Technol Coll Informat & Commun Engn 1 Jinji Rd Guilin 541004 Peoples R China

Radiative cooling is a groundbreaking technology for sustainable temperature regulation to combat global warming and the urban heat island effect. Traditional optimization algorithms for cooling system structures are often brute-force and even limited to local optimal solutions. Here, we propose a joint simulation method based on a multilayer perceptron (MLP) and a convolutional neural network (CNN) with position encoding (PE), namely the hybrid MLP-CNN model with PE. This method can significantly reduce simulation time, avoid getting trapped in local optima, and accurately predict the optical response and radiative cooling power of the structure. Applying PE enables the MLP model to learn the relationship between structural parameters and optical response more quickly and effectively, reducing loss during training and more minor root mean square error (RMSE) and mean relative error (MRE) during testing. In addition, the residual network can reduce the problem of overfitting in the network. Compared to using only an MLP, the hybrid MLP-CNN with PE model can effectively reduce loss and achieve more accurate prediction outputs. The hybrid model is much faster than traditional electromagnetic simulations, taking only 1% of the time, and the model dramatically reduces the time needed to predict the optical response of structures and effectively decreases the cost of designing optical devices. Utilizing this hybrid model not only avoids the potential drawbacks of being limited to suboptimal solutions and significantly reduces the simulation time, but also effectively predicts the optical response of the device. This approach can demonstrate remarkable adaptability in optimizing geometric parameters of various optical apparatuses, including polarization converters, beam splitters, metalens, and other two-dimensional nanostructures.

关键词： positional encoding Radiative cooling Multilayer perceptron Convolutional neural network

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：