Neural network-based small cursor detection for embedded assistive technology

作     者：Darmawan, Jeremie Theddy Sigalingging, Xanno Kharis Faisal, Muhamad Leu, Jenq-Shiou Ratnasari, Nanda Rizqia Pradana 

作者机构：Department of Electronic and Computer Engineering National Taiwan University of Science and Technology No. 43 Section 4 Keelung Rd Da’an District Taipei City106 Taiwan Department of Bioinformatics School of Life Sciences Indonesia International Institute for Life Sciences Jl. Pulomas Barat No.Kav. 88 RT.4/RW.9 Kayu Putih Kec. Pulo GadungSpecial Capital Region of Jakarta East Jakarta13210 Indonesia Department of Biochemistry Yong Loo Lin School of Medicine National University of Singapore C/O MD7 Level 2 8 Medical Drive Singapore117597 Singapore 

出 版 物：《Visual Computer》 (Visual Comput)

年 卷 期：2024年第40卷第12期

页      面：8425-8439页

核心收录：

学科分类：0710[理学-生物学] 08[工学] 0835[工学-软件工程] 0836[工学-生物工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

主　　题：Assistive technology 

摘      要：Assistive technology (AT) is invaluable to people with special educational needs and disabilities, enabling them to interact with computers more efficiently. It is able to improve the interactivity between humans and computers for learning purposes. However, there is a lack of accessibility and interactivity with touchscreen devices and visual content that AT users encounter. These issues are critical in providing much-needed assistance and enhancing the daily lives of disabled individuals. Therefore, we propose an AT framework based on neural networks using embedded systems for improving the accessibility and interactivity of AT users. The proposed framework utilizes the Jetson Nano and is mainly used with a speech-to-intent neural network to process speech and move cursors. When improved with cursor object detection, the framework could obtain the location of the cursors in external displays and move the cursors of other devices. Since cursor datasets are very limited and not many detection models are up for the task, we investigated the use of Slicing Aided Hyper Inference (SAHI) pipeline along with two fine-tuned models, Fully Convoluted One-Stage (FCOS) and Task-aligned One-stage Object Detection (TOOD), to identify the minimum data required for these models to work optimally. With less than 120 annotated images and a data multiplier of 5 and 30, both models were able to achieve ~ 52 and ~ 60mAP, respectively. These results were comparable to performance on other small object detection datasets. In addition, we also present a working proof-of-concept for our proposed embedded assistive technology framework. © The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024.