CMCL: Cross-Modal Compressive Learning for Resource-Constrained Intelligent IoT Systems

作     者：Chen, Bin Tang, Dong Huang, Yujun An, Baoyi Wang, Yaowei Wang, Xuan 

作者机构：Harbin Inst Technol Shenzhen Sch Comp Sci & Technol Shenzhen 518055 Guangdong Peoples R China Guangdong Prov Key Lab Novel Secur Intelligence Te Shenzhen Peoples R China Peng Cheng Lab Res Ctr Artificial Intelligence Shenzhen 518055 Guangdong Peoples R China Tsinghua Univ Tsinghua Shenzhen Int Grad Sch Shenzhen 518055 Guangdong Peoples R China Huawei Technol Co Ltd Network Technol Lab Shenzhen 518055 Peoples R China 

出 版 物：《IEEE INTERNET OF THINGS JOURNAL》 (IEEE Internet Things J.)

年 卷 期：2024年第11卷第15期

页      面：25534-25542页

核心收录：

学科分类：0810[工学-信息与通信工程] 0808[工学-电气工程] 08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：National Natural Science Foundation of China [62301189, 62376037] Guangdong Basic and Applied Basic Research Foundation [2021A1515110066] Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies [2022B1212010005] Shenzhen Science and Technology Program [JCYJ20220818101012025, RCBS20221008093124061, GXWD20220811172936001, GXWD20220811170225001] PCNL KEY Project [PCL2023AS6-1] 

主　　题：Compressed sensing Task analysis Internet of Things Image coding Transformers Servers Privacy Compressive learning (CL) image captioning Internet of Things (IoT) transformer 

摘      要：Compressive learning (CL) has proven to be highly successful in executing joint signal sampling and inference for intricate vision tasks through resource-limited Internet of Things (IoT) devices. Recent studies have turned their attention toward utilizing the deep neural networks (DNNs) methodology, also known as DeepCL, to enhance performance in unimodal vision tasks. This approach incorporates learnable compressed sensing in a comprehensive, end-to-end manner. Current DeepCL techniques typically employ initial signal reconstruction as the input for subsequent DNNs for inference. However, this practice presents potential risks, such as privacy breaches and reduced performance due to information processing inequality. To address these issues, this article introduces the first cross-modal CL (CMCL) approach that enables image captioning directly on compressed measurements. When compared to previous DeepCL strategies, the proposed CMCL offers significant improvements in computational efficiency and privacy protection. Extensive experiments demonstrate that CMCL performance is nearly on par with leading image captioning methods, showcasing a metric value that is merely 2.75% lower than the uncompressed method when the data is compressed eightfold.