检索结果-内蒙古大学图书馆

CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Xinbei Zhang, Zhuosheng Zhao, Hai School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University China Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China

Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments, especially for graphical user interface (GUI) automation. However, those GUI agents require comprehensive cognition ability including exhaustive perception and reliable action response. We propose a Comprehensive Cognitive LLM Agent, CoCo-Agent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP), to systematically improve the GUI automation performance. First, CEP facilitates the GUI perception through different aspects and granularity, including screenshots and complementary detailed layouts for the visual channel and historical actions for the textual channel. Second, CAP decomposes the action prediction into sub-problems: action type prediction and action target conditioned on the action type. With our technical design, our agent achieves new state-of-the-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios. Code is available at https://***/xbmxb/CoCo-Agent. Copyright © 2024, The Authors. All rights reserved.

关键词： Graphical user interfaces

HCILab at Memotion 2.0 2022: Analysis of Sentiment, Emotion and Intensity of Emotion Classes from Meme Images using Single and Multi Modalities 1

学校读者我要写书评

暂无评论

HCILab at Memotion 2.0 2022: Analysis of Sentiment, Emotion ...

1st Workshop on Multimodal Fact-Checking and Hate Speech Detection, DE-FACTIFY 2022

作者： Nguyen, Thanh Tin Pham, Nhat Truong Nguyen, Ngoc Duy Nguyen, Hai Nguyen, Long H. Kim, Yong-Guk Human Computer Interaction Lab Department of Computer Engineering Sejong University Seoul Korea Republic of Division of Computational Mechatronics Institute for Computational Science Ton Duc Thang University Ho Chi Minh City Viet Nam Faculty of Electrical and Electronics Engineering Ton Duc Thang University Ho Chi Minh City Viet Nam Institute for Intelligent Systems Research and Innovation Deakin University Victoria Australia Khoury College of Computer Sciences Northeastern University Boston United States Faculty of Information Technology Ton Duc Thang University Ho Chi Minh City Viet Nam

Nowadays, memes found on internet are overwhelming. Although they are innocuous and sometimes entertaining, there exist memes that contain sarcasm, offensive, or motivational feelings. In this study, several approaches are proposed to solve the multiple modality problem in analysing the given meme dataset. The imbalance issue has been addressed by using a new Auto Augmentation method and the uncorrelation issue has been mitigated by adopting deep Canonical Correlation Analysis to find the most correlated projections of visual and textual feature embedding. In addition, both stacked attention and multi-hop attention network are employed to efficiently generate aggregated features. As a result, our team, i.e. HCILab, achieved a weighted F1 score of 0.4995 for sentiment analysis, 0.7414 for emotion classification, and 0.5301 for scale/intensity of emotion classes on the leaderboard. This results are obtained by using concatenation between image and text model and our code can be found at https://***/JMRa8. © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (***)

关键词： Sentiment analysis

How Deep is Love in LLMs’ Hearts? Exploring Semantic Size in human-like Cognition

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Yao, Yao Yang, Yifei Ma, Xinbei Yang, Dongjie Zhang, Zhuosheng Li, Zuchao Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 Shanghai China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University Shanghai China National Engineering Research Center for Multimedia Software School of Computer Science Wuhan University Wuhan University Wuhan430072 China

How human cognitive abilities are formed has long captivated researchers. However, a significant challenge lies in developing meaningful methods to measure these complex processes. With the advent of large language models (LLMs), which now rival human capabilities in various domains, we are presented with a unique testbed to investigate human cognition through a new lens. Among the many facets of cognition, one particularly crucial aspect is the concept of semantic size—the perceived magnitude of both abstract and concrete words or concepts. This study seeks to investigate whether LLMs exhibit similar tendencies in understanding semantic size, thereby providing insights into the underlying mechanisms of human cognition. We begin by exploring metaphorical reasoning, comparing how LLMs and humans associate abstract words with concrete objects of varying sizes. Next, we examine LLMs’ internal representations to evaluate their alignment with human cognitive processes. Our findings reveal that multi-modal training is crucial for LLMs to achieve more human-like understanding, suggesting that real-world, multi-modal experiences are similarly vital for human cognitive development. Lastly, we examine whether LLMs are influenced by attention-grabbing headlines with larger semantic sizes in a real-world web shopping scenario. The results show that multi-modal LLMs are more emotionally engaged in decision-making, but this also introduces potential biases, such as the risk of manipulation through clickbait headlines. Ultimately, this study offers a novel perspective on how LLMs interpret and internalize language, from the smallest concrete objects to the most profound abstract concepts like ‘love’. The insights gained not only improve our understanding of LLMs but also provide new avenues for exploring the cognitive abilities that define human intelligence. Our code and dataset is publicly available at https://***/Zoeyyao27/LLM-Semantic-Size-Understanding. © 2025, CC BY.

关键词： Semantics

Structural and functional neural correlates of sensorimotor deficits in progression of hepatic encephalopathy

学校读者我要写书评

暂无评论

Magnetic Resonance Letters 2024年

作者： Sha, Guanchen Liu, Yuefan Cao, Yutong Zhang, Qi Zhang, Yining Chen, Yuanyuan Fan, Qiuyun Cheng, Yue Department of Biomedical Engineering College of Precision Instruments and Optoelectronics Engineering Tianjin University Tianjin300072 China Tianjin Key Laboratory of Brain Science and Neuroengineering Tianjin300072 China Academy of Medical Engineering and Translational Medicine Tianjin University Tianjin300072 China Department of Medical Imaging Tianjin First Central Hospital Tianjin300192 China Haihe Laboratory of Brain-Computer Interaction and Human-Machine Integration Tianjin300392 China State Key Laboratory of Advanced Medical Materials and Devices Tianjin300072 China

Hepatic encephalopathy (HE) is a neurological condition that occurs as a complication of liver dysfunction that involves sensorimotor symptoms in addition to cognitive and behavioral changes, particularly in cases of severe liver disease or cirrhosis. Previous studies have reported spatially distributed structural and functional abnormalities related to HE, but the exact relationship between the structural and functional alterations with respect to disease progression remains unclear. In this study, we performed surface-based cortical thickness comparisons and functional connectivity (FC) analyses between three cross-sectional groups: healthy controls (HC, N = 51), patients with minimal hepatic encephalopathy (MHE, N = 50), patients with overt hepatic encephalopathy (OHE, N = 51). In addition to the distributed cortical thinning that is extensively thought to be associated with cognitive decline in HE, we found significant cortical thickening in the left parahippocampal gyrus cortex in the OHE group (p © 2024 The Authors

关键词： Diseases

Mitigating Misleading Chain-of-Thought Reasoning with Selective Filtering

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wu, Yexin Zhang, Zhuosheng Zhao, Hai College of Zhiyuan Shanghai Jiao Tong University China School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University China Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China

Large language models have manifested remarkable capabilities by leveraging chain-of-thought (CoT) reasoning techniques to solve intricate questions through step-by-step reasoning chains. Despite its success, the efficacy of such reasoning is inherently contingent upon the quality of CoT. However, flawless CoT reasoning cannot be guaranteed due to the presence of indecomposable questions and the potential for erroneous reasoning chains, particularly in the case of small-scale language models. To tackle this challenge, we propose a novel approach called the selective filtering reasoner (SelF-Reasoner) that assesses the entailment relationship between the question and the candidate reasoning chain. Then, we proceed with CoT reasoning when the reasoning chain demonstrates confidence;otherwise, we opt to predict the answer directly. SelF-Reasoner improves the fine-tuned T5 baseline consistently over the scienceQA, ECQA, and LastLetter tasks. Code is available at https://***/LibroWu/SelF-Reasoner. © 2024, CC BY-SA.

关键词： Computational linguistics

Towards Understanding Interactive Sonic Gastronomy with Chefs and Diners 25

学校读者我要写书评

暂无评论

Towards Understanding Interactive Sonic Gastronomy with Chef...

Proceedings of the 2025 CHI Conference on human Factors in Computing Systems

作者： Hongyue Wang Jialin Deng Linjia He Nathalie Overdevest Ryan Wee Yan Wang Phoebe O. Toups Dugas Don Samitha Elvitigala Florian Floyd Mueller Exertion Games Lab Department of Human-Centred Computing Monash University Melbourne Australia Exertion Games Lab Department of Human-Centred Computing Monash University Melbourne Australia and Diverse-ability Interaction Lab School of Computer Science Bristol United Kingdom Exertion Games Lab Department of Human-Centred Computing Monash University Melbourne Australia and Faculty of Engineering and Information Technology University of Technology Sydney Sydney Australia Exertion Games Lab Department of Human-Centred Computing Monash University Melbourne Australia and Mineral Resources Commonwealth Scientific and Industrial Research Organisation Melbourne Australia

On the Robustness of Editing Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Xinbei Ju, Tianjie Qiu, Jiyang Zhang, Zhuosheng Zhao, Hai Liu, Lifeng Wang, Yulong School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University China Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China Baichuan Intelligent Technology China

Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates. Model editing enables the manipulation of specific knowledge memories and the behavior of language generation without retraining. However, the robustness of model editing remains an open question. This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI. We focus on three key research questions. RQ1: Can edited LLMs behave consistently resembling communicative AI in realistic situations? RQ2: To what extent does the rephrasing of prompts lead LLMs to deviate from the edited knowledge memory? RQ3: Which knowledge features are correlated with the performance and robustness of editing? Our empirical studies uncover a substantial disparity between existing editing methods and the practical application of LLMs. On rephrased prompts that are flexible but common in realistic applications, the performance of editing experiences a significant decline. Further analysis shows that more popular knowledge is memorized better, easier to recall, and more challenging to edit effectively. Copyright © 2024, The Authors. All rights reserved.

关键词： Computational linguistics

Artificial intelligence for geoscience:Progress,challenges,and perspectives

学校读者我要写书评

暂无评论

The Innovation 2024年第5期5卷 136-160,135页

作者： Tianjie Zhao Sheng Wang Chaojun Ouyang Min Chen Chenying Liu Jin Zhang Long Yu Fei Wang Yong Xie Jun Li Fang Wang Sabine Grunwald Bryan MWong Fan Zhang Zhen Qian Yongjun Xu Chengqing Yu Wei Han Tao Sun Zezhi Shao Tangwen Qian Zhao Chen Jiangyuan Zeng Huai Zhang Husi Letu Bing Zhang Li Wang Lei Luo Chong Shi Hongjun Su Hongsheng Zhang Shuai Yin Ni Huang Wei Zhao Nan Li Chaolei Zheng Yang Zhou Changping Huang Defeng Feng Qingsong Xu Yan Wu Danfeng Hong Zhenyu Wang Yinyi Lin Tangtang Zhang Prashant Kumar Antonio Plaza Jocelyn Chanussot Jiabao Zhang Jiancheng Shi Lizhe Wang Aerospace Information Research Institute Chinese Academy of SciencesBeijing 100094China School of Computer Science China University of GeosciencesWuhan 430078China State Key Laboratory of Mountain Hazards and Engineering Resilience Institute of Mountain Hazards and EnvironmentChinese Academy of SciencesChengdu 610299China Key Laboratory of Virtual Geographic Environment(Ministry of Education of PRC) Nanjing Normal UniversityNanjing 210023China Data Science in Earth Observation Technical University of Munich80333 MunichGermany The National Key Laboratory of Water Disaster Prevention Yangtze Institute for Conservation and DevelopmentHohai UniversityNanjing 210098China Institute of Computing Technology Chinese Academy of SciencesBeijing 100190China School of Geographical Sciences Nanjing University of Information Science and TechnologyNanjing 210044China State Key Laboratory of Soil and Sustainable Agriculture Institute of Soil ScienceChinese Academy of SciencesNanjing 210008China Soil Water and Ecosystem Sciences DepartmentUniversity of FloridaPO Box 110290GainesvilleFLUSA Materials Science Engineering Program Cooperating Faculty Member in the Department of Chemistry and Department of Physics Astronomy University of CaliforniaCaliforniaRiversideCA 92521USA Institute of Remote Sensing and Geographical Information System School of Earth and Space SciencesPeking UniversityBeijing 100871China Key Laboratory of Computational Geodynamics University of Chinese Academy of SciencesBeijing 100049China International Research Center of Big Data for Sustainable Development Goals Beijing 100094China College of Geography and Remote Sensing Hohai UniversityNanjing 211100China Department of Geography The University of Hong KongHong Kong 999077SARChina Jiangsu Key Laboratory of Atmospheric Environment Monitoring and Pollution Control Nanjing 210044China School of Environmental Science and Engineering Nanjing University of Information Science&TechnologyNanjing 210044China Collaborative Inno

This paper explores the evolution of geoscientific inquiry,tracing the progression from traditional physics-based models to modern data-driven approaches facilitated by significant advancements in artificial intelligence(AI)and data collection *** models,which are grounded in physical and numerical frameworks,provide robust explanations by explicitly reconstructing underlying physical ***,their limitations in comprehensively capturing Earth’s complexities and uncertainties pose challenges in optimization and real-world *** contrast,contemporary data-driven models,particularly those utilizing machine learning(ML)and deep learning(DL),leverage extensive geoscience data to glean insights without requiring exhaustive theoretical *** techniques have shown promise in addressing Earth science-related ***,challenges such as data scarcity,computational demands,data privacy concerns,and the“black-box”nature of AI models hinder their seamless integration into *** integration of physics-based and data-driven methodologies into hybrid models presents an alternative *** models,which incorporate domain knowledge to guide AI methodologies,demonstrate enhanced efficiency and performance with reduced training data *** review provides a comprehensive overview of geoscientific research paradigms,emphasizing untapped opportunities at the intersection of advanced AI techniques and *** examines major methodologies,showcases advances in large-scale models,and discusses the challenges and prospects that will shape the future landscape of AI in *** paper outlines a dynamic field ripe with possibilities,poised to unlock new understandings of Earth’s complexities and further advance geoscience exploration.

关键词： Earth utilizing landscape

Static Force-Based Modeling and Parameter Estimation for a Deformable Link Composed of Passive Spherical Joints With Preload Forces

学校读者我要写书评

暂无评论

IEEE/CAA Journal of Automatica Sinica 2021年第11期8卷 1817-1826页

作者： Gaofeng Li Dezhen Song Lei Sun Shan Xu Hongpeng Wang Jingtai Liu Nankai University Tianjin 300350China Humanoids and Human Centered Mechatronics Research Line Italian Institute of Technology16163 GenovaItaly Department of Computer Science and Engineering Texas A&M UniversityCollege StationTexas 77843 USA Institute of Robotics and Automatic Information System Nankai UniversityTianjin 300350China Tianjin Key Laboratory of Intelligent Robotics Nankai UniversityTianjin 300350China

To balance the contradiction between higher flexibility and heavier load bearing capacity,we present a novel deformable manipulator which is composed of active rigid joints and deformable *** deformable link is composed of passive spherical joints with preload forces between socket-ball *** estimate the load bearing capacity of a deformable link,we present a static force-based model of spherical joint with preload force and analyze the static force propagation in the deformable *** yields an important result that the load bearing capacity of a spherical joint only depends on its radius,preload force,and static friction *** further develop a parameter estimation method to estimate the product of preload force and static friction *** experimental results validate our model.80.4%of percentage errors on the maximum payload mass prediction are below 15%.

关键词： Deformable link parameters identification spherical joint with preload force static force-based model