检索结果-内蒙古大学图书馆

A Causality-Aware Paradigm for Evaluating Creativity of multimodal large language models

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2025年第5期47卷 3830-3846页

作者： Huang, Zhongzhan Zhong, Shanshan Zhou, Pan Gao, Shanghua Zitnik, Marinka Lin, Liang Sun Yat Sen Univ Guangzhou 510275 Peoples R China Singapore Management Univ Singapore 188065 Singapore Harvard Univ Cambridge MA 02138 USA Sun Yat Sen Univ Sch Comp Sci & Engn Guangzhou 510275 Peoples R China Guangdong Key Lab Big Data Anal & Proc Guangzhou 510275 Peoples R China Peng Cheng Lab Shenzhen 518108 Peoples R China

Recently, numerous benchmarks have been developed to evaluate the logical reasoning abilities of large language models (LLMs). However, assessing the equally important creative capabilities of LLMs is challenging due to the subjective, diverse, and data-scarce nature of creativity, especially in multimodal scenarios. In this paper, we consider the comprehensive pipeline for evaluating the creativity of multimodal LLMs, with a focus on suitable evaluation platforms and methodologies. First, we find the Oogiri game-a creativity-driven task requiring humor, associative thinking, and the ability to produce unexpected responses to text, images, or both. This game aligns well with the input-output structure of modern multimodal LLMs and benefits from a rich repository of high-quality, human-annotated creative responses, making it an ideal platform for studying LLM creativity. Next, beyond using the Oogiri game for standard evaluations like ranking and selection, we propose LoTbench, an interactive, causality-aware evaluation framework, to further address some intrinsic risks in standard evaluations, such as information leakage and limited interpretability. The proposed LoTbench not only quantifies LLM creativity more effectively but also visualizes the underlying creative thought processes. Our results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable. Furthermore, we observe a strong correlation between results from the multimodal cognition benchmark MMMU and LoTbench, but only a weak connection with traditional creativity metrics. This suggests that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts. Project Page.

关键词： Creativity Games Cognition Standards Benchmark testing Training Pipelines Manuals large language models Information leakage multimodal large language models benchmark causal intervention

来源：评论

学校读者我要写书评

暂无评论

LLM4CAD: multimodal large language models for Three-Dimensional Computer-Aided Design Generation

引用

JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING 2025年第2期25卷 021005页

作者： Li, Xingang Sun, Yuewan Sha, Zhenghui Univ Texas Austin Walker Dept Mech Engn Austin TX 78712 USA

The evolution of multimodal large language models (LLMs) capable of processing diverse input modalities (e.g., text and images) holds new prospects for their application in engineering design, such as the generation of 3D computer-aided design (CAD) models. However, little is known about the ability of multimodal LLMs to generate 3D design objects, and there is a lack of quantitative assessment. In this study, we develop an approach to enable LLMs to generate 3D CAD models (i.e., LLM4CAD) and perform experiments to evaluate their efficacy where GPT-4 and GPT-4V were employed as examples. To address the challenge of data scarcity for multimodal LLM studies, we created a data synthesis pipeline to generate CAD models, sketches, and image data of typical mechanical components (e.g., gears and springs) and collect their natural language descriptions with dimensional information using Amazon Mechanical Turk. We positioned the CAD program (programming script for CAD design) as a bridge, facilitating the conversion of LLMs' textual output into tangible CAD design objects. We focus on two critical capabilities: the generation of syntactically correct CAD programs (Cap1) and the accuracy of the parsed 3D shapes (Cap2) quantified by intersection over union. The results show that both GPT-4 and GPT-4V demonstrate great potential in 3D CAD generation by just leveraging their zero-shot learning ability. Specifically, on average, GPT-4V outperforms when processing only text-based input, exceeding the results obtained using multimodal inputs, such as text with image, for Cap 1 and Cap 2. However, when examining category-specific results of mechanical components, the prominence of multimodal inputs is increasingly evident for more complex geometries (e.g., springs and gears) in both Cap 1 and Cap 2. The potential of multimodal LLMs to improve 3D CAD generation is clear, but their application must be carefully calibrated to the complexity of the target CAD models to be generated.

关键词： multimodal large language models GPT-4 GPT-4V computer-aided design generative design

来源：评论

学校读者我要写书评

暂无评论

Exploring the prospects of multimodal large language models for Automated Emotion Recognition in education: Insights from Gemini

引用

COMPUTERS & EDUCATION 2025年 232卷

作者： Yu, Shuzhen Androsov, Alexey Yan, Hanbing East China Normal Univ Dept Educ Informat Technol Shanghai 200062 Peoples R China Xian Eurasia Univ Sch Humanities & Educ Xian 710065 Peoples R China East China Normal Univ Sch Open Learning & Educ Shanghai 200062 Peoples R China

Emotions play a pivotal role in daily judgments and decision-making, particularly in educational settings, where understanding and responding to learners' emotions is essential for personalized learning. While there has been growing interest in emotion recognition, traditional methods, such as manual observations and self-reports, are often subjective and time-consuming. The rise of AI has led to the development of Automated Emotion Recognition (AER), offering transformative opportunities for educational reform by enabling personalized learning through emotional insights. However, AER continues to face challenges, including reliance on large-scale labeled databases, limited flexibility, and inadequate adaptation to diverse educational contexts. Recent advancements in AI, particularly multimodal large language models (MLLMs), show promise in addressing these challenges, though their application in AER remains underexplored. This study aimed to fill this gap by systematically evaluating the performance of Gemini, a pioneering MLLM, in image-based AER tasks across five databases: CK+, FER-2013, RAF-DB, OL-SFED and DAiSEE. The analysis examined recognition accuracy, error patterns, emotion inference mechanisms, and the impact of image preprocessing techniques - such as face cropping, bilinear interpolation, and super-resolution - on the model's performance. The results revealed that Gemini achieved high emotion recognition accuracy, especially in distinguishing emotional polarities across all databases. Image preprocessing significantly improved the recognition of basic emotions, though its effect on academic emotion recognition was minor. The confusion in academic emotion recognition stemmed from Gemini's limited understanding of academic emotion features and its insufficient ability to capture contextual cues. Building on the results, this study outlines specific future research directions from both technological and educational perspectives. These findings offer valu

关键词： Data science applications in education Teaching/learning strategies multimodal large language models Automated Emotion Recognition

来源：评论

学校读者我要写书评

暂无评论

(In)forming the new building envelope: A pedagogical study in generative design with precedents and multimodal large language models

引用

INTERNATIONAL JOURNAL OF ARCHITECTURAL COMPUTING 2025年第1期23卷 96-121页

作者： Veloso, Pedro Univ Arkansas Dept Architecture Fayetteville AR USA

This pedagogical study delves into integrating established and emerging computational methods into architectural education, with a specific focus on building envelope design within a ***. course. Students employ parametric modeling (PM), design optimization (DO), and multimodal large language models (MLLMs) to analyze and reinterpret building envelope precedents. Parametric design and optimization are utilized to explore envelope variations based on parametric logic and performance evaluation. In the case of MLLMs, students leverage visual patterns from precedents as a form-giving construct for new 3D envelope proposals. While students adeptly integrate MLLMs into their design process, generating successful 3D models, challenges arise in control and translation across representations, leading to unclear scale and tectonics in some design proposals. Survey results reveal that students perceive MLLMs as a valuable, uncomplicated method for rapid design ideation and refinement, but challenges persist in addressing real architectural constraints. Parametric modeling is viewed as a tool for structuring design and DO is seen as a later stage for refining designs based on metrics. The study underscores the importance of evolving user interfaces for MLLMs in specific design tasks, addressing challenges in precision and design scale through prompts and guiding images. It also discusses the potential to combine MLLMs with various generative methods and modeling software during transitions between design media to support future initiatives integrating computational methods into the design process.

关键词： Building envelope generative design architectural education multimodal large language models parametric modeling design optimization

来源：评论

学校读者我要写书评

暂无评论

Explore the Textual Perception Ability on the Images for multimodal large language models 13th

Explore the Textual Perception Ability on the Images for Mul...

引用

13th International Conference on Natural language Processing and Chinese Computing

作者： Kuang, Jiayi Ouyang, Jiarui Shen, Ying Sun Yat Sen Univ Guangzhou Guangdong Peoples R China

ISBN: (纸本)9789819794423;9789819794430

Chinese Spell Checking or Chinese Spelling Correction (CSC) in writing assistants has significantly enhanced users' text quality. multimodal CSC expands this task into the multimodal domain and introduces a new research challenge: understanding textual information at the image level. The traditional two-stage strategy, which directly maps images to text before performing checking and correction, often faces issues such as the many-to-one mapping problem of fake characters and misrecognizing fake characters as correct ones. To address these issues, we propose a unified one-stage framework based on the emerging multimodal large language models (MLLMs) to achieve a discrete semantic understanding of text directly at the image level, thus alleviating the image-to-text mapping problem. Additionally, we introduce an adaptation strategy for MLLM in the multimodal CSC task, enhancing its few-shot learning capability through in-context learning with prompt design. We evaluate different MLLM base models and verify the effectiveness of this one-stage framework, demonstrating performance that matches or even surpasses fine-tuned baseline models without fine-tuning. Furthermore, we conduct a series of analyses that provide insights into visual perception and text correction.

关键词： multimodal Chinese Spell Checking multimodal large language models

来源：评论

学校读者我要写书评

暂无评论

FoodMLLM-JP: Leveraging multimodal large language models for Japanese Recipe Generation 31st

FoodMLLM-JP: Leveraging Multimodal Large Language Models for...

引用

31st International Conference on Multimedia Modeling

作者： Imajuku, Yuki Yamakata, Yoko Aizawa, Kiyoharu Univ Tokyo Tokyo Japan

ISBN: (纸本)9789819620531;9789819620548

Research on food image understanding using recipe data has been a long-standing focus due to the diversity and complexity of the data. Moreover, food is inextricably linked to people's lives, making it a vital research area for practical applications such as dietary management. Recent advancements in multimodal large language models (MLLMs) have demonstrated remarkable capabilities, not only in their vast knowledge but also in their ability to handle languages naturally. While English is predominantly used, they can also support multiple languages including Japanese. This suggests that MLLMs are expected to significantly improve performance in food image understanding tasks. We fine-tuned open MLLMs LLaVA-1.5 and Phi-3 Vision on a Japanese recipe dataset and benchmarked their performance against the closed model GPT-4o. We then evaluated the content of generated recipes, including ingredients and cooking procedures, using 5,000 evaluation samples that comprehensively cover Japanese food culture. Our evaluation demonstrates that the open models trained on recipe data outperform GPT-4o, the current state-of-the-art model, in ingredient generation. Our model achieved F1 score of 0.531, surpassing GPT-4o's F1 score of 0.481, indicating a higher level of accuracy. Furthermore, our model exhibited comparable performance to GPT-4o in generating cooking procedure text.

关键词： food computing recipe text generation multimodal large language models large multimodal models vision and language

来源：评论

学校读者我要写书评

暂无评论

CalorieVoL: Integrating Volumetric Context Into multimodal large language models for Image-Based Calorie Estimation 31st

CalorieVoL: Integrating Volumetric Context Into Multimodal L...

引用

31st International Conference on Multimedia Modeling

作者： Tanabe, Hikaru Yanai, Keiji Univ Electrocommun Chofu Tokyo Japan

ISBN: (纸本)9789819620708;9789819620715

multimodal large language models (MLLMs) can perform various food-related tasks with high quality. Notably, high-performance MLLMs, such as GPT-4V, can even estimate caloric content from food images. However, these MLLMs often struggle to accurately recognize volume information, which often leads to errors in calorie estimation. To address this issue, we propose a new MLLM framework called CalorieVoL, designed to enhance the recognition of volume information in food items. By integrating this framework into MLLMs like GPT-4V, we achieved higher scores in terms of MAE and correlation coefficients on Nutrition5k compared to simple MLLMs. Our experiments also showed that the volume-aware recognition improved responses in scenarios where accurate volume estimation is critical.

关键词： multimodal large language models Image-based Calorie Estimation Volume Estimation

来源：评论

学校读者我要写书评

暂无评论

Leveraging multimodal large language models for Enhanced Learning and Application in Building Energy Modeling 9th

Leveraging Multimodal Large Language Models for Enhanced Lea...

引用

9th International Building Physics Conference

作者： Labib, Rania Prairie View A&M Univ Prairie View TX 77446 USA

ISBN: (纸本)9789819783151;9789819783137;9789819783120

This paper presents a novel application of multimodal large language models (LLMs) to enhance the learning and application of building energy modeling. The study leverages Retrieval-Augmented Generation (RAG) models integrated with a dataset of 59 publicly available YouTube video tutorials focused on EnergyPlus and OpenStudio. Unlike traditional LLM methods, this approach is unique in its utilization of three LLMs to process and integrate different types of data: text, screenshots, and video references. The preprocessing phase utilizes Google's transcription service to convert video content into text, which is then summarized using the T5 model and embedded with the Instructor Embedder model. Meta's Llama v2 7b model handles user queries, extracting relevant information and providing detailed responses enriched with visual references, including exact video minutes and screenshots. Unlike traditional LLM models, the proposed model delivers comprehensive responses that go beyond text, providing users with reference videos with exact timestamps and screenshots. Furthermore, the proposed web interface presents these enriched responses in one page, significantly reducing the time needed for users to understand and apply complex energy modeling concepts. This framework demonstrates the potential of multimodal LLMs in creating powerful educational tools for architects, engineers, and students. The entire workflow was completed on a laptop with a single 4GB GPU, demonstrating the feasibility of implementing such a system on relatively modest hardware.

关键词： multimodal large language models large language models for Energy Modeling Learning Retrieval Augmented Generation (RAG) with Video Tutorials Dataset

来源：评论

学校读者我要写书评

暂无评论

Exploring the Implicit Semantic Ability of multimodal large language models: A Pilot Study on Entity Set Expansion

Exploring the Implicit Semantic Ability of Multimodal Large ...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Wang, Hebin Li, Yangning Li, Yinghui Zheng, Hai-Tao Jiang, Wenhao Kim, Hong-Gee Shenzhen International Graduate School Tsinghua University China Shenzhen International Graduate School Tsinghua University Peng Cheng Laboratory China China Seoul National University Korea Republic of

ISBN: (纸本)9798350368741

The rapid development of multimodal large language models (MLLMs) has brought significant improvements to a wide range of tasks in real-world applications. However, LLMs still exhibit certain limitations in extracting implicit semantic information. In this paper, we applies MLLMs to the Multi-modal Entity Set Expansion (MESE) task, which aims to expand a handful of seed entities with new entities belonging to the same semantic class, and multi-modal information is provided with each entity. We explore the capabilities of MLLMs to understand implicit semantic information at the entity-level granularity through the MESE task, introducing a listwise ranking method LUSAR that maps local scores to global rankings. Our LUSAR demonstrates significant improvements in MLLM's performance on the MESE task, marking the first use of generative MLLM for ESE tasks and extending the applicability of listwise ranking. © 2025 IEEE.

关键词： Entity Set Expansion Implicit Semantic Reasoning multimodal large language models

来源：评论

学校读者我要写书评

暂无评论

Exploring the Role of CLIP Global Visual Features in multimodal large language models

Exploring the Role of CLIP Global Visual Features in Multimo...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Bai, Zixing Bai, Yuting School of Computer Science and Technology Fudan University Shanghai China

ISBN: (纸本)9798350368741

The next recognized development direction of large language models (LLMs) is to integrate and enhance multimodal capability. Although current multimodal large language models (MLLMs) have achieved impressive performance by combining the pre-trained visual encoder CLIP and LLM, these works mainly focus on using the CLIP patch visual features. In practice, we find that CLIP global visual features are more efficient than patch visual features in some scenarios, especially in multimodal reasoning tasks. Therefore, we explore the role of CLIP global visual features and propose a new MLLM with the usage of full global visual features in this paper. Our model adopts the parameter-efficient transfer learning (PETL) method Adapter to fine-tune the pre-trained models and a simple MLP-based network to connect the visual encoder and LLM. To validate the performance, we evaluate our model on the first large-scale multimodal science question dataset, ScienceQA. Our model achieves a new state-of-the-art (SoTA) result of 93.96% on ScienceQA, which is higher than the previous SoTA result of 92.53%. © 2025 IEEE.

关键词： language multimodal large language models Vision

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：