检索结果-内蒙古大学图书馆

Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ding, Peng Wu, Jingyu Kuang, Jun Ma, Dan Cao, Xuezhi Cai, Xunliang Chen, Shi Chen, Jiajun Huang, Shujian National Key Laboratory for Novel Software Technology Nanjing University Nanjing China College of Computer Science and Technology Zhejiang University Hangzhou China Meituan Shanghai China Meituan Beijing China Zhejiang-Singapore Innovation and AI Joint Research Lab Zhejiang University Hangzhou China

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on various visual-language understanding and generation tasks. However, MLLMs occasionally generate content inconsistent with the given images, which is known as "hallucination". Prior works primarily center on evaluating hallucination using standard, unperturbed benchmarks, which overlook the prevalent occurrence of perturbed inputs in real-world scenarios-such as image cropping or blurring-that are critical for a comprehensive assessment of MLLMs' hallucination. In this paper, to bridge this gap, we propose Hallu-PI, the first benchmark designed to evaluate Hallucination in MLLMs within Perturbed Inputs. Specifically, Hallu-PI consists of seven perturbed scenarios, containing 1,260 perturbed images from 11 object types. Each image is accompanied by detailed annotations, which include fine-grained hallucination types, such as existence, attribute, and relation. We equip these annotations with a rich set of questions, making Hallu-PI suitable for both discriminative and generative tasks. Extensive experiments on 12 mainstream MLLMs, such as GPT-4V and Gemini-Pro Vision, demonstrate that these models exhibit significant hallucinations on Hallu-PI, which is not observed in unperturbed scenarios. Furthermore, our research reveals a severe bias in MLLMs' ability to handle different types of hallucinations. We also design two baselines specifically for perturbed scenarios, namely Perturbed-Reminder and Perturbed-ICL. We hope that our study will bring researchers' attention to the limitations of MLLMs when dealing with perturbed inputs, and spur further investigations to address this issue. Our code and datasets are publicly available at https://***/NJUNLP/Hallu-PI. © 2024, CC BY.

关键词： Visual languages

DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Jia, Yongzhe Zhang, Xuyun Hu, Hongsheng Choo, Kim-Kwang Raymond Qi, Lianyong Xu, Xiaolong Beheshti, Amin Dou, Wanchun State Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University China School of Computing Macquarie University Australia School of Information and Physical Sciences University of Newcastle Australia Department of Information Systems and Cyber Security University of Texas San Antonio United States China School of Computer and Software Nanjing University of Information Science and Technology China

Federated learning (FL) has emerged as a prominent machine learning paradigm in edge computing environments, enabling edge devices to collaboratively optimize a global model without sharing their private data. However, existing FL frameworks suffer from efficacy deterioration due to the system heterogeneity inherent in edge computing, especially in the presence of domain shifts across local data. In this paper, we propose a heterogeneous FL framework DapperFL, to enhance model performance across multiple domains. In DapperFL, we introduce a dedicated Model Fusion Pruning (MFP) module to produce personalized compact local models for clients to address the system heterogeneity challenges. The MFP module prunes local models with fused knowledge obtained from both local and remaining domains, ensuring robustness to domain shifts. Additionally, we design a Domain Adaptive Regularization (DAR) module to further improve the overall performance of DapperFL. The DAR module employs regularization generated by the pruned model, aiming to learn robust representations across domains. Furthermore, we introduce a specific aggregation algorithm for aggregating heterogeneous local models with tailored architectures and weights. We implement DapperFL on a real-world FL platform with heterogeneous clients. Experimental results on benchmark datasets with multiple domains demonstrate that DapperFL outperforms several state-of-the-art FL frameworks by up to 2.28%, while significantly achieving model volume reductions ranging from 20% to 80%. Our code is available at: https://***/jyzgh/DapperFL. © 2024, CC BY.

关键词： Federated learning

PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Yu, Fangxu Jiang, Lai Huang, Shenyi Wu, Zhen Dai, Xinyu National Key Laboratory for Novel Software Technology Nanjing University China School of Artificial Intelligence Nanjing University China Department of Computer Science and Engineering Shanghai Jiao Tong University China University of California San DiegoCA United States

The ability to understand and predict the mental states of oneself and others, known as the Theory of Mind (ToM), is crucial for effective social interactions. Recent research has emerged to evaluate whether Large Language Models (LLMs) exhibit a form of ToM. Although recent studies have evaluated ToM in LLMs, existing benchmarks focus predominantly on physical perception with principles guided by the Sally-Anne test in synthetic stories and conversations, failing to capture the complex psychological activities of mental states in real-life social interactions. To mitigate this gap, we propose PERSUASIVETOM, a benchmark designed to evaluate the ToM abilities of LLMs in persuasive dialogues. Our framework introduces two categories of questions: (1) ToM Reasoning, assessing the capacity of LLMs to track evolving mental states (e.g., desire shifts in persuadees), and (2) ToM Application, evaluating whether LLMs can take advantage of inferred mental states to select effective persuasion strategies (e.g., emphasize rarity) and evaluate the effectiveness of persuasion strategies. Experiments across eight state-of-the-art LLMs reveal that while models excel on multiple questions, they struggle to answer questions that need tracking the dynamics and shifts of mental states and understanding the mental states in the whole dialogue comprehensively. Our aim with PERSUASIVETOM is to allow an effective evaluation of the ToM reasoning ability of LLMs with more focus on complex psychological activities. Our code is available at https://***/YuFangxu/PersuasiveToM. Copyright © 2025, The Authors. All rights reserved.

关键词： Economic and social effects

How to get better embeddings with code pre-trained models? An empirical study

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhao, Yu Gong, Lina Zhang, Haoxiang Yu, Yaoshen Huang, Zhiqiu College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics State Key Lab for Novel Software Technology Nanjing University Key Laboratory of Safety-Critical Software Nanjing University of Aeronautics and Astronautics China Queen’s University KingstonON Canada

Pre-trained language models have demonstrated powerful capabilities in the field of natural language processing (NLP). Recently, code pretrained model (PTM), which draw from the experiences of the NLP field, have also achieved state-of-the-art results in many software engineering (SE) downstream tasks. These code PTMs take into account the differences between programming languages and natural languages during pre-training and make adjustments to pre-training tasks and input data. However, researchers in the SE community still inherit habits from the NLP field when using these code PTMs to generate embeddings for SE downstream classification tasks, such as generating semantic embeddings for code snippets through special tokens and inputting code and text information in the same way as pre-training the PTMs. In this paper, we empirically study five different PTMs (i.e. CodeBERT, CodeT5, PLBART, CodeGPT and CodeGen) with three different architectures (i.e. encoder-only, decoder-only and encoder-decoder) on four SE downstream classification tasks (i.e. code vulnerability detection, code clone detection, just-in-time defect prediction and function docstring mismatch detection) with respect to the two aforementioned aspects. Our experimental results indicate that (1) regardless of the architecture of the code PTMs used, embeddings obtained through special tokens do not sufficiently aggregate the semantic information of the entire code snippet;(2) the quality of code embeddings obtained by combing code data and text data in the same way as pretraining the PTMs is poor and cannot guarantee richer semantic information;(3) using the method that aggregates the vector representations of all code tokens, the decoder-only PTMs can obtain code embeddings with semantics as rich as or even better quality than those obtained from the encoder-only and encoder-decoder PTMs. Based on our findings, we recommend that researchers in the SE community (1) pay attention to the vector represen

关键词： Semantics

Infrared: A Meta Bug Detector

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhang, Chi Wang, Yu Wang, Linzhang State Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University Jiangsu Nanjing210023 China

The recent breakthroughs in deep learning methods have sparked a wave of interest in learning-based bug detectors. Compared to the traditional static analysis tools, these bug detectors are directly learned from data, thus, easier to create. On the other hand, they are difficult to train, requiring a large amount of data which is not readily available. In this paper, we propose a new approach, called meta bug detection, which offers three crucial advantages over existing learning-based bug detectors: bug-type generic (i.e., capable of catching the types of bugs that are totally unobserved during training), self-explainable (i.e., capable of explaining its own prediction without any external interpretability methods) and sample efficient (i.e., requiring substantially less training data than standard bug detectors). Our extensive evaluation shows our meta bug detector (MBD) is effective in catching a variety of bugs including null pointer dereference, array index out-of-bound, file handle leak, and even data races in concurrent programs;in the process MBD also significantly outperforms several noteworthy baselines including Facebook Infer, a prominent static analysis tool, and FICS, the latest anomaly detection method. © 2022, CC BY.

关键词： Static analysis

Spatial-information Guided Adaptive Context-aware Network for Efficient RGB-D Semantic Segmentation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Yang Xiong, Chenyun Liu, Junjie Ye, Xuhui Sun, Guodong The School of Mechanical Engineering Hubei University of Technology Wuhan430068 China The Department of Electronic Engineering The Chinese University of Hong Kong Hong Kong The National Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University Nanjing210023 China

Efficient RGB-D semantic segmentation has received considerable attention in mobile robots, which plays a vital role in analyzing and recognizing environmental information. According to previous studies, depth information can provide corresponding geometric relationships for objects and scenes, but actual depth data usually exist as noise. To avoid unfavorable effects on segmentation accuracy and computation, it is necessary to design an efficient framework to leverage cross-modal correlations and complementary cues. In this paper, we propose an efficient lightweight encoder-decoder network that reduces the computational parameters and guarantees the robustness of the algorithm. Working with channel and spatial fusion attention modules, our network effectively captures multi-level RGB-D features. A globally guided local affinity context module is proposed to obtain sufficient high-level context information. The decoder utilizes a lightweight residual unit that combines short- and long-distance information with a few redundant computations. Experimental results on NYUv2, SUN RGB-D, and Cityscapes datasets show that our method achieves a better trade-off among segmentation accuracy, inference time, and parameters than the state-of-the-art methods. © 2023, CC BY-NC-ND.

关键词： Semantic Segmentation

Diffuse Map Guiding Unsupervised Generative Adversarial Network for SVBRDF Estimation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Luo, Zhiyao Chen, Hongnan Computer Science and Technology Jinan University Guangzhou China State Key Lab for Novel Software Technology Nanjing University Nanjing China

Reconstructing materials in the real world has always been a difficult problem in computer graphics. Accurately reconstructing the material in the real world is critical in the field of realistic rendering. Traditionally, materials in computer graphics are mapped by an artist, then mapped onto a geometric model by coordinate transformation, and finally rendered with a rendering engine to get realistic materials. For opaque objects, the industry commonly uses physical-based bidirectional reflectance distribution function (BRDF) rendering models for material modeling. The commonly used physical-based rendering models are Cook-Torrance BRDF, Disney BRDF. In this paper, we use the Cook-Torrance model to reconstruct the materials. The SVBRDF material parameters include Normal, Diffuse, Specular and Roughness. This paper presents a Diffuse map guiding material estimation method based on the Generative Adversarial Network(GAN). This method can predict plausible SVBRDF maps with global features using only a few pictures taken by the mobile phone. The main contributions of this paper are: 1) We preprocess a small number of input pictures to produce a large number of non-repeating pictures for training to reduce over-fitting. 2) We use a novel method to directly obtain the guessed diffuse map with global characteristics, which provides more prior information for the training process. 3) We improve the network architecture of the generator so that it can generate fine details of normal maps and reduce the possibility to generate over-flat normal maps. The method used in this paper can obtain prior knowledge without using dataset training, which greatly reduces the difficulty of material reconstruction and saves a lot of time to generate and calibrate datasets. Copyright © 2022, The Authors. All rights reserved.

关键词： Generative adversarial networks

3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Bao, Yanqi Ding, Tianyu Huo, Jing Liu, Yaoli Li, Yuxin Li, Wenbin Gao, Yang Luo, Jiebo State Key Laboratory for Novel Software Technology Nanjing University 210023 China Applied Sciences Group Microsoft Corporation Redmond United States Department of Computer Science University of Rochester United States

3D Gaussian Splatting (3DGS) has emerged as a prominent technique with the potential to become a mainstream method for 3D representations. It can effectively transform multiview images into explicit 3D Gaussian through efficient training, and achieve real-time rendering of novel views. This survey aims to analyze existing 3DGS-related works from multiple intersecting perspectives, including related tasks, technologies, challenges, and opportunities. The primary objective is to provide newcomers with a rapid understanding of the field and to assist researchers in methodically organizing existing technologies and challenges. Specifically, we delve into the optimization, application, and extension of 3DGS, categorizing them based on their focuses or motivations. Additionally, we summarize and classify nine types of technical modules and corresponding improvements identified in existing works. Based on these analyses, we further examine the common challenges and technologies across various tasks, proposing potential research opportunities. © 2024, CC BY.

关键词： 3D modeling

AdvMask: A Sparse Adversarial Attack Based Data Augmentation Method for Image Classification

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Yang, Suorong Li, Jinqiao Zhao, Jian Shen, Furao State Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University Nanjing210023 China The State Key Laboratory for Novel Software Technology School of Electronic Science and Engineering Nanjing University Nanjing210023 China

Data augmentation is a widely used technique for enhancing the generalization ability of convolutional neural networks (CNNs) in image classification tasks. Occlusion is a critical factor that affects on the generalization ability of image classification models. In order to generate new samples, existing data augmentation methods based on information deletion simulate occluded samples by randomly removing some areas in the images. However, those methods cannot delete areas of the images according to their structural features of the images. To solve those problems, we propose a novel data augmentation method, AdvMask, for image classification tasks. Instead of randomly removing areas in the images, AdvMask obtains the key points that have the greatest influence on the classification results via an end-to-end sparse adversarial attack module. Therefore, we can find the most sensitive points of the classification results without considering the diversity of various image appearance and shapes of the object of interest. In addition, a data augmentation module is employed to generate structured masks based on the key points, thus forcing the CNN classification models to seek other relevant content when the most discriminative content is hidden. AdvMask can effectively improve the performance of classification models in the testing process. The experimental results on various datasets and CNN models verify that the proposed method outperforms other previous data augmentation methods in image classification tasks. Copyright © 2022, The Authors. All rights reserved.

关键词： Image classification