检索结果-内蒙古大学图书馆

Investigating Efficacy of Perplexity in Detecting LLM-Generated Code

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Xu, Jinwei Zhang, He Yang, Yanjin Cheng, Zeru Lyu, Jun Liu, Bohan Zhou, Xin Yang, Lanxin Bacchelli, Alberto Chiam, Yin Kia Chiew, Thiam Kian State Key Laboratory of Novel Software Technology Software Institute Nanjing University Nanjing210008 China Department of Informatics University of Zürich Zürich8006 Switzerland Department of Software Engineering Faculty of Computer Science and Information Technology University of Malaya Kuala Lumpur50603 Malaysia

Large language model-generated code (LLMgCode) has become increasingly prevalent in software development. Many studies report that LLMgCode has more quality and security issues than human-authored code (HaCode). It is common for LLMgCode to mix with HaCode in a code change, while the change is signed by only human developers, without being carefully checked. Many automated methods have been proposed to detect LLMgCode from HaCode, in which the perplexity-based method (PERPLEXITY for short) is the state-of-the-art method. However, the efficacy evaluation of PERPLEXITY has focused on the detection accuracy. In this article, we are interested in whether PERPLEXITY is good enough in a wider range of realistic evaluation settings. To this end, we devise a large-scale dataset that includes 11,664 HaCode snippets and 13,164 LLMgCode snippets, and based on that, we carry out a family of experiments to compare PERPLEXITY against feature-based and pre-training-based methods from three perspectives: (1) detection accuracy in terms of programming language, degree of difficulty, and scale of solution, (2) generalization capability, and (3) inference efficiency. The experimental results show that PERPLEXITY has the best generalization capability while it has low accuracy and efficiency in most cases. Based on the experimental results and detection mechanism of PERPLEXITY, we discuss implications into both the strengths and limitations of PERPLEXITY, e.g., PERPLEXITY is unsuitable for high-level programming languages while it has good interpretability. As the first large-scale investigation on detecting LLMgCode from HaCode, this article provides a wide range of evidence for future improvement. Copyright © 2024, The Authors. All rights reserved.

关键词： High level languages

AoI-aware Incentive Mechanism for Mobile Crowdsensing using Stackelberg Game

学校读者我要写书评

暂无评论

AoI-aware Incentive Mechanism for Mobile Crowdsensing using ...

IEEE Annual Joint Conference: INFOCOM, IEEE computer and Communications Societies

作者： Mingjun Xiao Yin Xu Jinrui Zhou Jie Wu Sheng Zhang Jun Zheng School of Computer Science and Technology University of Science and Technology of China China Suzhou Institute for Advanced Research University of Science and Technology of China China Department of Computer and Information Sciences Temple Universityy State Key Laboratory for Novel Software Technology Nanjing University China

Mobile CrowdSensing (MCS) is a mobile computing paradigm, through which a platform can coordinate a crowd of workers to accomplish large-scale data collection tasks using their mobile devices. Information freshness has attracted much focus on MCS research worldwide. In this paper, we investigate the incentive mechanism design in MCS systems that take the freshness of collected data and social benefits into concerns. First, we introduce the Age of Information (AoI) metric to measure the freshness of data. Then, we model the incentive mechanism design with AoI guarantees as a novel incomplete information two-stage Stackelberg game with multiple constraints. Next, we derive the optimal strategies of this game so as to determine the optimal reward paid by the platform and the optimal data update frequency for each worker. Moreover, we prove that these optimal strategies form a unique Stackelberg equilibrium. Based on the optimal strategies, we propose an AoI-Aware Incentive (AIAI) mechanism for the MCS system, whereby the platform and all workers can maximize their utilities simultaneously. Meanwhile, the system can ensure that the AoI values of all data uploaded to the platform are not larger than a given threshold to achieve high data freshness. Extensive simulations on real-world traces are conducted to demonstrate the significant performance of AIAI.

关键词：

ATOM: Automated Black-Box Testing of Multi-label Image Classification Systems

学校读者我要写书评

暂无评论

ATOM: Automated Black-Box Testing of Multi-Label Image Class...

IEEE International Conference on Automated software Engineering (ASE)

作者： Shengyou Hu Huayao Wu Peng Wang Jing Chang Yongjun Tu Xiu Jiang Xintao Niu Changhai Nie Department of Computer Science and Technology State Key Laboratory for Novel Software Technology Nanjing University China Guangdong OPPO Mobile Telecommunications Corp. Ltd. China

Multi-label Image Classification Systems (MICSs) developed based on Deep Neural Networks (DNNs) are extensively used in people's daily life. Currently, although there are a variety of approaches to test DNN-based systems, they typically rely on the internals of DNNs to design test cases, and do not take the core specification of MICS (i.e., correctly recognizing multiple objects in a given image) into account. In this paper, we propose ATOM, an automated and systematic black-box testing framework for testing MICS. Specifically, ATOM exploits the label combination as the testing adequacy criteria, hoping to systematically examine the impact of correlations between a fixed number of labels on the classification ability of MICS. Then, ATOM leverages image search engine and natural language processing to find test images that are not only common to the real-world, but also relevant to target label combinations. Finally, ATOM combines metamorphic testing and label information to realize test oracle identification, based on which the ability of MICS in classifying different label combinations is evaluated. To evaluate the effectiveness of ATOM, we have performed experiments on two popular datasets of MICS, VOC and COCO (each with five state-of-the-art DNN models), and one real-world photo tagging application from our industrial partner. The experimental results reveal that the performance of current DNN-based MICSs remains less satisfactory even in recognizing correlations between only two labels, as ATOM triggers a total number of 6,049 such label combination related errors for all MICSs studied. In particular, ATOM reports 587 error-revealing images for the industrial MICS, in which 92% of them are confirmed by the developers.

关键词：

Semi-Supervised Image Captioning Considering Wasserstein Graph Matching

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yang, Yang The Nanjing University of Science and Technology Nanjing210094 China PCA Lab Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education Jiangsu Key Lab of Image and Video Understanding for Social Security School of Computer Science and Engineering Nanjing University of Science and Technology China Ministry of Education State Key Lab. for Novel Software Technology Nanjing University China

Image captioning can automatically generate captions for the given images, and the key challenge is to learn a mapping function from visual features to natural language features. Existing approaches are mostly supervised ones, i.e., each image has a corresponding sentence in the training set. However, considering that describing images always requires a huge of manpower, we usually have limited amount of described images (i.e., image-text pairs) and a large number of undescribed images in real-world applications. Thereby, a dilemma is the "Semi-Supervised Image Captioning". To solve this problem, we propose a novel Semi-Supervised Image Captioning method considering Wasserstein Graph Matching (SSIC-WGM), which turns to adopt the raw image inputs to supervise the generated sentences. Different from traditional single modal semi-supervised methods, the difficulty of semi-supervised cross-modal learning lies in constructing intermediately comparable information among heterogeneous modalities. In this paper, SSIC-WGM adopts the successful scene graphs as intermediate information, and constrains the generated sentences from two aspects: 1) inter-modal consistency. SSIC-WGM constructs the scene graphs of the raw image and generated sentence respectively, then employs the wasserstein distance to better measure the similarity between region embeddings of different graphs. 2) intra-modal consistency. SSIC-WGM takes the data augmentation techniques for the raw images, then constrains the consistency among augmented images and generated sentences. Consequently, SSIC-WGM combines the cross-modal pseudo supervision and structure invariant measure for efficiently using the undescribed images, and learns more reasonable mapping function. Experiments show that our method can outperform state-of-the-art comparison methods on the MS-COCO "Karpathy" offline test split, under different complex semi-supervised scenarios. Copyright © 2024, The Authors. All rights reserved.

关键词： Visual languages

Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Hu, Quanqi Qiu, Zi-Hao Guo, Zhishuai Zhang, Lijun Yang, Tianbao Department of Computer Science and Engineering Texas A&M University College StationTX United States National Key Laboratory for Novel Software Technology Nanjing University Nanjing China

In this paper, we consider non-convex multi-block bilevel optimization (MBBO) problems, which involve m 1 lower level problems and have important applications in machine learning. Designing a stochastic gradient and controlling its variance is more intricate due to the hierarchical sampling of blocks and data and the unique challenge of estimating hyper-gradient. We aim to achieve three nice properties for our algorithm: (a) matching the state-of-the-art complexity of standard BO problems with a single block;(b) achieving parallel speedup by sampling I blocks and sampling B samples for each sampled block per-iteration;(c) avoiding the computation of the inverse of a high-dimensional Hessian matrix estimator. However, it is non-trivial to achieve all of these by observing that existing works only achieve one or two of these properties. To address the involved challenges for achieving (a, b, c), we propose two stochastic algorithms by using advanced blockwise variance-reduction techniques for tracking the Hessian matrices (for low-dimensional problems) or the Hessian-vector products (for high-dimensional problems), and prove an iteration −3 complexity of (formula presented) for finding an ϵ-stationary point under appropriate conditions. We also conduct experiments to verify the effectiveness of the proposed algorithms comparing with existing MBBO algorithms. © 2023, CC BY.

关键词： Iterative methods

JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Cao, Jialun Chen, Zhiyong Wu, Jiarong Cheung, Shing-Chi Xu, Chang Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong Guangzhou HKUST Fok Ying Tung Research Institute Guangzhou China State Key Laboratory for Novel Software Technology Nanjing University Nanjing China

Code generation benchmarks such as HumanEval are widely adopted to evaluate LLMs’ capabilities. However, after consolidating the latest 24 benchmarks, we noticed three significant imbalances. First, imbalanced programming language. 95.8% of benchmarks involve Python, while only 5 benchmarks involve Java, resulting in an insufficient understanding of LLMs’ capability to generate Java code. Second, imbalanced code granularity. Function-/statement-level benchmarks account for over 83.3% of benchmarks. Only a mere handful extends to class-/project-levels, and all are limited to Python. Third, lacking advanced features. Existing benchmarks primarily assess basic coding skills (e.g., variables, operators, and control structures), while overlooking advanced Object-Oriented Programming (OOP) features (i.e., encapsulation, inheritance, and polymorphism). Considering the prevalence of these advanced features in real-world Java project development, constructing benchmarks to test LLMs on handling OOP features is necessary. To fill these gaps, we propose JavaBench, a project-level Java benchmark that exercises OOP features. It comprises four Java projects with 389 methods in 106 Java classes. The test coverage is up to 92%, and JavaBench is attested by 282 undergraduate students, reaching a 90.93/100 average score (i.e., pass rate against the test suite), ensuring the quality of documentation, code skeleton, and tests. To better evaluate LLM’s capability against JavaBench, we introduce a systematic evaluation design covering three context settings and five synthesis strategies at two granularities using three hierarchical metrics. Our extensive experiment yields several interesting findings. First, we noticed that regarding project-level Java programming, LLMs are far behind undergraduate students (no project can be correctly completed by any studied LLMs, and at most 41.17% Pass@5 in a more relaxed evaluation). Second, using method signature as prompt context may strike an ide

关键词： Students

Investigating the Impact of Bug Dependencies on Bug-Fixing Time Prediction

学校读者我要写书评

暂无评论

Investigating the Impact of Bug Dependencies on Bug-Fixing T...

International Symposium on Empirical software Engineering and Measurement, ESEM

作者： Chenglin Li Yangyang Zhao Yibiao Yang Yuming Zhou Liming Nie Zuohua Ding School of Computer Science and Technology Zhejiang Sci-Tech University Hangzhou China State Key Lab for Novel Software Technology Nanjing University Nanjing China School of Computer Science and Engineering Nanyang Technological University Singapore

Background: Bug dependencies refer to the link relationships between bugs and related issues, which are commonly observed in software evolution. It has been found that bugs with bug dependencies often take longer time to be resolved than other bugs without any dependencies. Despite the potential impact of bug dependencies on bug-fixing time, previous studies use traditional metrics without considering bug dependencies to build bug-fixing time prediction models. As a result, there is currently little empirical evidence to support the use of bug dependencies in improving prediction accuracy. Aims: We aim to conduct a comprehensive empirical study to investigate the value of considering bug dependencies for bug-fixing time prediction. Method: We define a set of bug dependency metrics based on bug dependencies. We first investigate the correlation between bug dependency metrics and bug-fixing time to investigate whether bugs with more complex dependencies are more time-consuming to be fixed. Next, we employ principal component analysis to study whether bug dependency metrics capture additional dimensions of a bug compared to traditional metrics. Finally, we build multivariate prediction models to explore whether considering bug dependencies can improve the effectiveness of bug-fixing time prediction. Results: The experimental results suggest that: (1) bugs with more complex dependencies require more time to be fixed; (2) bug dependency metrics are complementary to traditional metrics; (3) considering bug dependencies can improve the effectiveness of bug-fixing time prediction. Conclusions: These findings highlight the importance of considering bug dependencies in bug-fixing time prediction, and provide valuable insights into the potential impact of bug dependencies on software development processes.

关键词：

MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Gan, Shiwei Yin, Yafeng Jiang, Zhiwei Wen, Hongkai Xie, Lei Lu, Sanglu State Key Laboratory for Novel Software Technology Nanjing University Nanjing210023 China Department of Computer Science University of Warwick CoventryCV4 7AL United Kingdom

—Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (e.g., object identification, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. In fact, sign language tasks require focusing on sign-related regions, including the collaboration between different regions (e.g., left hand region and right hand region) and the effective content in a single region. To capture such region-related features, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs and designs the following three graph modules for feature extraction, i.e., Local Sign Graph (LSG) module, Temporal Sign Graph (TSG) module and Hierarchical Sign Graph (HSG) module. Specifically, the LSG module learns the correlation of intra-frame cross-region features within one frame, i.e., focusing on spatial features. The TSG module tracks the interaction of inter-frame cross-region features among adjacent frames, i.e., focusing on temporal features. The HSG module aggregates the same-region features from different-granularity feature maps of a frame, i.e., focusing on hierarchical features. In addition, to further improve the performance of sign language tasks without gloss annotations, we propose a simple yet counter-intuitive Text-driven CTC Pre-training (TCP) method, which generates pseudo gloss labels from text labels for model pretraining. Extensive experiments conducted on current five public sign language datasets demonstrate the superior performance of the proposed model. Notably, our model surpasses the SOTA models on multiple sign language tasks across several datasets, without relying on any additional cues. Copyright © 2025, The Authors. All rights reserved.

关键词： Feature Selection

Not All Data Matters: An End-to-End Adaptive Dataset Pruning Framework for Enhancing Model Performance and Efficiency

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yang, Suorong Yang, Hongchao Guo, Suhan Shen, Furao Zhao, Jian National Key Laboratory for Novel Software Technology Nanjing University China Department of Computer Science and Technology Nanjing University China School of Artificial Intelligence Nanjing University China School of Electronic Science and Engineering Nanjing University China

While deep neural networks have demonstrated remarkable performance across various tasks, they typically require massive training data. Due to the presence of redundancies and biases in real-world datasets, not all data in the training dataset contributes to the model performance. To address this issue, dataset pruning techniques have been introduced to enhance model performance and efficiency by eliminating redundant training samples and reducing computational and memory overhead. However, previous works most rely on manually crafted scalar scores, limiting their practical performance and scalability across diverse deep networks and datasets. In this paper, we propose AdaPruner, an end-to-end Adaptive DAtaset PRUNing framEwoRk. AdaPruner can perform effective dataset pruning without the need for explicitly defined metrics. Our framework jointly prunes training data and fine-tunes models with task-specific optimization objectives. AdaPruner leverages (1) An adaptive dataset pruning (ADP) module, which iteratively prunes redundant samples to an expected pruning ratio;and (2) A pruning performance controller (PPC) module, which optimizes the model performance for accurate pruning. Therefore, AdaPruner exhibits high scalability and compatibility across various datasets and deep networks, yielding improved dataset distribution and enhanced model performance. AdaPruner can still significantly enhance model performance even after pruning up to 10-30% of the training data. Notably, these improvements are accompanied by substantial savings in memory and computation costs. Qualitative and quantitative experiments suggest that AdaPruner outperforms other state-of-the-art dataset pruning methods by a large margin. Copyright © 2023, The Authors. All rights reserved.

关键词： Deep neural networks