The MAP (Mean Average Precision) metric is one of the most popular performance metrics in the field of Information Retrieval Fault Localization (IRFL). However, there are problematic implementations of this MAP metric...
详细信息
ISBN:
(纸本)9798350329964
The MAP (Mean Average Precision) metric is one of the most popular performance metrics in the field of Information Retrieval Fault Localization (IRFL). However, there are problematic implementations of this MAP metric used in IRFL research. These implementations deviate from the text book definitions of MAP, rendering the metric sensitive to the truncation of retrieval results and inaccuracies and impurities of the used datasets. The application of such a deviating metric can lead to performance overestimation. This can pose a problem for comparability, transferability, and validity of IRFL performance results. In this paper, we discuss the definition and mathematical properties of MAP and common deviations and pitfalls in its implementation. We investigate and discuss the conditions enabling such overestimation: the truncation of retrieval results in combination with ground truths spanning multiple files and improper handling of undefined AP results. We demonstrate the overestimation effects using the Bench4BL benchmark and five well known IRFL techniques. Our results indicate that a flawed implementation of the MAP metric can lead to an overestimation of the IRFL performance, in extreme cases by up to 70 %. We argue for a strict adherence to the text book version of MAP with the extension of undefined AP values to be set to 0 for all IRFL experiments. We hope that this work will help to improve comparability and transferability in IRFL research.
Old style cryptographic techniques are genuinely compromised by the development of quantum computing, which requires the formation of safety ideal models that are impervious to quantum mistakes. A new permissioned blo...
详细信息
A magnetically coupled resonant wire-free installation system is designed. Aiming at the problem that the temperature of the barrel and primary coil increases due to the metal material barrel and gun firing, and the t...
详细信息
Machine learning models are being increasingly relied on for many natural language processing tasks. However, these models are vulnerable to adversarial attacks, i.e., inputs designed to target models into making a wr...
详细信息
This paper investigates the needs and expectations of both planners and clients, identifying the main barriers to the implementation of climate adaptation software tools. It also seeks to identify the main issues on s...
详细信息
Afra is an Eclipse-based tool for the modeling and model checking of Rebeca family models. Together with the standard enriched editor, easy to trace counter-example viewer, modular temporal property definition, export...
详细信息
software-Defined Networking (SDN) represents a significant shift in network architecture, providing exceptional programmability, flexibility, and simplified management. However, this paradigm shift introduces a unique...
详细信息
The proceedings contain 26 papers. The topics discussed include: partial bidirectionalization of model transformation languages;10 years of model federation with Openflexo: challenges and lessons learned;EditQL: a tex...
ISBN:
(纸本)9798400705045
The proceedings contain 26 papers. The topics discussed include: partial bidirectionalization of model transformation languages;10 years of model federation with Openflexo: challenges and lessons learned;EditQL: a textual query language for evolving models;model everything but with intellectual property protection — the Deltachain approach;AlloyASG: alloy predicate code representation as a compact structurally balanced graph;product lines of graphical modelling languages;tree-based versus hybrid graphical-textual model editors: an empirical study of testing specifications;modeling languages for digital twins: a survey among the German automotive industry;advancing domain-specific high-integrity model-based tools: insights and future pathways;and a comparative analysis of energy consumption between visual scripting models and C++ in unreal engine: raising awareness on the importance of green MDD.
In view of the insufficient ability of the currently existing deep learning-based methods to repair image high-frequency information and the small sensory field of the traditional convolutional methods. A two-stage im...
详细信息
In case of formulating more realistic inventory models, two factors of the problem have been raising attention of the researchers, one being the change in the demand rate and the other being the different types of tra...
详细信息
暂无评论