Parameter tuning or default values? An empirical investigation in search-based software engineering

作     者：Arcuri, Andrea Fraser, Gordon 

作者机构：Simula Res Lab Certus Software V&V Ctr Lysaker Norway Univ Sheffield Dept Comp Sci Sheffield S1 4DP S Yorkshire England 

出 版 物：《EMPIRICAL SOFTWARE ENGINEERING》 (经验软件工程)

年 卷 期：2013年第18卷第3期

页      面：594-623页

核心收录：

学科分类：08[工学] 0835[工学-软件工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：Norwegian Research Council Google Focused Research Award on "Test Amplification" 

主　　题：Search-based software engineering Test data generation Object-oriented Unit testing Tuning EvoSuite Java Response surface Design of experiments 

摘      要：Many software engineering problems have been addressed with search algorithms. Search algorithms usually depend on several parameters (e.g., population size and crossover rate in genetic algorithms), and the choice of these parameters can have an impact on the performance of the algorithm. It has been formally proven in the No Free Lunch theorem that it is impossible to tune a search algorithm such that it will have optimal settings for all possible problems. So, how to properly set the parameters of a search algorithm for a given software engineering problem? In this paper, we carry out the largest empirical analysis so far on parameter tuning in search-based software engineering. More than one million experiments were carried out and statistically analyzed in the context of test data generation for object-oriented software using the EvoSuite tool. Results show that tuning does indeed have impact on the performance of a search algorithm. But, at least in the context of test data generation, it does not seem easy to find good settings that significantly outperform the default values suggested in the literature. This has very practical value for both researchers (e.g., when different techniques are compared) and practitioners. Using default values is a reasonable and justified choice, whereas parameter tuning is a long and expensive process that might or might not pay off in the end.