Testing a large-scale, real-life commercial software application is a very challenging task due to the constant changes in the software, the involvement of multiple programmers and testers, and a large amount of code....
详细信息
ISBN:
(纸本)0769524656
Testing a large-scale, real-life commercial software application is a very challenging task due to the constant changes in the software, the involvement of multiple programmers and testers, and a large amount of code. Integrating testing with development can help find program bugs at an earlier stage and hence reduce the overall cost. In this paper, we report our experience on how to apply eXVantage(1) (a tool suite for code coverage testing, debugging, performance profiling, etc.) to a large, complex Java application at the implementation and unit testing phases in Avaya. Our results suggest that programmers and testers can benefit from using eXVantage to monitor the testing process, gain confidence on the quality of their software, detect bugs which are otherwise difficult to reveal, and identify performance bottlenecks in terms of which part of code is most frequently executed.
Dynamic Symbolic Execution (DSE) is an important method for testing of programs. An important system on DSE is KLEE [1] which inputs a C/C++ program annotated with symbolic variables, compiles it into LLVM, and then e...
详细信息
ISBN:
(纸本)9783030452346;9783030452339
Dynamic Symbolic Execution (DSE) is an important method for testing of programs. An important system on DSE is KLEE [1] which inputs a C/C++ program annotated with symbolic variables, compiles it into LLVM, and then emulates the execution paths of LLVM using a specified backtracking strategy. The major challenge in symbolic execution is path explosion. The method of abstraction learning [7] has been used to address this. The key step here is the computation of an interpolant to represent the learned abstraction. TracerX, our tool, is built on top of KLEE and it implements and utilizes abstraction learning. The core feature in abstraction learning is subsumption of paths whose traversals are deemed to no longer be necessary due to similarity with already-traversed paths. Despite the overhead of computing interpolants, the pruning of the symbolic execution tree that interpolants provide often brings significant overall benefits. In particular, TracerX can fully explore many programs that would be impossible for any non-pruning system like KLEE to do so.
Continuous Integration (CI) is the practice of automating and improving the frequency of code integration. CI has been widely adopted by software development teams and has brought the attention of researchers to study...
详细信息
ISBN:
(纸本)9781728130941
Continuous Integration (CI) is the practice of automating and improving the frequency of code integration. CI has been widely adopted by software development teams and has brought the attention of researchers to study its benefits. Existing research shows that CI can improve software quality by identifying the errors earlier in the software development lifecycle. One question that remains open, however, is whether CI increases the adoption of testing practices in software projects. The goal of our work is to investigate the evolution of software tests and its relationship with the adoption of Continuous Integration. We set out to compare 82 projects that adopted CI (CI projects) and 82 projects that have never adopted CI (NOCI projects). In total, we studied 3,936 versions of our studied projects to investigate trends on the test code ratio and coverage. We observe that 40.2% of the CI projects have a rising testcode ratio trend while only 17% of the NOCI projects have a rising trend. Additionally, we find evidences that CI projects have improved the overall test coverage while NOCI projects do not experience the same growth. Finally, we build a mixed-effects model to study software development factors than can possibly explain the test ratio. Our models reveal that test ratio is largely explained by the project inherent context rather than code or process factors. In overall, our work demonstrates that Continuous Integration can be empirically associated with a healthier test code evolution.
Automated test generation can reduce the manual effort to improve software quality. A test generation method employs code coverage, such as the widely-used branch coverage, to guide the inference of test cases. These ...
详细信息
ISBN:
(纸本)9781728162805
Automated test generation can reduce the manual effort to improve software quality. A test generation method employs code coverage, such as the widely-used branch coverage, to guide the inference of test cases. These test cases can be used to detect hidden faults. An automatic tool takes a specific type of code coverage as a configurable parameter. Given an automated tool of test generation, a fault may be detected by one type of code coverage, but omitted by another. In frequently released software projects, the time budget of testing is limited. Configuring code coverage for a testing tool can effectively improve the quality of projects. In this paper, we conduct a preliminary study on whether a fault can be detected by specific code coverage in automated test generation. We build predictive models with 60 metrics of faulty source code to identify detectable faults under eight types of code coverage, such as branch coverage. In the experiment, an off-the-shelf tool, EvoSuite is used to generate test data. Experimental results show that different types of code coverage result in the detection of different faults. The extracted metrics of faulty source code can be used to predict whether a fault can be detected with the given code coverage;all studied code coverage can increase the number of detected faults that are missed by the widely-used branch coverage. This study can be viewed as a preliminary result to support the configuration of code coverage in the application of automated test generation.
The quantification of several software attributes (e.g., size, complexity, cohesion, coupling) is usually carried out in a static fashion, and several hundreds of measures have been defined to this end. However, stati...
详细信息
ISBN:
(纸本)9781450310567
The quantification of several software attributes (e.g., size, complexity, cohesion, coupling) is usually carried out in a static fashion, and several hundreds of measures have been defined to this end. However, static measurement may only be an approximation for the measurement of these attributes during software use. The paper proposes a theoretical framework based on Axiomatic Approaches for the definition of sensible dynamic software measures that can dynamically capture these attributes. Dynamic measures based on this framework are defined for dynamically quantifying size and coupling. In this paper, we also compare dynamic measures of size and coupling against well-known static measures by correlating them with fault-pronenesses of four case studies.
Mutation testing is a powerful technique for assessing and improving test suite quality that artificially introduces bugs and checks whether the test suites catch them. However, it is also computationally expensive an...
详细信息
ISBN:
(纸本)9798400703270
Mutation testing is a powerful technique for assessing and improving test suite quality that artificially introduces bugs and checks whether the test suites catch them. However, it is also computationally expensive and rarely scales to large projects. One promising recent approach to tackling this problem uses machine learning to predict whether the tests will detect the synthetic bugs, without actually running those tests. However, existing predictive mutation testing approaches still misclassify 33% of a randomly sampled set of mutant-test suite pairs. We introduce MutationBERT, an approach for predictive mutation testing that simultaneously encodes the source method mutation and test method, capturing key context in the input representation. Thanks to its higher precision, MutationBERT saves 33% of the time spent by prior work to verify live mutants, and improves precision, recall, and F1 score in both same project and cross project settings. MutationBERT not only enhances the state-of-the-art in predictive mutation testing, but also presents practical benefits for real-world applications, both in saving developer time and finding hard to detect mutants.
This article presents an algorithm for reducing JUnit test suites, as well as a tool (implemented as an Eclipse plugin) that realizes it. The algorithm obtains high reductions in the size of the test suite while prese...
详细信息
ISBN:
(纸本)9789899624702
This article presents an algorithm for reducing JUnit test suites, as well as a tool (implemented as an Eclipse plugin) that realizes it. The algorithm obtains high reductions in the size of the test suite while preserves the coverage.
Heuristic-based optimization techniques have been increasingly used to automate different types of code coverage analysis. Several studies suggest that interdependencies (in the form of comparisons) may exist between ...
详细信息
Heuristic-based optimization techniques have been increasingly used to automate different types of code coverage analysis. Several studies suggest that interdependencies (in the form of comparisons) may exist between the condition constructs, of variables and constant values, in the branching conditions of real-world programs, e.g. (i≤ 100) or (i= = j), etc. In this work, by interdependencies we refer to the situations where, to satisfy a branching condition, there must be a certain relationship between the values of some specific condition constructs (which may or may not be a part of the respective condition predicates). For example, the values of variables i and j must be equal to satisfy the condition of (i= = j), and the value of variable k must be equal to 100 for the satisfaction of the condition of (k= = 100). To date, only the Ariadne, a Grammatical Evolution (GE)-based system, exploits these interdependencies between input variables (e.g. of the form (i≤ j) or (i= = j), etc.) to efficiently generate test data. Ariadne employs a simple attribute grammar to exploit these dependencies, which enables it to evolve complex test data, and has been compared favourably to other well-known techniques in the literature. However, Ariadne does not benefit from interdependencies involving constants, e.g. (i≤ 100) or (j= = 500), etc., due to the difficulty in evolving precise values, and these are equally important constructs of condition predicates. Furthermore, constant creation in GE can be difficult, particularly with high precision. We propose to seed the grammar with constants extracted from the source code of the program under test to enhance and extend Ariadne’s capability to exploit richer types of dependencies (involving all combinations of both variables and constant values). We compared our results with the original system of Ariadne against a large set of benchmark problems which include 10 numeric programs in addition to the ones originally used for Ariadne
The argument of correctness in refinement-based formal software design often disregards source code analysis and code generation. To mitigate the risk of errors in these phases, certifications issued by regulation ent...
详细信息
ISBN:
(纸本)9783030311575;9783030311568
The argument of correctness in refinement-based formal software design often disregards source code analysis and code generation. To mitigate the risk of errors in these phases, certifications issued by regulation entities demand or recommend testing the generated software using a code coverage criteria. We propose improvements for the BTest-Box, a tool for automatic generation of tests for software components developed with the B method. BTestBox supports several code coverage criteria and code generators for different languages. The tool uses a constraint solver to produce tests, thus being able to identify dead code and tautological branching conditions. It also generates reports with different metrics and may be used as an extension to the Atelier B. Our tool performs a double task: first, it acts on the B model, by checking the code coverage. Second, the tool performs the translation of lower level B specifications into programming language code, runs tests and compares their results with the expected output of the test cases. The present version of BTestBox uses parallelisation techniques that significantly improve its performance. The results presented here are encouraging, showing performance numbers that are one order of magnitude better than the ones obtained in the tool's previous version.
More than eighty different test environments need to be created and maintained for debugging the Marvell Ethernet PHY chip if the traditional industrial verification methodology is being used. This can easily incite v...
详细信息
ISBN:
(纸本)9781538635063
More than eighty different test environments need to be created and maintained for debugging the Marvell Ethernet PHY chip if the traditional industrial verification methodology is being used. This can easily incite very complicated debugging procedures and cause the problems and concerns of a multitude of engineering resources. The latest Marvell Ethernet PHY IC UVM verification platform integrating with the Marvell selected IP vendor's UVM library has been developed. A unique centralized-management methology block/architecture idea is brought in to build this UVM verification platform for the latest Marvell Ethernet PHY integrated circuit. All the eighty different test modes can be tested and verified in the same single UVM platform environment. This UVM verification platform environment significantly reduces the number of engineering resources needed to create and maintain the test cases. It also greatly saves debugging time and reduces chip development time. In the meanwhile, a novel random input stimulus controlled variables table idea is also implemented in this UVM verification platform to manage and improve the function coverage much more easily and efficiently.
暂无评论