Difference between revisions of "Eiffel Mutation Testing Tool"
| Line 15: | Line 15: | ||
* Instrument Eiffel source code to generate mutant schema file | * Instrument Eiffel source code to generate mutant schema file | ||
* Create mutant objects from mutant schema file and execute test cases on mutants to calculate the mutation score of test cases | * Create mutant objects from mutant schema file and execute test cases on mutants to calculate the mutation score of test cases | ||
| + | |||
| + | === Instrumentation === | ||
| + | ==== Evaluation of different mutation insertion methods ==== | ||
| + | As discussed, there is more than one possible method of inserting mutations into the sourcecode of a class. The first approach we considered is compiling the class under test until degree 3 and then using a changed ET_AST_PRINTER to change the features and print the new mutation schema class to a file. There are of course several up- and downsides in this approach: | ||
| + | |||
| + | Plus: | ||
| + | *Very generic approach, can be done for structural and behavioral mutations | ||
| + | *Clean interface between mutant schema generation and mutant instatiation | ||
| + | *Rather easy to implement, maintain and extend | ||
| + | |||
| + | Minus: | ||
| + | *Slow because the class under test first has to be parsed and compiled (until degree 3) and then the mutant schema has to be written to a file. In the instantisation stage the mutant schema has to be parsed and compiled again. | ||
| + | |||
| + | |||
| + | The second approach would be to change the eiffel parser (geyacc) used to parse the classes under test and insert mutations while parsing. | ||
| + | |||
| + | Plus: | ||
| + | *Rather fast since only parsing has to be done for both the class under test and the mutation schema (and this could also be optimised using the parsed class as interface between instrumentation and instantiation rather then giving a written file) | ||
| + | *Clean interface (if the optimization mentioned before is not applied) | ||
| + | *Implementation not easy but straightforward | ||
| + | |||
| + | Minus: | ||
| + | *This method only works for behavioral mutations, structural changes cannot be done while parsing | ||
| + | *Maintainability and extendability rather limited | ||
| + | |||
| + | |||
| + | As a third possibility Silvio and I discussed changing the interface between instrumentation and instantiaton in such a way that we (based on the first method) only have to compile once for every class under text. This would mean that the instantiation part would work directly on the AST generated and mutated during instrumentation. | ||
| + | |||
| + | Plus: | ||
| + | *Behavioral and structural mutations can be done, thus quite generic | ||
| + | *Rather fast | ||
| + | *Implementation of instrumentation rather easy (same as in method one) | ||
| + | *Extendible solution | ||
| + | |||
| + | Minus: | ||
| + | *No clean interface between instrumentation and instantiation | ||
| + | *Instantiation harder to implement | ||
| + | *Maintainability depends heavily on changes in the compiler (as long as the AST representation isn't changed it would be rather good) | ||
| + | |||
| + | |||
| + | Since, in our (Silvios and mine) opinion, reliability, extendibility and flexibility is more important than speed (although speed is important of course) we will focus on the first approach which will give us two almost completely independant components which can be easily exchanged and extended. | ||
| + | |||
== Milestones == | == Milestones == | ||
Revision as of 06:20, 1 December 2006
Contents
Overview
The whole goal of Eiffel mutation testing tool is:
- Defining a set of efficient Eiffel mutation operators
- Implementing these mutation operators
- Executing test cases on mutants to evaluate their quality
This tool is mainly used for evaluating the quality of test strategies, especially the evaluation of fault detectablity and redundance.
Goals
We aim to implement a completely automatic Eiffel mutation testing tool.
Steps
- Instrument Eiffel source code to generate mutant schema file
- Create mutant objects from mutant schema file and execute test cases on mutants to calculate the mutation score of test cases
Instrumentation
Evaluation of different mutation insertion methods
As discussed, there is more than one possible method of inserting mutations into the sourcecode of a class. The first approach we considered is compiling the class under test until degree 3 and then using a changed ET_AST_PRINTER to change the features and print the new mutation schema class to a file. There are of course several up- and downsides in this approach:
Plus:
- Very generic approach, can be done for structural and behavioral mutations
- Clean interface between mutant schema generation and mutant instatiation
- Rather easy to implement, maintain and extend
Minus:
- Slow because the class under test first has to be parsed and compiled (until degree 3) and then the mutant schema has to be written to a file. In the instantisation stage the mutant schema has to be parsed and compiled again.
The second approach would be to change the eiffel parser (geyacc) used to parse the classes under test and insert mutations while parsing.
Plus:
- Rather fast since only parsing has to be done for both the class under test and the mutation schema (and this could also be optimised using the parsed class as interface between instrumentation and instantiation rather then giving a written file)
- Clean interface (if the optimization mentioned before is not applied)
- Implementation not easy but straightforward
Minus:
- This method only works for behavioral mutations, structural changes cannot be done while parsing
- Maintainability and extendability rather limited
As a third possibility Silvio and I discussed changing the interface between instrumentation and instantiaton in such a way that we (based on the first method) only have to compile once for every class under text. This would mean that the instantiation part would work directly on the AST generated and mutated during instrumentation.
Plus:
- Behavioral and structural mutations can be done, thus quite generic
- Rather fast
- Implementation of instrumentation rather easy (same as in method one)
- Extendible solution
Minus:
- No clean interface between instrumentation and instantiation
- Instantiation harder to implement
- Maintainability depends heavily on changes in the compiler (as long as the AST representation isn't changed it would be rather good)
Since, in our (Silvios and mine) opinion, reliability, extendibility and flexibility is more important than speed (although speed is important of course) we will focus on the first approach which will give us two almost completely independant components which can be easily exchanged and extended.
Milestones
Todo
Team
Philippe Masson
Silvio Kohler
