Mutation Testing an Important Black Box Software Testing Technique
Mutation testing also known as mutation analysis is one of the black box software testing strategies that are critical to the reliability of the ultimate software application. In mutation testing the product under test is changed under controlled ways and then tested. Here one-token defects, like "<" replaced with "<=," are created in the components. For each of these defects a new version of the affected components is created and tested to see if the prepared test cases reveal the defects. If they don’t they will have to be examined and corrected to find the planted defects - and one hopes, more real defects as well.
Mutation testing technique is based on software faults & it is aimed at finding out the adequacy of the tests by estimating the potential of a particular test suite in differentiating a particular program or mutant from another mutant. Generally, introducing a single, small, legal syntactic change in the software creates a mutant. Given a program "P" and corresponding test suite "T", the first step of the mutation testing is to construct the alternative programs - that is, mutants, (P). Then each mutant and the original program "P" has to be executed against each test case. If the
output of mutant "Q" remains the same as the output of the original program "P", we consider "Q" as alive; otherwise, "Q" is considered to be dead or distinguished. "Q" remaining alive can be due to the consequence of two factors like:
1) Inadequate test data
2) The mutant is equivalent to the original program.
If the mutant is equivalent to the original program, then it will not contribute to the adequacy of "T" and should be eliminated.
After equivalent mutants are removed, the number of the remaining mutants that are detected by the test suite can be used to evaluate the corresponding test adequacy. This is formally defined as mutation score:
For early mutating testing techniques, the total number of mutants is usually a very large number, and therefore, it is very expensive to carry out mutation testing. Software testing experts through recent research have declared that by carefully selecting appropriate subsets of mutants, mutation testing can be more efficient and effective, and mutant strength can be equivalent to data-flow testing or domain testing in its ability to determine program correctness.
Thus it is evident that, even though traditional mutation testing demonstrates high potentials to reveal bugs in a system, the mutation-testing technique typically requires the source code of the program under test, and for complex and large software, it is usually very expensive to be performed.
Then, an important question arises as to why do we consider mutation testing for software component testing where source codes for components may not available and where efficiency also has a very high priority for software component testing techniques?
The answer lies in the fact that component interfaces are the point of interaction, and instead of creating mutants based on the program source code, mutants of software components can be created by syntactically modifying the component interface specification. The interface is the only way to interact with a component, and therefore, this type of mutation testing may provide sufficient confidence to the component. Moreover, for traditional mutation testing, the total number of mutants is often proportional to the square of the number of lines of code. As for mutation testing, the total number of mutations will be proportional to the number of lines of interface specification, which is a much smaller number, and consequently, efficiency could be guaranteed.
To adequately create mutants based on component interfaces, typical interface errors need to be studied carefully. Testing experts have identified some typical interface-related faults that include wrong specifications and wrong parameter orders. For instance for CORBA-IDL component interfaces, following five distinct mutation operators are available according to the errors. These can be used with other type of component interfaces as well, but with suitable modifications.
1) Replace operator: For instance in CORBA-IDL interface specification, it needs to specify the parameter as "in", "out", and "inout", which means input, output or input-and-output parameter. The replace operator will replace one of the parameter types with another one in the list.
2) Swap operator: In the interface specification, the order for parameters with the same type requires additional caution. Swap operation will change the order of this type of parameter to test possible specification errors.
3) Twiddle operator: Twiddling is done for numerical or character variables that substitute x with succ(x) or pre(x).
4) Set operator: The set operator will assign a certain value to a parameter. The value usually derived form boundary analysis.
5) Nullify operator: Nullify operator nullifies an object reference.
Similar to traditional mutation testing techniques, for interface-based mutation testing an interface-based mutation score can be defined as D/N where "D" is the number of mutants that are distinguished and "N" is the number of nonequivalent interface mutants that are generated. Correspondingly, the adequacy criterion requires the interface mutation score to be 1.
Many More Articles & Tutorials on Black Box Testing