Dynamics of Reporting System of Software Testing Progress & its Completion
Test managers play an active role in managing the entire process of testing & plan concrete strategies for an effective reporting of completion of the testing effort. A discussion is being presented here to explain the system of test progress management & subsequent reporting on completion.
Test execution, recording, control, re-testing, and regression testing must be continued until we believe that the exit criteria have been achieved.
The purpose of the test progress and completion reporting is to stay in control of the testing and deliver the results of the testing activities in such ways that they are understandable and useful for the stakeholders.
The inputs on which the process of test progress and completion reporting is based are:
1) Test plan
2) Measurements from the test development and execution processes
The activities involved in test progress and completion reporting are:
1) Comparing actual measurements with estimates and planned values
2) Reporting test results
The output comprises of the following:
1) Presentation of test progress
2) Test report
Input to Test Progress and Completion Reporting:
The input from the level test plan that we need for this process is:
1) Scheduling and staffing for the activities
2) Exit criteria
Documentation of Test Progress and Completion Reporting:
The documentation of the progress must be presented in various ways according to who is receiving it. The audience may be the customer, higher management, project management and participants, and testers.
At the completion of each test level a test summary report should be produced. The ultimate documentation of completion is the final test summary report for the entire test assignment.
Activities for Test Progress and Completion Reporting:
Checking for Completion:
A check against the test exit criteria is mandatory before we can say that the testing is completed at any level. To warrant a stop it is important to ensure that the product has the required quality.
The exit criteria are tightly connected to the coverage items for the test, the test case design techniques used, and the risk of the product. The exit criteria therefore vary from test level to test level.
Examples of exit criteria are as under:
1) Specified coverage has been achieved
2) Specified number of failures found per test effort has been achieved
3) No known serious faults
4) The benefits of the system as it is are bigger than known problems
If the exit criteria are not met the test cannot just be stopped. Iteration in the test process must take place: We have to go back to where something can be repeated to ensure that the exit criteria are fulfilled.
In most cases additional test procedures are required. This means that the test analysis and design process must be revisited and more test cases and procedures added to increase coverage. These test procedures must then be executed, and the results recorded and checked. Finally the checking of the exit criteria must be completed.
As an alternative, we can revise the test plan by adding relaxation to the test exit criteria or making it stricter.
All changes made in the test completion criteria need to be properly documented. It is better to firstly identify the associated risks & reach a consensus with the customer on different changes. Changing the test plan by adjusting the completion criteria should be regarded as an emergency situation and be very well accounted for.
When all test completion criteria are met and the report approved, the test object can be released. Release has different meanings at different points in the development life cycle:
1) When the test is a static test the test object (usually a document) can be released for use as the basis for further work.
2) When the test is a test level for dynamic test the test object is progressively released from one test level to the next.
3) Ultimately the product can be released to the customer.
Metrics for Progress and Completion Reporting:
Metrics to be defined for the progress and control activities themselves can include:
1) Number of tasks commenced over time
2) Task completion percentage over time
3) Number of task completed over time
4) Time spent on each task over time
This will of course have to be compared to the estimates and schedule of the test progress and completion activities.