Expert Strategies for Maintenance Testing of Changes in Existing Software
In large number of organizations the major chunk of the useful time is not spent on the new development, rather it is spent on making changes to the existing systems. We know that testing such maintenance changes properly is fundamental if ongoing software quality is to be maintained.
Testing changes is extremely important to the new development. We can not afford to have the luxury of testing a static product. Planning for the needs of software testing is to be done while the system is changing. Fixes & corrections are applied as we move along, & the testing of changes & system re-testing remains a major concern.
In both the maintenance & the new development environments, special focus is given to the testing changes. Having accepted the fact that some changes have been made,
the following three basic questions the software testing manager need to answer are:
Q. 1: What has to be re-tested to provide proper confidence that the system will continue to perform as intended?
Q. 2: What amount of testing is required?
Q. 3: How should the testing be structured & who should perform it?
Although all the above questions are of great importance, yet in majority of the organizations these get ignored. If software testing itself is improvised & poorly structured, testing software changes do not even get considered. Any change, no matter how much small it might be, yet involves its own mini development cycle covering at least the following activities.
1) Planning of changes by the software testing manager that his/her team is going to make. This requires understanding the system or program logic; understanding the intended new result, & coming up with the changes that have to be made to achieve it.
2) Next activity is to incorporate the changes. This involves actually changing the code & modifying the system to conform to the new design.
3) Last activity is to implement the changes & install a new system version. This involves replacing the changed modules in the production library, ensuring that all users are familiar with any new impacts, updating documentation etc. etc.All these software testing related activities must be carried out, just as every step of the normal development cycle is tested. The fact that the change might involve only a few statements does not take away the need to test the design, test the changes in the individual modules & test the impact of changes in overall systems. Fact remains that generally most of these changes are not tested sufficiently.
Testing the impact of changes on the overall system:
After changes have been tested in the programmer�s personal libraries & released for carrying out the software testing of the overall system, the decision must be made as to how much systems & regression testing is required. If possible, the entire test data set should be rerun. If this is viewed as difficult or too expensive, the Retest Planning Matrix is one tool that can be useful for planning the change testing needed. This matrix can also be used as a routine part of the software testing in the system wide planning. Once completed, it is useful for guiding re-testing during development & for determining the testing required for later maintenance changes.
On careful examination of the following sample retest planning matrix, we can see that the matrix relates test cases (folios, test procedures, or just individual cases) to system programs & modules. A check entry indicates that the case is to be re tested when the module is changed. A question mark (?) signals that re-testing will be required in some cases & not in others – that is, we must examine the change to determine whether the test is necessary. No entry means that the test is not required. In the following example, we have twenty modules & fifty test cases. The completed row tells us that case number 3 is to be rerun whenever modules 2, 9, 15, 16, or 20 are changed. The completed column tells us that if we make a change in module 2, we have to rerun test cases 2, 3, 8, 49, & 50 & possibly tests 7 & 9.
Completing a retest planning matrix during initial development takes very little time. The planning matrix may be reviewed by system users & will help show the level of re-testing expected whenever changes are made. Only minor continuing effort is required to keep the matrix latest. Whenever modules are changed, the matrix should be reviewed to assure that it still indicates an appropriate re-testing plant. If new cases or modules are added, then new rows or columns are simply added to the existing matrix. The matrix may also be generated automatically by doing the instrumentation of every module so as to trigger a utility, which could indicate which modules are executed by which test procedures.
The retest-planning matrix helps us to decide what should be re-tested. The data needed to execute the selected cases will be available if we have maintained it since the initial development of the system. If not, the cost of recreating it quickly becomes prohibitive. It is impractical to have to start from scratch & prepare tests for small changes. Constructing new test data every time a change is made takes a very long time. To test changes effectively we must save the test data for reuse!
It should be emphasized that running the complete system test set i.e. full regression run of all test cases, is necessary in high-risk situations to ensure that all functions & system capabilities continue to perform as required after a change has been made. Even assuming a “perfect” change, it is possible for systems to fail because of “old” faults already existing in the system. Short of re-testing all functions, there is no way to prevent such faults from causing a major failure or unacceptable system behavior!
When the cost or time required for full regression is very high, it is helpful to group or make batches of the changes to be tested. Overall system wide software testing for a group of related changes takes no more effort than testing any one individually. Batch making of fixes & corrections minimizes re-testing effort & justifies more comprehensive software testing. Maintenance organized so that changes are grouped or batched & only applied at regular intervals is called scheduled maintenance.
Schedules are established for each system, generally quarterly or six monthly. Changes are held & applied all at once at the designated interval. This permits a single design review of all the proposed changes, avoids having to change the same code multiple times, & allows a thorough systems & acceptance test to be carried out before the changes are made effective. It greatly improves change quality & saves a lot of money at the same time. Of course, if too many changes are grouped, then debugging becomes complicated by added difficulty in quickly isolating the source of the problem.
1) Changes must be designed, built, & implemented, & testing each step is necessary to have proper confidence that the change will work as expected.
2) Testing every smaller module is important, hence Every module undergoing changed must be carefully tested.
3) Testing of the impact of changes on the overall system is extremely important, hence careful planning is required to determine the tests needed. Full regression re-testing should be done whenever possible.
4) We must maintain & test data sets & use them for re-testing whenever changes are applied.
5) We must group or batch the changes together for testing & review whenever possible!.
6) We must test the installation readiness thoroughly before making the change effective.