After changes have been tested in the programmerís personal libraries &
released for carrying out the software testing of the overall system, the
decision must be made as to how much systems & regression testing is
required. If possible, the entire test data set should be rerun. If this is
viewed as difficult or too expensive, the Retest Planning Matrix is one tool
that can be useful for planning the change testing needed. This matrix can also
be used as a routine part of the software testing in the system wide planning.
Once completed, it is useful for guiding re-testing during development & for
determining the testing required for later maintenance changes.
On careful examination of the following sample retest planning matrix, we can
see that the matrix relates test cases (folios, test procedures, or just
individual cases) to system programs & modules. A check entry indicates that
the case is to be re tested when the module is changed. A question mark (?)
signals that re-testing will be required in some cases & not in others -
that is, we must examine the change to determine whether the test is necessary.
No entry means that the test is not required. In the following example, we have
twenty modules & fifty test cases. The completed row tells us that case
number 3 is to be rerun whenever modules 2, 9, 15, 16, or 20 are changed. The
completed column tells us that if we make a change in module 2, we have to rerun
test cases 2, 3, 8, 49, & 50 & possibly tests 7 & 9.
Completing a retest planning matrix during initial development takes very
little time. The planning matrix may be reviewed by system users & will help
show the level of re-testing expected whenever changes are made. Only minor
continuing effort is required to keep the matrix latest. Whenever modules are
changed, the matrix should be reviewed to assure that it still indicates an
appropriate re-testing plant. If new cases or modules are added, then new rows
or columns are simply added to the existing matrix. The matrix may also be
generated automatically by doing the instrumentation of every module so as to
trigger a utility, which could indicate which modules are executed by which test
The retest-planning matrix helps us to decide what should be re-tested. The
data needed to execute the selected cases will be available if we have
maintained it since the initial development of the system. If not, the cost of
recreating it quickly becomes prohibitive. It is impractical to have to start
from scratch & prepare tests for small changes. Constructing new test data
every time a change is made takes a very long time. To test changes effectively
we must save the test data for reuse!
It should be emphasized that running the complete system test set i.e. full
regression run of all test cases, is necessary in high-risk situations to ensure
that all functions & system capabilities continue to perform as required
after a change has been made. Even assuming a "perfect" change, it is possible
for systems to fail because of "old" faults already existing in the system.
Short of re-testing all functions, there is no way to prevent such faults from
causing a major failure or unacceptable system behavior!
When the cost or time required for full regression is very high, it is
helpful to group or make batches of the changes to be tested. Overall system
wide software testing for a group of related changes takes no more effort than
testing any one individually. Batch making of fixes & corrections minimizes
re-testing effort & justifies more comprehensive software testing.
Maintenance organized so that changes are grouped or batched & only applied
at regular intervals is called scheduled maintenance.
established for each system, generally quarterly or six monthly. Changes are
held & applied all at once at the designated interval. This permits a single
design review of all the proposed changes, avoids having to change the same code
multiple times, & allows a thorough systems & acceptance test to be
carried out before the changes are made effective. It greatly improves change
quality & saves a lot of money at the same time. Of course, if too many
changes are grouped, then debugging becomes complicated by added difficulty in
quickly isolating the source of the problem.