necessary to spend long time in testing before you come face to face with a
simple reality: code ships before it’s perfect. The fact becomes apparent from
the number of fix packs that supersede almost all-complex commercial software.
Software testing engineers don’t like this reality too much. They would rather
see the code they are testing be defect-free before it goes out of the door, but
that is not realistic.
Experts declare that all complex
software has bugs. Together with another popular software testing axiom, namely
that we can’t test quality into a product, and the conclusion is inescapable:
software testing engineers will never be satisfied with what gets shipped. They
always tend to wish that another bug could have been removed. The trick is to
get beyond the desire for perfection to be able to see the difference between
software that is ready to go, and software that is truly in trouble.
Good exit criteria can be of great
help here. But even when it is clear that the software isn’t ready, the
challenge comes up as to what should be done and matured and convincing
decision-makers really do it.
Consequences of premature termination of
If pressure to exit a test is intense even though
the job isn’t finished, one of the most persuasive arguments for restraint
involves the potential for disastrous effects down the line.
Identification of Risks:
First, identify the risks. Try to
probe the answers to the following questions:
# If there known defects that have
not been fixed? The answer to this question is invariably “yes.”
# How severe are such defects, both
individually as well as cumulatively?
For instance, a single bug that
causes the corruption of critical data is catastrophic. While many other bugs
may not be too much troublesome when looked at individually, but when viewed as
a group clustered in a particular function they may be crippling.
# Are there any major functional
areas that have not been fully tested, perhaps because scenarios are blocked by
# Are such areas optional, or they
are central to the software’s operation? Try to project worst-case and
probable-case failure scenarios for those areas, based on defect trends the
testing has revealed so far.
Assessment of Impacts in future:
Once the risks
are identified, assess their potential impact on the software’s next consumer.
# If another test phase is due to
receive the code, how likely are they to be affected by the risk?
# Is it even worth their while to
start in light of known deficiencies, or will those problems cause so much
redundant error discovery and debug that starting now will actually lengthen the
project’s overall timeline?
# Are areas that have yet to be
tested critical to the next test phase’s planned activities? Can their entry
criteria be met?
# If the software is decided to be
shipped to the customers, it is essential to analyze as to how will they react
to the identified risks?
# Will they get scared from them?
If so, will they refuse to install the software?
# If not, what happens when they
put the code into production and those risks manifest themselves?
# Will their business be seriously
# Will their confidence in the
product be badly shaken? Will their satisfaction level drop off a drastically?
# Will there be any possibility of
denouncing the product at user group conferences or in Web chat
By coolly articulating both the
potential risks and likely impacts of prematurely ending a test, intelligent
project manager removes the hysteria surrounding an approaching end date, both
from the minds of software testing engineers who want a perfect product and the
testing managers under pressure to release it. The team can then focus on
devising realistic solutions.
Shipment Date is on the Neck & Testing is still On: What
should be done?
Despite Herculean efforts by the software
testing and development teams, we have reached the end of the line. The end date
has come, and the test team has not achieved its exit criteria. Major bugs
remain unresolved, significant chunks of testing are not complete, or both.
You understand that the code may
never be perfect, but as a tester and the software’s first real user, you can’t
sign off on releasing it as is. It is the time for taking crucial
First, a reality check: as a
tester, the choice of whether or not to ship a product is not yours to make. It
remains a business decision to be made by senior managers who will take into
account many factors, only one of which is test status. However, it is likely
that the test team will have an opinion on what that decision should be. What
solutions might you recommend? How can you convey your opinion in a way that’s
There can be four
Postponing the shipment:
This one is every tester’s favorite,
and sometimes it is the best answer. If problems in the software are serious and
pervasive or testing is far behind in major functional areas, there may be
little choice but to postpone the end date.
Such a choice is rarely made
lightly, however. If the test team feels strongly that a delay is required, they
must present their rationale to the decision makers in a precise and powerful
In such situations, data speaks.
Your problem tracking and test measurement charts should provide a solid base,
but may not be enough.
You should try to hunt for ways to
dig out the extraneous information and present your core message in a highly
For instance, if you want to convey
that many areas of test are blocked, that story may not come through clearly in
a standard “S” curve showing overall test progress. Instead of that you could
possibly create a bar chart, with each bar representing a major functional area
of the software and showing its planned versus actual accumulation of testing
points. If many bars register a low score, the true state of the situation
Pulling out few functions out of the package:
If the trouble
is focused in one or two distinct areas of the software, another solution might
be to keep the end date intact but pull those specific functions from the
Sometimes this is easy to
accomplish, sometimes it is not. If the function is intertwined with the rest of
the offering, pulling it may not be realistic. In that case, it is sometimes
possible to isolate such a function, leaving it in place but disabling it. In
fact, if testers are involved during the code design phase, they can sometimes
sway the development team to create code that is easily disabled in case it
Either way, well-presented data can
help you make your case. The bar chart for test progress by function can work
here as well, but this time it will show only one or two areas as dramatically
behind the others. Pair that with a similar bar chart showing open defects by
function, backed up with a detailed list of defects, and others should quickly
see your point.
Creating a list of restrictions until further notice:
broken areas can be bypassed by the user, then it may be enough to simply
provide a warning. Create a list of restrictions describing functions to avoid
until further notice.
This approach can helpful if the
next consumer of the software is another test team. If that team is able to
stage its testing in such a way as to avoid the problem areas initially, that
will give your team and development a chance to clean them up. This approach may
also work with beta customers, particularly if there are only a handful of them
and they are well known and can be robustly supported. Even in that case, be
aware that users have a tendency to ignore lists of restrictions, and then
complain aloud when functions don’t work.
A list of restrictions is generally
an easy sell, sometimes too easy. Generally It is quite tempting to try to
document your way out of a tough spot that really requires more direct action.
If the restrictions list is longer than the working functions list, or the
broken areas are too embedded in mainline function for a user to avoid them,
it’s probably best to consider some other alternative.
Planning a calculated risk of an overlap:
Majority of the
commercial software products have a window between the date that testing is
supposed to finish and the date the software is shipped to customers. This time
is generally used to pack the software onto CD-ROMs, finish documentation, etc.
Another approach is to exploit this
window to finish any remaining testing, resolve outstanding defects, and create
a plan to ship a package of fixes at the same time as the product. This remains
a risky proposition. There is no guarantee that all critical issues will be
fixed in time. Teams deciding to go this route may win the gamble and live to
fight another day. Or they might not. It is probably best to consider other
options before accepting this one.
approaches helpful in easing out shipment deadlines
learned the lesson once, the entire team get wiser & tries to
introspectively devise ways & means to be more pragmatic in approach to the
projects in times to come. Following are few approaches talked out by experts
that can possibly ease out some of the last minute pressure from the project
team, provided these are practiced right from the beginning.
1) Systematic march towards the
Let’s assume your testing is so successful that
you uncover an ever-growing mountain of bugs. The biggest problem then becomes
getting them fixed. Without fixes, a good software testing engineer eventually
becomes a victim of his own success. As the heap of bugs mounts, so does the
list of test cases for which progress is blocked. Eventually all testing avenues
become choked with defects, and there’s little for the tester to do but watch
the clock as the end date nears and testing stagnates.
You need a strong tool to solve
this problem. For many teams, the tool of choice is a set of tried & tested
measurements. Good metrics don’t necessarily serve up convenient answers so much
as lead people to ask the right questions. Unfortunately, when used improperly,
measurements can become a source of endless busywork, paper shuffling, and fancy
charts with a value far below their cost. But when used correctly, a good
measurement can effectively communicate to those running the project which
defects are putting the end date in jeopardy, so the key ones are fixed
Approach of breaking down to basic essential elements:
test plan will have a list of variations or scenarios to execute. Perhaps the
simplest technique for measuring your progress is to count how many variations
are successful each week and plot them on a graph over time.
However, this method omits
important information. It does not account for variations that have been
attempted but are not yet successful, so it does not present a complete picture
of the testing effort which has been spent so far nor the effort that is
pending. It also does not draw attention to tests that are blocked or the
defects that are blocking them.
A better approach is to view each
variation as being in one of five states:
Attempted State: The variation has not yet been
Attempted State: Work has begun on the variation, but isn’t
finished. This state may indicate an elaborate variation that requires
considerable work, one that was started but became sidetracked due a setup
error, or one that stumbled for some other reason still under
Failed State: The target code failed the test due to a
Blocked State: A defect has been hit by another variation which
will also affect this one. It’s pointless to attempt this variation until the
defect is fixed. A variation might also be marked as blocked if delivery of the
code it is intended to test has been delayed.
Successful State: Everything worked as planned.
Other states are also possible. For
instance, you might include an “unblocked” state for variations that were
previously blocked but now are available to execute. You could also include a
“fixtest” state for variations that were previously marked as failed but now
have fixes available. You can probably think of other useful states, but these
initial five make a good core set.
This finer-grained classification
offers a more accurate picture of each variation’s status, and clearly
identifies those that are impacted by defects. But it still has some weaknesses.
It doesn’t distinguish between a variation that takes three minutes to execute
and one that takes three days. All are treated equally, so the true effort
levels aren’t reflected. Also, it’s not obvious how to translate these
classifications into a simple chart that will clearly convey the team’s status
trends. For example, a pie chart could show a snapshot of the current spread of
variation states, but a pie chart can’t give a view of progress over time. A
series of graphs could be plotted week to week for each possible variation
state, but multiple graphs could muddy the team’s main message.
Assigning weights on a 10 point scale:
These weaknesses can
be addressed through a further refinement: assign a weight to each variation
corresponding to the estimated amount of time it will take to perform. For
example, you might assign weights on a scale of 1 to 10, with the higher weights
associated with the more complex scenarios. Or you could define a weight of one
for any scenario that will require some baseline amount of time, then assign a
two to any scenario that will take twice that amount of time, and so on.
Many teams have been successful by
simply assigning a value of one to any scenario that will require up to a half
day, a value of two for a scenario expected to need between a half and a full
day, and so on. As long as the same baseline is used for scoring all scenarios,
the actual value of that baseline doesn’t matter. What is important is that you
capture the relative differences in effort required for the scenarios, not any
Combining these weights, or
“points,” with target dates for the completion of each variation allows you to
create a fairly accurate graph of expected test progress over time. Plot total
cumulative points on the Y-axis, and time on the X-axis. As variations are
completed, their points are added to the current test total. Partial points can
also be claimed on a sliding scale for variations in “attempted” status. Every
week (or whatever reporting interval you choose), you plot the new grand total
on your graph. The result is a nice, clear chart showing testing progress over
time. When scenarios are in a failed or blocked state, you can sum up the points
for each of those scenarios and quickly convey the total amount of testing
effort that is being delayed by the offending defects or late code deliveries.
An example of this is shown in Figure 18.1.
With these relatively simple
techniques you can do a reasonable job of conveying the testing effort that’s
been expended, the effort remaining, and the impact of defects on testing
progress. Undoubtedly, this combination does not deliver a measurement nirvana.
But it does the job and requires fairly minimal effort, so more time can be
spent on testing than tracking. For many testers, that’s close
Tracking & measurement of progress:
Testers are sometimes
annoyed by the need to track their progress and link bugs to variations, even
when the effort required to do so is minimum. Rather they should not be so. In
fact, testers should love tracking this information, because it is a strong tool
they can use to ensure that bugs are fixed.
When testing is behind the
schedule, the software testing team leader becomes a natural target at project
status review meetings. All eyes are on him as he defends his team. He can
panic. Or, he can display a simple graph that clearly shows actual versus
projected test progress, with the difference representing a realistic,
effort-based gap. On the same chart he can include an indicator showing what
percentage of that gap is due to unresolved defects. Next, he can show a chart
listing all outstanding defects, ordered by the amount of testing progress each
At this point, all attention will
shift from the test team leader to the development leads. The burden suddenly
falls on them to explain why development is behind in fixing defects, and what
they are going to do to catch up. The result is that the most critical bugs get
the prompt attention they deserve, and maybe testing has a good chance to finish
most crucial measurement:
Despite all such wisdom, what needs
to be done is, having constant focus on the most crucial measurement out of them
all is - Impact to the customers.
Always ensure the defects with the
highest potential for annoying the customers are fixed first, regardless of what
the test progress chart happens to show. Fortunately, once such defects are
clearly described, getting them fixed is usually an easy sell. Just don’t let
fancy measurements get in the way of protecting customers from any