comparing our defect arrival rates with a
Rayleigh distribution - releasing when we are well out in the tail under the
assumption that this model is telling us that our defects are few.
How do we compare with the
One of the reasons we plan is to provide a means for
controlling the scope of our testing. Even if our testing is less than perfect -
testing is always less than perfect - we still have recourse to our plan to
control the sequence and completion of activities. A plan does not mean that we
are locked into a specific course of action, but it provides structure as well
as a benchmark against which we can compare our actual set of actions. We can
also build into our plan the increase in the number of test cases and the
concurrent decrease in remaining issues found (Refer following
Can we say that our testing is
We say that testing is never perfect because we
do not have the following needs ever satisfied:
# Monetary resources are limited.
# Human resources are often scheduled for other activities.
# Important documents like the design (DFMEA) may not be constructed well and
# Sample sizes are usually too small.
# Statistical confidence is often a nearly meaningless value.
We are not saying that we should not test. What we are saying is that we must
understand the limitations of software testing. We must ensure that we are using
all the tools available with us to produce a great product.
Ethics of Software Testing:
Good software testing managers maintain & follow certain ethics during
the entire testing life cycle, few of them are being shared here.
A) Ethics related to our Test
The minimum test plan we must execute against a product
will be derived from the customer specification or our own standard, if we have
one. We also recommend that our test group conduct product tests to failure and
thence to destruction (if destruction makes sense in the characterization of the
product). We should submit the compliance test plan for approval by the
customer. This document may also contain some tests, verifications, and
validations we have recommended.
The characterization test plan describes activities we do for our own
benefit. It is not necessary that this document be revealed to the customer.
They did not ask for it, they are not paying for it, and they should not get it.
This test plan is the safeguard against internal and external design and
B) Ethics related to the Test
When we execute a test plan, we generally update the
plan in such a way that it transforms into the report - this way, all the
relevant information will be contained in a single document. For any test
required by the customer, we will indicate the results honestly and clearly. In
some ways, we are nicely positioned in the enterprise because we report what is
wrong - it is not our responsibility to fix the issues we discover. On the other
hand, we have a tremendous responsibility to report everything we see.
The issue becomes more interesting when we execute our internal test plan.
Now we wish to report the results as close to concurrently with the testing as
we can so that the design people are receiving updates about the product in
nearly real-time. For our design verification testing to have any meaning, we
must execute meaningful tests as soon as we have working models, prototypes, or
parts. For example, we do not have to wait for tooled mechanical parts to
commence execution of our characterization testing because we can still test the
C) Ethics related to Customer
of faults before the product launch:
Before the launch of a
new product, our team should communicate discovered faults only if they need
customer input or if the correction of those faults will cause a launch delay.
Otherwise, we see nothing unethical in not reporting faults that will be
corrected unless we have people using prototype parts who might become injured
from the fault.
Prototypes and working models should be clearly identified as such before
they leave our plants. If the customer does not want these pieces to be
identified this way, we need a written release from the customer before we can
ship pre-launch parts to them. At this stage, when we are shipping pre-launch
parts, we should indicate
# Current status of the development
# Current status of the testing
# Potential field issues with unimplemented features
# Potential field issues with features implemented incorrectly
Communication of faults after the product
If we detect a fault after the product launch, we
must report it to the customer. We are now selling the product, and it would be
unethical to do otherwise, particularly if we are dealing with any safety
issues. In some cases, our reports on defects or faults may initiate a campaign
for or a recall of the product. If the customer finds the fault significant
enough for the black-eye of a recall, we can consider it significant. Although
recalls are financial catastrophes, they are far, far better than the legal
battles that could follow tragic accidents.
It is unethical to conceal negative test results from customers after the
product has been launched and we have material in the field. While it is
sometimes seen that the customers become extremely unhappy, in the final
analysis, the entire incident proceeds more smoothly than might otherwise have
been the case. In situations where executives are seen trying to "spin" the
topic, the ultimate customer results were consistently negative. Hence , never
even try to spin a product fault!
Ethics of communicating the launch
Any test issue that will potentially result in a
delay of the product launch must be communicated to the customer at the earliest
possible moment. The customer often has their own set of launch plans, and a
delay in launching our product may throw off their schedule. Once we have
reported the issue, the situation then becomes a commercial problem. If we are
working for a fairly large organization, we will have a marketing department
that is more competent than we are to deal with the effects of the launch
Although launch delays can result in customer dissatisfaction, we feel that
it is better to run the product through a complete test suite rather than try to
abbreviate the test sequences to appease the marketing staff. If our
documentation shows that we slacked off or that we rushed, we may have problems
if we ever have to go to a court of law. Hence, it is better that the
organization adequately negotiates a launch delay with the customer that allows
for completion of all testing up through system testing.
It is a known fact
that testing is infinite. Nobody ever truly completes testing - they just run
out of time. At the most following must be ensured:
1) Intelligent software testing managers must make sure that their team at
least achieves a responsible amount of testing that they can define with the
# Use of FMEA
# Product characterization
# Test to failure
# Test to
# Instantiation of relatively realistic tests
2) All reports should be brutally honest, reporting every anomaly and
observation that are seen. Obscuring test data and conclusions is unethical and
could lead to severe repercussions if our organization has to go to court over
3) Software testing engineers, are like the gatekeepers. These are the people
who should raise their voice and say, "You shall not pass through here." Our
responsibility is always to
# Protect the interest of our customers
# Never send trash to a customer (whether it is an internal customer or an
# Practice safety first.