Ethical Rules Practiced during Testing and its Exit Criteria
Software testing managers declare some sort of well defined exit criteria to ensure that they have some form of closure to their testing strategy for a specific product. This product or service may receive further testing but, if the testing team is part of a development sequence, there must be some well-defined criteria that tells when the product is sufficiently well developed to be released to the customers.
If we are using failure mode and effects analysis (FMEA) tool during product development, we will ascertain if we have developed our verification and validation plans to respond to this document. Because the FMEA approach is a systematic effort to eliminate significant issues before they actually become problems, we need to ensure that our test documents are designed to reflect such a need.
When do we say, Now it’s enough of testing?
If we do not define our “enough,” we can go on testing forever, trying to achieve
# Adequate sample sizes
# Sufficient durations to reveal time-based flaws
# Sufficiently random representation of the population of parts
No matter how important these considerations may be, we must choose a point at which we make a business decision to release the product. We might support our decision, as with software, by
comparing our defect arrival rates with a Rayleigh distribution – releasing when we are well out in the tail under the assumption that this model is telling us that our defects are few.
How do we compare with the Plan?
One of the reasons we plan is to provide a means for controlling the scope of our testing. Even if our testing is less than perfect – testing is always less than perfect – we still have recourse to our plan to control the sequence and completion of activities. A plan does not mean that we are locked into a specific course of action, but it provides structure as well as a benchmark against which we can compare our actual set of actions. We can also build into our plan the increase in the number of test cases and the concurrent decrease in remaining issues found (Refer following Figure).
Can we say that our testing is completed?
We say that testing is never perfect because we do not have the following needs ever satisfied:
# Monetary resources are limited.
# Human resources are often scheduled for other activities.
# Important documents like the design (DFMEA) may not be constructed well and updated promptly.
# Sample sizes are usually too small.
# Statistical confidence is often a nearly meaningless value.
We are not saying that we should not test. What we are saying is that we must understand the limitations of software testing. We must ensure that we are using all the tools available with us to produce a great product.
Ethics of Software Testing:
Good software testing managers maintain & follow certain ethics during the entire testing life cycle, few of them are being shared here.
A) Ethics related to our Test Plan:
The minimum test plan we must execute against a product will be derived from the customer specification or our own standard, if we have one. We also recommend that our test group conduct product tests to failure and thence to destruction (if destruction makes sense in the characterization of the product). We should submit the compliance test plan for approval by the customer. This document may also contain some tests, verifications, and validations we have recommended.
The characterization test plan describes activities we do for our own benefit. It is not necessary that this document be revealed to the customer. They did not ask for it, they are not paying for it, and they should not get it. This test plan is the safeguard against internal and external design and manufacturing incompetence.
B) Ethics related to the Test Reports:
When we execute a test plan, we generally update the plan in such a way that it transforms into the report – this way, all the relevant information will be contained in a single document. For any test required by the customer, we will indicate the results honestly and clearly. In some ways, we are nicely positioned in the enterprise because we report what is wrong – it is not our responsibility to fix the issues we discover. On the other hand, we have a tremendous responsibility to report everything we see.
The issue becomes more interesting when we execute our internal test plan. Now we wish to report the results as close to concurrently with the testing as we can so that the design people are receiving updates about the product in nearly real-time. For our design verification testing to have any meaning, we must execute meaningful tests as soon as we have working models, prototypes, or parts. For example, we do not have to wait for tooled mechanical parts to commence execution of our characterization testing because we can still test the electronics.
C) Ethics related to Customer Relationship:
Communication of faults before the product launch:
Before the launch of a new product, our team should communicate discovered faults only if they need customer input or if the correction of those faults will cause a launch delay. Otherwise, we see nothing unethical in not reporting faults that will be corrected unless we have people using prototype parts who might become injured from the fault.
Prototypes and working models should be clearly identified as such before they leave our plants. If the customer does not want these pieces to be identified this way, we need a written release from the customer before we can ship pre-launch parts to them. At this stage, when we are shipping pre-launch parts, we should indicate
# Current status of the development
# Current status of the testing
# Potential field issues with unimplemented features
# Potential field issues with features implemented incorrectly
Communication of faults after the product launch:
If we detect a fault after the product launch, we must report it to the customer. We are now selling the product, and it would be unethical to do otherwise, particularly if we are dealing with any safety issues. In some cases, our reports on defects or faults may initiate a campaign for or a recall of the product. If the customer finds the fault significant enough for the black-eye of a recall, we can consider it significant. Although recalls are financial catastrophes, they are far, far better than the legal battles that could follow tragic accidents.
It is unethical to conceal negative test results from customers after the product has been launched and we have material in the field. While it is sometimes seen that the customers become extremely unhappy, in the final analysis, the entire incident proceeds more smoothly than might otherwise have been the case. In situations where executives are seen trying to “spin” the topic, the ultimate customer results were consistently negative. Hence , never even try to spin a product fault!
Ethics of communicating the launch delays:
Any test issue that will potentially result in a delay of the product launch must be communicated to the customer at the earliest possible moment. The customer often has their own set of launch plans, and a delay in launching our product may throw off their schedule. Once we have reported the issue, the situation then becomes a commercial problem. If we are working for a fairly large organization, we will have a marketing department that is more competent than we are to deal with the effects of the launch delay.
Although launch delays can result in customer dissatisfaction, we feel that it is better to run the product through a complete test suite rather than try to abbreviate the test sequences to appease the marketing staff. If our documentation shows that we slacked off or that we rushed, we may have problems if we ever have to go to a court of law. Hence, it is better that the organization adequately negotiates a launch delay with the customer that allows for completion of all testing up through system testing.
It is a known fact that testing is infinite. Nobody ever truly completes testing – they just run out of time. At the most following must be ensured:
1) Intelligent software testing managers must make sure that their team at least achieves a responsible amount of testing that they can define with the following:
# Use of FMEA
# Product characterization
# Test to failure
# Test to destruction
# Instantiation of relatively realistic tests
2) All reports should be brutally honest, reporting every anomaly and observation that are seen. Obscuring test data and conclusions is unethical and could lead to severe repercussions if our organization has to go to court over product misbehavior.
3) Software testing engineers, are like the gatekeepers. These are the people who should raise their voice and say, “You shall not pass through here.” Our responsibility is always to
# Protect the interest of our customers
# Never send trash to a customer (whether it is an internal customer or an external one)
# Practice safety first.