Key Aspects of Risk-Based Product Testing Managed by Software Testing Managers
Generally the project managers get scared with the thorny word “Risk”. It appears scaring like police, taxes or the death. Project managers live with the fact that it is impossible to test everything. Testing is nothing but a sample control. There is an involvement of risk in sample control & there always remains a risk of overlooking faults in the untested areas.
Testing is certainly the best & the major defense against risk, and an intelligent software testing manager knows how to use the risks to his/her best advantage in getting more resources and time, and distribute them according to the level of risk.
Two aspects of risk can be represented in the form of following equation
Risk level = Probability x Effect
Therefore it is evident that if we have no “Probability” or no “Effect” there is no risk.
The risks having a risk exposure more than zero are the risks we need to tackle very carefully.
Type of Risks: Risks that affect different places are:
1) Business risks
2) Process risks
3) Project risks
4) Product risks
In this article, the fourth type of risk i.e. the product risk is described in greater details.
1) Business risks: These risks are threatening the entire organization from the business point of view
2) Process risks: These risks are related to the processes or the way work is performed.
3) Project risks: These risks are related to the project and the successful completion of the project. In the situation it may be late with the developers incurring some penalty clause, or reduced probability of gaining some later contract.
4) Product risks: Project risks and product risks can influence and be the cause of each other. A project risk may cause a product risk, and a product risk may cause a project risk. Out of different risks that the project may present, the risks related to the final product need be analyzed since some features are more risk bearing than others. In the situation the product may not work properly with the result that some penal loss may be imposed on the developers. It could be that the users may use the product “incorrectly,” in a way that was not allowed for and which, while no blame is associated with the developers, might prejudice the probability of the developers getting similar contracts in future.
The risks threatening the product are the software testing engineer’s main point of concern. They are the risks of defects remaining in the product when it is delivered. We want to deliver the required quality and reliability. This cannot be tested into the product at the end of the development, but must be worked into the product through the work products produced during development and in the implementation of the components.
Best way is to recognize & track the product related risks from the moment requirements are written. While it may not be possible to assign a cost to the risk from the beginning, it should be possible to assign a hazard potential (High, Medium, or Low) to each requirement such that it can be eventually transformed into a risk.
In this way we can possibly do the following:
1) Identify the features most in need of early prototyping, or formal specification.
2) Determine the level and rigor of tests needing to be written.
3) Assign the bug fixing priorities.
4) Assign effort.
5) Monitor the feature status.
6) Identify when the features are likely to be used, from the operational profile that will provide the risk exposure factor.
Source of the Product Risks: Product risks can be originated from the following sources wherein requirements being the prime source.
1) Functional and nonfunctional requirement
2) Missing requirements
3) Ambiguous requirements
4) Misunderstood requirements
5) Requirements on which stakeholders do not agree
Product risks have adverse effect on the customer satisfaction. Product risks may be related to different requirement types, such as functionality, safety, and security and political and technical factors.
Few examples of product risks are:
1) A small, but important functionality has been overlooked in the requirements and is therefore not implemented
2) A calculation of discounts is wrongly implemented, and the customer may lose a lot of money
3) The instrument may reset to the default values if it is dropped on the floor
4) It is possible to print a report of confidential customer information through a loophole in the reporting facility
5) An installation procedure is difficult to follow
These risks are the main concern of the testers, since testing may mitigate the risks. The test strategy for a product should be based on the product risks that have been identified and analyzed.
Relationship between Product Testing and Risk Management: Testing and management of risks are generally very tightly interwoven as they support each other. Testing can be based on the results of risk analysis, and test results can give valuable feedback to support continuous risk analysis.
The result of a product risk analysis can be used in test planning to make the test as effective as possible. It can be used to target the testing effort, since different types of testing are most effective for different risks.
The risk analysis results can also be used to prioritize and distribute the test effort. The areas with risk exposure are generally planned for testing first and given the most time and other resources.
Finally the product risk analysis can be used to qualify the testing already done. If test effort is related to risks it is possible to report on dissolved and remaining risks at any time.
Testing can dissolve or mitigate the product risks. The testing reduces the probability of sending a product with defects out to the customers, as testing locates failures and the subsequent correction of the defects.
Who are involved in Software Risk Analysis?
Going by the best practice suggested by the experts, risk analysis activity is entrusted to different group of experts forming a unified team. The participants can include users, customers, testers, developers as well as other capable persons who can contribute well.
Risk Perception of different people: People in different professions can have different perceptions on risks.
1) Project Managers: Project managers are generally under the time pressure due to which they are used to compromises. They very well know that even though things may look dark, the heaven is not going to fall.
2) Developers: Developers that is programmers, designers and analysts always feel proud of their work, and they know how it was done. They have really done their best, and they are usually reluctant to accept that there may still be defects left in their piece of work.
3) Testers: Testers usually have a pessimistic view on products. Experienced testers remember their previous experiences where they received objects for testing and got far more failures than they expected.
4) End users: The end users are usually highly failure-tolerant. They tend to remember previous experiences, but what they remember is that even though the system failed, they found a way around it or another way of doing their work.
Scales for Risk Analysis: The analysis of risks uses metrics for probability and effect. When we work with metrics it is mandatory to use agreed and understood scales. This is true in case of risk analysis as well.
We can work with two different kinds of scale, that is:
1) Qualitative scale
2) Quantitative scale
1) Qualitative scale: In this we work with feelings or assessments.
“Effect” can be expressed as “Bad” > “Worse” > “Worst”
“Probability” can be expressed as “Not likely” > “Likely” > “Very likely”
2) Quantitative scale: In this we work with exact measures or numbers.
“Effect” can be expressed as actual cost in $ or Rs. or �.
“Probability” can be expressed as <= 10%, >10% & <= 50%, >50% & <=80%, > 80%
Whichever way to do it, project managers define and agree on scales for both probability and effect before the start of risk analysis,
Estimation of Risk Level:
The risk level is calculated for each of the identified risks & expressed in the form of following equation
Risk Level = Final Effect x Final Probability
The distribution of the final risk level over individual risks is used to plan the test activities. It can be used to prioritize the test activities and to distribute the available time and other resources according to the relative risk level. A test plan based on a risk analysis is more trusted than a plan based on “Gut Feeling.”
The testing results can be used as input to the continuous risk analysis. If we get more defects than expected in a particular area it means that the probability is higher than we expected, and the area hence has a higher risk level. On the other hand if we get small number of defects than expected the risk level is lower.