Meeting Shipment Deadline or Removing Bugs-Which Challenge is Greater
There is an expert statement written on the wall that, �No matter how much testing you do, it will never be enough�. Very rarely you come across a testing project which despite the best efforts finishes with time to spare. Instead, the final stages of most tests seem more like a mad rushing towards the finishing line. As the scheduled deadline approaches near & series of defect reports continue to mount pressure, nervous project managers start looking for workable solutions. In the worst scenario an order might shoot out to stop finding any more bugs so that the software can be shipped.
Testing managers are well aware of the fact that customer too plays an important role while doing some testing at its own end. However it doesn�t mean either that you can pass your buck on to the customer simply because your software testing couldn�t finish on time.
Million dollar decision – Ship or continue testing?
is not necessary to spend long time in testing before you come face to face with a simple reality: code ships before it�s perfect. The fact becomes apparent from the number of fix packs that supersede almost all-complex commercial software. Software testing engineers don�t like this reality too much. They would rather see the code they are testing be defect-free before it goes out of the door, but that is not realistic.Experts declare that all complex software has bugs. Together with another popular software testing axiom, namely that we can�t test quality into a product, and the conclusion is inescapable: software testing engineers will never be satisfied with what gets shipped. They always tend to wish that another bug could have been removed. The trick is to get beyond the desire for perfection to be able to see the difference between software that is ready to go, and software that is truly in trouble.
Good exit criteria can be of great help here. But even when it is clear that the software isn�t ready, the challenge comes up as to what should be done and matured and convincing decision-makers really do it.
Consequences of premature termination of testing:
If pressure to exit a test is intense even though the job isn�t finished, one of the most persuasive arguments for restraint involves the potential for disastrous effects down the line.
Clear Identification of Risks:
First, identify the risks. Try to probe the answers to the following questions:
# If there known defects that have not been fixed? The answer to this question is invariably �yes.�
# How severe are such defects, both individually as well as cumulatively?
For instance, a single bug that causes the corruption of critical data is catastrophic. While many other bugs may not be too much troublesome when looked at individually, but when viewed as a group clustered in a particular function they may be crippling.
# Are there any major functional areas that have not been fully tested, perhaps because scenarios are blocked by open defects?
# Are such areas optional, or they are central to the software�s operation? Try to project worst-case and probable-case failure scenarios for those areas, based on defect trends the testing has revealed so far.
Assessment of Impacts in future:
Once the risks are identified, assess their potential impact on the software�s next consumer.
# If another test phase is due to receive the code, how likely are they to be affected by the risk?
# Is it even worth their while to start in light of known deficiencies, or will those problems cause so much redundant error discovery and debug that starting now will actually lengthen the project�s overall timeline?
# Are areas that have yet to be tested critical to the next test phase�s planned activities? Can their entry criteria be met?
# If the software is decided to be shipped to the customers, it is essential to analyze as to how will they react to the identified risks?
# Will they get scared from them? If so, will they refuse to install the software?
# If not, what happens when they put the code into production and those risks manifest themselves?
# Will their business be seriously impacted?
# Will their confidence in the product be badly shaken? Will their satisfaction level drop off a drastically?
# Will there be any possibility of denouncing the product at user group conferences or in Web chat rooms?
By coolly articulating both the potential risks and likely impacts of prematurely ending a test, intelligent project manager removes the hysteria surrounding an approaching end date, both from the minds of software testing engineers who want a perfect product and the testing managers under pressure to release it. The team can then focus on devising realistic solutions.
Shipment Date is on the Neck & Testing is still On: What should be done?
Despite Herculean efforts by the software testing and development teams, we have reached the end of the line. The end date has come, and the test team has not achieved its exit criteria. Major bugs remain unresolved, significant chunks of testing are not complete, or both.
You understand that the code may never be perfect, but as a tester and the software�s first real user, you can�t sign off on releasing it as is. It is the time for taking crucial decision.
First, a reality check: as a tester, the choice of whether or not to ship a product is not yours to make. It remains a business decision to be made by senior managers who will take into account many factors, only one of which is test status. However, it is likely that the test team will have an opinion on what that decision should be. What solutions might you recommend? How can you convey your opinion in a way that�s convincing?
There can be four possibilities like:
1) Postponing the shipment:
This one is every tester�s favorite, and sometimes it is the best answer. If problems in the software are serious and pervasive or testing is far behind in major functional areas, there may be little choice but to postpone the end date.
Such a choice is rarely made lightly, however. If the test team feels strongly that a delay is required, they must present their rationale to the decision makers in a precise and powerful way.
In such situations, data speaks. Your problem tracking and test measurement charts should provide a solid base, but may not be enough.
You should try to hunt for ways to dig out the extraneous information and present your core message in a highly compelling way.
For instance, if you want to convey that many areas of test are blocked, that story may not come through clearly in a standard �S� curve showing overall test progress. Instead of that you could possibly create a bar chart, with each bar representing a major functional area of the software and showing its planned versus actual accumulation of testing points. If many bars register a low score, the true state of the situation becomes obvious.
2) Pulling out few functions out of the package:
If the trouble is focused in one or two distinct areas of the software, another solution might be to keep the end date intact but pull those specific functions from the package.
Sometimes this is easy to accomplish, sometimes it is not. If the function is intertwined with the rest of the offering, pulling it may not be realistic. In that case, it is sometimes possible to isolate such a function, leaving it in place but disabling it. In fact, if testers are involved during the code design phase, they can sometimes sway the development team to create code that is easily disabled in case it becomes necessary.
Either way, well-presented data can help you make your case. The bar chart for test progress by function can work here as well, but this time it will show only one or two areas as dramatically behind the others. Pair that with a similar bar chart showing open defects by function, backed up with a detailed list of defects, and others should quickly see your point.
3) Creating a list of restrictions until further notice:
If the broken areas can be bypassed by the user, then it may be enough to simply provide a warning. Create a list of restrictions describing functions to avoid until further notice.
This approach can helpful if the next consumer of the software is another test team. If that team is able to stage its testing in such a way as to avoid the problem areas initially, that will give your team and development a chance to clean them up. This approach may also work with beta customers, particularly if there are only a handful of them and they are well known and can be robustly supported. Even in that case, be aware that users have a tendency to ignore lists of restrictions, and then complain aloud when functions don�t work.
A list of restrictions is generally an easy sell, sometimes too easy. Generally It is quite tempting to try to document your way out of a tough spot that really requires more direct action. If the restrictions list is longer than the working functions list, or the broken areas are too embedded in mainline function for a user to avoid them, it�s probably best to consider some other alternative.
4) Planning a calculated risk of an overlap:
Majority of the commercial software products have a window between the date that testing is supposed to finish and the date the software is shipped to customers. This time is generally used to pack the software onto CD-ROMs, finish documentation, etc.
Another approach is to exploit this window to finish any remaining testing, resolve outstanding defects, and create a plan to ship a package of fixes at the same time as the product. This remains a risky proposition. There is no guarantee that all critical issues will be fixed in time. Teams deciding to go this route may win the gamble and live to fight another day. Or they might not. It is probably best to consider other options before accepting this one.
Few approaches helpful in easing out shipment deadlines pressures:
Having learned the lesson once, the entire team get wiser & tries to introspectively devise ways & means to be more pragmatic in approach to the projects in times to come. Following are few approaches talked out by experts that can possibly ease out some of the last minute pressure from the project team, provided these are practiced right from the beginning.
1) Systematic march towards the bugs:
Let�s assume your testing is so successful that you uncover an ever-growing mountain of bugs. The biggest problem then becomes getting them fixed. Without fixes, a good software testing engineer eventually becomes a victim of his own success. As the heap of bugs mounts, so does the list of test cases for which progress is blocked. Eventually all testing avenues become choked with defects, and there�s little for the tester to do but watch the clock as the end date nears and testing stagnates.
You need a strong tool to solve this problem. For many teams, the tool of choice is a set of tried & tested measurements. Good metrics don�t necessarily serve up convenient answers so much as lead people to ask the right questions. Unfortunately, when used improperly, measurements can become a source of endless busywork, paper shuffling, and fancy charts with a value far below their cost. But when used correctly, a good measurement can effectively communicate to those running the project which defects are putting the end date in jeopardy, so the key ones are fixed promptly.
2) Approach of breaking down to basic essential elements:
Your test plan will have a list of variations or scenarios to execute. Perhaps the simplest technique for measuring your progress is to count how many variations are successful each week and plot them on a graph over time.
However, this method omits important information. It does not account for variations that have been attempted but are not yet successful, so it does not present a complete picture of the testing effort which has been spent so far nor the effort that is pending. It also does not draw attention to tests that are blocked or the defects that are blocking them.
3) Multistate Approach:
A better approach is to view each variation as being in one of five states:
i) Not Attempted State: The variation has not yet been tried.
ii) Attempted State: Work has begun on the variation, but isn�t finished. This state may indicate an elaborate variation that requires considerable work, one that was started but became sidetracked due a setup error, or one that stumbled for some other reason still under investigation.
iii) Failed State: The target code failed the test due to a defect.
iv) Blocked State: A defect has been hit by another variation which will also affect this one. It�s pointless to attempt this variation until the defect is fixed. A variation might also be marked as blocked if delivery of the code it is intended to test has been delayed.
v) Successful State: Everything worked as planned.
Other states are also possible. For instance, you might include an �unblocked� state for variations that were previously blocked but now are available to execute. You could also include a �fixtest� state for variations that were previously marked as failed but now have fixes available. You can probably think of other useful states, but these initial five make a good core set.
This finer-grained classification offers a more accurate picture of each variation�s status, and clearly identifies those that are impacted by defects. But it still has some weaknesses. It doesn�t distinguish between a variation that takes three minutes to execute and one that takes three days. All are treated equally, so the true effort levels aren�t reflected. Also, it�s not obvious how to translate these classifications into a simple chart that will clearly convey the team�s status trends. For example, a pie chart could show a snapshot of the current spread of variation states, but a pie chart can�t give a view of progress over time. A series of graphs could be plotted week to week for each possible variation state, but multiple graphs could muddy the team�s main message.
4) Assigning weights on a 10 point scale:
These weaknesses can be addressed through a further refinement: assign a weight to each variation corresponding to the estimated amount of time it will take to perform. For example, you might assign weights on a scale of 1 to 10, with the higher weights associated with the more complex scenarios. Or you could define a weight of one for any scenario that will require some baseline amount of time, then assign a two to any scenario that will take twice that amount of time, and so on.
Many teams have been successful by simply assigning a value of one to any scenario that will require up to a half day, a value of two for a scenario expected to need between a half and a full day, and so on. As long as the same baseline is used for scoring all scenarios, the actual value of that baseline doesn�t matter. What is important is that you capture the relative differences in effort required for the scenarios, not any absolute value.
Combining these weights, or �points,� with target dates for the completion of each variation allows you to create a fairly accurate graph of expected test progress over time. Plot total cumulative points on the Y-axis, and time on the X-axis. As variations are completed, their points are added to the current test total. Partial points can also be claimed on a sliding scale for variations in �attempted� status. Every week (or whatever reporting interval you choose), you plot the new grand total on your graph. The result is a nice, clear chart showing testing progress over time. When scenarios are in a failed or blocked state, you can sum up the points for each of those scenarios and quickly convey the total amount of testing effort that is being delayed by the offending defects or late code deliveries. An example of this is shown in Figure 18.1.
With these relatively simple techniques you can do a reasonable job of conveying the testing effort that�s been expended, the effort remaining, and the impact of defects on testing progress. Undoubtedly, this combination does not deliver a measurement nirvana. But it does the job and requires fairly minimal effort, so more time can be spent on testing than tracking. For many testers, that�s close enough.
5) Tracking & measurement of progress:
Testers are sometimes annoyed by the need to track their progress and link bugs to variations, even when the effort required to do so is minimum. Rather they should not be so. In fact, testers should love tracking this information, because it is a strong tool they can use to ensure that bugs are fixed.
When testing is behind the schedule, the software testing team leader becomes a natural target at project status review meetings. All eyes are on him as he defends his team. He can panic. Or, he can display a simple graph that clearly shows actual versus projected test progress, with the difference representing a realistic, effort-based gap. On the same chart he can include an indicator showing what percentage of that gap is due to unresolved defects. Next, he can show a chart listing all outstanding defects, ordered by the amount of testing progress each is blocking.
At this point, all attention will shift from the test team leader to the development leads. The burden suddenly falls on them to explain why development is behind in fixing defects, and what they are going to do to catch up. The result is that the most critical bugs get the prompt attention they deserve, and maybe testing has a good chance to finish on time.
6) The most crucial measurement:
Despite all such wisdom, what needs to be done is, having constant focus on the most crucial measurement out of them all is – Impact to the customers.
Always ensure the defects with the highest potential for annoying the customers are fixed first, regardless of what the test progress chart happens to show. Fortunately, once such defects are clearly described, getting them fixed is usually an easy sell. Just don�t let fancy measurements get in the way of protecting customers from any harm.
An expert on R&D, Online Training and Publishing. He is M.Tech. (Honours) and is a part of the STG team since inception.