An Insight to Six Principles of Testing Prescribed by Software Testing Experts

An Insight to Six Principles of Testing Prescribed by Software Testing Experts

Following six core principles of testing are prescribed by Great Testing Gurus, & out of them each one carries significant importance for the practitioners as well as the testing managers. These principles help us to better understand what testing is all about and serve to provide a foundation for the methods and techniques used by the software testing engineers.

Six key software-testing principles are:

1) Complete testing is not possible.
2) Testing is creative and difficult.
3) An important reason for testing is to prevent errors.
4) Testing is risk-based.
5) Testing must be planned.
6) Testing requires independence.

The principles, being described here are nothing but an accepted or professed truth that provide us an insight about the software testing discipline.

Testing Principle – 1: Complete Testing is Not Possible
Many programmers believe that they can and do test their programs “thoroughly.” When asking about testing practices, we

generally come across statements like the following:

“I will stop testing when I am sure it works”;

“We will implement the system as soon as all the errors are corrected.”

Such statements assume that testing can be “finished,” & that one can be certain that all defects are removed, or that one can become totally confident of success. This is an incorrect view. We cannot hope to achieve complete testing, and, as we shall see, the reasons are both practical limitations and theoretical impossibility.The total number of possible tests in even the simplest of situations (such as our name search) is essentially infinite. Practically speaking, it is impossible to try all the cases. Instead the tester must select a very small subset of the possibilities involved. The subset must be big enough to provide a reasonable measure or gauge of the performance (that is, it must provide a satisfactory level of confidence), but not so big that it becomes impractical to administer the tests and ceases to provide a significantly enhanced measurement.

Some Practical Limits to Software Testing:
In any form of testing it is impossible to achieve total confidence. The only exhaustive testing there is so much testing that the tester gets exhausted!

Several people have demonstrated that if our objective was to “prove” a program free of defects, then testing is not only practically, but also theoretically, impossible.

These theoretical limits tell us that there will never be a way to be sure we have a perfect understanding of what a program is supposed to do and that any testing system we might construct will always have some possibility of failing. In short, we cannot achieve 100 percent confidence, no matter how much time and energy we put into it.

Testing Principle – 2: Testing Work Is Creative and Difficult
Anyone who has had much experience in testing work knows that testing a piece of software is not simple. Despite this we perpetuate several myths that have caused a great deal of harm.

Users are frequently assigned major testing responsibilities without being given any training in how to perform such work. Programmers, Analysts, and Project Leaders have volumes of standards detailing how development is carried out, but they are left to their own resources for testing. It seems to be assumed that testing is dirty work, that people don’t enjoy it, and that senior-level personnel need not get involved. In a broad sense testing suffers from the same maligned reputation as documentation and maintenance work.

This viewpoint is expressed through the answers of the question “Why testing is not simple?”

# To test effectively we must thoroughly understand a system.

# Systems are neither simple nor simple to understand.

Therefore, testing is not simple.

One can’t begin to design and develop an effective test until one has a detailed understanding of what a system is supposed to do. Managers routinely assign their most senior business analysts to answer such questions and have no trouble recognizing that it is difficult to understand complex systems and their interactions within the enterprise, yet they fail to appreciate that testers must have this knowledge as a prerequisite. The software testing engineer who is unable to fully appreciate what a system is trying to do is in just as laughable a situation, but, sadly, many of us fail to recognize it.

Testing is difficult. It is difficult because it requires a great deal of insight and knowledge as well as experience to perform effectively. Following are essential ingredients to success in any testing project:

a) Creativity and insight are important.
b) Business knowledge is important.
c) Testing experience is important.
d) Testing methodology is important.

Basic creativity and insight probably cannot be taught, and application or business knowledge must be learned on the job or as a broader element of overall career development. The need for all four points is what makes software testing so difficult. Far from anyone being able to do testing, it is rare to find all the ingredients in any single individual.

Testing Principle – 3: An Important Reason for Testing is to Prevent Deficiencies from Occurring
Most of the practitioners do not appreciate the significance of testing in terms of preventing defects instead of just discovering them. As mentioned, the traditional view of testing was as an after-the-fact activity that happened after all coding was completed. Gradually, during the last fifteen years, a very different attitude has emerged: we now look at testing not merely as a step within the development cycle, but view it as a continuous / ongoing activity over the entire development cycle.

The best testing gives rapid feedback. As we complete a segment of work we want to be able to test it as soon as possible. Many have observed that the “process” of thinking out what we are going to test helps us to avoid certain errors or realize that deficiencies exist.

Some believe that the best possible thing we can do to prevent defects is establish the testing first. By establishing what we do to convince ourselves that something is working properly, we increase our understanding and insight about what we are working on.

To truly understand something we must know how to test it, and vice versa. Students who “design” or think out the final exam they would use if they were the teacher often report a major benefit in terms of subject comprehension and aid in preparation for an actual exam. This reinforces the principle that testing plays a most important role in preventing errors or, at the very least, discovering them much earlier. One day we may well consider that the highest goal in testing is not finding defects, but avoiding them.

Testing Principle – 4: Testing is Risk-Based
Important questions that come to our mind; “How much of software testing we would be planning to do if we already knew that risk of software failure or the presence of defects is minimal?”. Another question arises, “How much of software testing we would be planning to do if we knew that such a defect can be as expensive to fix as spending the entire savings of our life or even more important like a threat to our life itself”.

Reflecting on such questions emphasizes that good testing is inherently risk-based. The amount of testing we should (and are willing to) perform depends directly on the risk involved. Programs or systems with high risk require more test cases and more test emphasis. Programs with low risk or limited impact of failure do not warrant the same concern.

Understanding the risk-based nature of software testing is the key to dealing with the chronic problem of inadequate testing resources. As Principle – 1 asserts, we are not able to “completely” test anything. Risk must be used as the basis for allocating the test time that is available and for helping us to make the selection of what to test and where to place our emphasis.

Testing Principle – 5: Testing Must Be Planned
There is no one in the software testing world that doesn�t agree with this principle, The problem is that most of us do not discipline ourselves to act on it. We know that good testing requires thinking out an overall approach, designing tests and establishing expected results for each of the test cases we choose. Since we can’t test everything (Principle -1 described above tells us we are limited to only a very small sample out of the total possibilities), we must make a selection, and the planning and care we expend on that selection accounts for much of the difference between good and poor testers.

A document that defines the overall testing objectives and the testing approach is called a test plan. A document or statement that defines what we have selected to be tested and describes the results expected is called a test design. Test plans and test designs can be developed for any level of testing – requirements and designs, programs, subsystems, and complete systems. A quick difference between “Test Plans” vs. “Test Designs” is given below

Test Plans and Test Designs may be formal or informal. A “formal” plan is one that is written down and made available as a permanent project document. An “informal” plan is one that exists only in someone’s mind or as temporary notes that are not intended to be reviewed or saved. Test plans may be organized as integrated documents covering the entire testing activity or be broken down hierarchically into a series of related documents.

Planning for software testing, whether formally or informally, is necessary. Remember that we have not tested something if we have simply run a few inputs through it. The purpose of software testing is to measure the software quality. Ad hoc or haphazard testing does not provide sufficient information to give reasonable measurements. Such testing may even be harmful in leading us to a false sense of security. It takes careful planning to select a good set of tests. The what-to-test and when-to-stop questions must be answered, and that involves examining the cost versus the confidence that can be gained. We know we have to be selective in deciding what to test and when to stop. Certain groups of tests are “better” than others, and by planning carefully and submitting the plan and design to others for review we are likely to develop a systematic quality measurement.

A second major benefit of formal test plans is that they impose the discipline of working out “expected results” before test cases are run. Teachers who carefully prepare “expected answers” for the questions they are considering produce better exams and measure their students more effectively. Expected results in the test plan provide independent judgments of what systems are supposed to do and help us to spot many errors – often without even running the test.

Planning can help every practitioner to become a better software testing engineer. The implication is not that a lot of time must be invested in planning – just that some time be set aside to focus on what has to be tested and to write that down on paper. If we have not put it on to paper then no one else can see it and help review it, and the opportunity for a permanent record even for ourselves gets lost.

Testing Principle – 6: Testing Requires Independence
When we want an “unbiased” measurement we need an unbiased person to do the measuring. Exhaustive test-planning exercises are must for software testing engineers. Such groups need to work intensively for several days to develop a master test plan for a selected scenario. Group proposals need to be presented, reviewed, and evaluated competitively by each participant.

There are many means of achieving independence. Some organizations have established independent testing or quality control units that report outside of the software development function. Other projects are provided with a “verification and validation” external vendor who oversees the entire effort and is charged with independently verifying system quality. Other approaches can achieve the same goal without requiring different people or added organizational structure. Team approaches and periodic reviews throughout a project serve the same end. The requirement is that an independence of spirit be achieved, not necessarily that a separate individual or group do the testing.

The need for independence is what makes the question of who does testing work so interesting. Principle – 2 emphasizes the importance and need for fully understanding any system and Principle – 6 states the need for independence; the dilemma is that they are often contradictory. The person with the most understanding of the system may be the least independent.

Many More Articles on Software Testing Approaches

Yogindernath Gupta

An expert on R&D, Online Training and Publishing. He is M.Tech. (Honours) and is a part of the STG team since inception.