Importance of Testing the Requirements Documents before any other Software Testing

Importance of Testing the Requirements Documents before any other Software Testing

Requirements Testing Objectives

There is a general tendency to ignore the importance of testing the Requirements documents first. In general requirements documents are the most poorly “tested” work products in the development cycle. Only recently the concept of testing such work products has become popular.

Testing the Requirements documents involves answering two basic questions:

Question-1: Are any requirements missing?

a) Have all needed functions been addressed?
b) Is required performance specified?
c) Is the software quality specified?
d) Is the software fully defined?

Question-2: Can any of the requirements be simplified or eliminated?

a) Should requirements  

be combined?
b) Are any requirements overly restrictive?
c) Are any requirements redundant or contradictory?Answering these two questions effectively depends almost completely on the formal review as the basic methodology. General techniques for achieving effective reviews apply to the testing of any phase. The highest cost/confidence tradeoffs derive from testing requirements thoroughly.

Testing Requirements through an Understanding of What to Test

Obtaining a statement of the requirements for a software project that is as simplified and unrestrictive as possible and yet covers the desired needs has proved to be exceptionally difficult. It is basically a task of “problem definition.” As testers we have to determine whether or not the problem statement in this phase has been properly formulated.

How do we test for correct problem definition? Paradoxically, the answer is by looking at the back end and coming up with what we would test to determine that a proposed solution actually solved the problem. There is a profound advantage to understanding what we are trying to do by discovering how we can convince ourselves that we did it!

Let us consider the following example.

Statement: The mother tells that she has a requirement for you to “eat three well-balanced, nutritious meals a day.” To understand and test the requirement we look at some possible tests mother might use to determine (test for) whether we are fulfilling it.

Three Possible Tests for Well-Balanced Meals

1) Adding up everything we consumed, did we meet minimum nutritional guidelines in terms of calories, proteins, vitamins, and so forth?

2) Did we eat exactly three meals?

3) Were the meals “well-balanced” in the sense that each weighed the same amount?

Test numbers 2 or 3 probably look unwanted because most of us would agree that mother wouldn’t care whether we ate twice or four times or that each meal weighed the same, as long as we got the nutritional content our bodies require.

Notice that attention to how a requirement will be tested helps focus on the real meaning of the requirement itself and enables us to formulate it more clearly. A much better statement of mother’s requirement would be simply that we met or exceeded minimal nutritional guidelines (as specified by an authority or reference) each day. Stating it in this manner makes implicit how it can be tested, and all fuzziness and confusion is removed.

Experience has demonstrated that to achieve a good statement of requirements you must specify the test for accepting a solution along with the statement of the requirement. When the statement and the test are listed together most problems associated with misunderstanding requirements disappear. Just such misunderstandings create most of the errors and problems within software systems.

Testing Requirements through Requirements-Based Test Case Design

Recognizing that requirements become clearer and that faults within them are recognized more readily when we understand how to test them has led to the practice of designing some test cases at requirements time. Such cases are called Requirements Based as they are derived solely from the external specification or requirements for the software. Since information about the software design and data structures is not available, the requirements-based tests designs may be incompletely specified. However, the purpose is to derive requirements-based test situations and use them as a test of requirement understanding and validation.

Let us consider the following examples:

Example 1: Dealing with Incompleteness

When requirements are found to be incomplete, the tester can use a test case to focus the missing information and get it clarified. For example, if the behavior or response to some input is not specified, a test case can be designed with that input and the question asked, “What should the system do in this case?” Such directed questions using specific instances and cases are much more effective than general comments that the requirements are incomplete.

Example 2: Dealing with Imprecision

Similarly, when requirements are fuzzy or imprecise, a test case may be constructed and the question asked, “Is this the result I should have in this situation?” The specific instance will focus attention on the imprecise answer or result and ensure that it is examined carefully.

The STEP testing methodology stresses completion of requirements-based test design before software design and coding is started. This is based on the conviction that taking the time and effort to develop a thorough test from the requirements pays for itself many times over through improved and more validated requirements before the software design and coding activity begins.

The Requirements Validation Matrix

One effective means of organizing all the requirements and ensuring that tests are specified for each of them is through a Requirements Validation Matrix, which is a matrix of requirements versus test cases. A portion of a sample validation matrix is illustrated below.

Organizing Requirements Tests with a Requirements Validation Matrix

The matrix lists each requirement and refers to the test cases or situations that have been created to test it. The test case numbers might refer to an online data set where the test is fully described or may simply refer back to a folder or folio number. The sample matrix tells us that test cases 87, 88, and 102 must be successfully completed in order to fulfill the testing for requirement 1. Note that test cases 87 and 88 are also used for requirements 2 and 4. Contrary to what we might first suspect, we seldom end up with unique test cases for each individual requirement. It usually takes several test cases to test a requirement effectively, and reusing some of those test cases to test other requirements is common. The absence of any test case numbers in the matrix for requirement 3 signifies that suitable tests have not yet been developed or agreed upon.

Benefits of Using a Requirements Validation Matrix

1) Ensures that requirements are listed
2) Identifies the tests associated with each requirement
3) Facilitates review of requirements and the tests
4) Provides an easy mechanism to track status of test case design and review progress
5) Is easily made a part of the master test plan and can be updated throughout the project to give a record of all requirements testing
The use of a Requirements Validation Matrix provides a number of important benefits. The most valuable benefit arises from the discipline of forcing test cases to be designed for each selected requirement and then subjecting both the requirement and the tests to a series of formal reviews (tests). The status columns may be used to indicate the review progress. The use of these columns may easily be adapted to a particular organization or project needs. The first column might indicate that the specification team had completed review, the second that a user management team had, and the third that the test case specification was complete and ready to be used. The Requirements Review Plan would specify how the matrix is to be used and who is responsible for completing it.

Testing Requirements through Prototypes or Models

A second important strategy for testing requirements (determining whether any requirements are missing or whether the needs can be simplified) is through the use of prototypes or test “models.” The technique consists simply of building a model or prototype system not with the intent to use it, but to test and confirm the understanding of the true requirements.
The Benefits of Models

1) A picture is worth a thousand sentences,
2) An example is worth a thousand pictures!

Test models are especially useful when so little is understood about the requirements that it is necessary to gain some “experience” with a working model. Such models recognize that requirements change with experience and that one of the best ways to test for proper understanding of true requirements is to use a system (or at least as much of a skeleton as may practically be provided for test purposes).

The popular practice of incremental development is an extension of the prototype concept. Incremental development simply means that the process of establishing requirements and designing, building, and testing a system is done in pieces or parts. Rather than attempting to solve a “total” problem, one selects a part of the problem and designs, builds, and implements a system to address it. Then, based on the experience with that subpart, one selects another part of the problem, and so on. The advantage is that requirements for later increments do not have to be specified prematurely and can be adjusted on the basis of working experience.

Some practical experience with incremental development suggests that it is best not to specify too many increments in advance. Most organizations try to define (or at least foresee) all of the increments in the initial organization of the effort. They quickly discover how much those later increments change as a result of changing requirements. The most dramatic changes are when the client or customer says, “I like what I have now and do not want any additional increments!” Here the testing of requirements achieved as a result of a working prototype (or stage of incremental development) has been so effective that all additional requirements are dropped and no longer seem to be necessary. Not every organization is quick to recognize such a situation as positive. Project managers may feel that they have “failed” when further work is canceled. Only later might they realize that the “ability to decide when to stop is a feature, not a failure!”

Other Requirements Testing Techniques

Testing for missing requirements is aided by any effort to organize or “structure” the existing list of requirements. Indexing and organizing by function, or by affected data element or system output so that related requirements are grouped together, is always useful, especially as a preparatory step to a formal review. Checklists can also be extremely useful as reminders of commonly overlooked requirement areas or previous omissions. When requirements are grouped together they can be analyzed in a block for simplicity, redundancy, and consistency. With a look at the entire group, one can make an effort to formulate simpler or higher-level requirements that specify the same needs more effectively, and the unnecessary requirements can simply be dropped out.

Automated tools and support aids for requirements testing are not well developed and currently offer little benefit. Experimentation is taking place with specification and requirements definition “languages.” Such languages provide a vehicle for stating requirements in a quasi-structured English and permit some forms of automated analysis. Most of the benefit arises from the discipline imposed in organizing the specification and then listing the test explicitly with each specification. As emphasized earlier, insisting on testability greatly improves the ability to understand and validate the requirements. Other automated aids include indexing programs to group related requirements, paragraph analyzers to flag confusing and overly complex sentences, and decision-table or cause-and-effect graph analyzers.

Anything that helps to make requirements more “testable” improves our ability to simplify or eliminate unnecessary requirements. Specifying requirements in the form of decision tables or as cause-and-effect graphs is an example. The use of a table to show possible conditions and specify intended actions provides an implicit test to determine success. Thus requirements provided in the form of a decision table (or as an equivalent cause-and-effect graph) are inherently testable and can usually be validated quite easily.

There is clearly no simple formula for testing requirements well. Good reviews are the essential ingredients. All the automated aids play only a minor role in actual requirements testing practice.

Many More Articles on Test Planning & Management