are summarized in the form of the numbers of test cases passed, failed, invalid, blocked, and untested and the reasons for not executing some test cases. A test case may not be tested because of several reasons, such as unavailability of equipment and difficulty in creating the test scenario using the available test bed setup. The latter may only be possible at a real beta-testing site. For each test cycle, the following data are given in this section:
a) Start and completion dates
b) Number of defects filed
c) Number of defects in different states, such as irreproducible, FAD, closed, shelved, duplicate, postponed, assigned, and open
Problems still remaining in the system, including any shortcoming or likelihood of failure, are summarized in this section in terms of their levels of severity and their potential impact on the customers. For example, if the password is case insensitive, then it should be stated so. As another example, if an operating system puts a limit of 20 on the number of windows that can be simultaneously opened, then it should be stated that the user should not open more than 20 windows simultaneously; exceeding the limit may cause the system to crash.
3) Performance characteristics: In the performance characteristics section of the document, the system response time, throughput, resource utilization, and delay measurement results are described along with the test environments and tools used in the measurement process.
4) Scaling limitations: The limits of the system are stated in the scaling limitation section of the document based on the findings of scalability testing. Once again the facilities and tools that are used in scaling testing are described here.
5) Stability observations: The stability observation section states the following information:
a) Uptime in number of hours of the system
b) Number of crashes observed by different groups, such as software developer, ST, and SIT, in each week of the test cycles
c) Descriptions of symptoms of the defects causing system crash that are in the irreproducible state
6) Interoperability of the system: We state in the interoperability section of the document those third-party software and hardware systems against which the system interoperability tests were conducted. The list may include software and hardware against which the system will not interoperate.
7) Hardware/software compatible matrix: A table shows what software versions are compatible with what hardware revisions in the hardware/software compatible matrix section. Generation of this table is based on the different kinds of hardware systems and different versions of the software used during system test cycles.
8) Compliance requirement status: Finally, in the compliance requirement status section of the document compliance statistics are summarized by generating them from the requirements database. Creating appropriate queries on the database generates the following reports:
a) Requirements that are in compliance, partial compliance, or noncompliance with the software image
b) Requirements that are verified by testing, analysis, demonstration, and inspection
c) Requirements that are covered by each test case of the test suite along with the status of the test case
Ref: Notes from an excellent book - Software Testing & QA: By Sagar Naik & Piyu Tripathy