Understanding the different elements of Final System Test Report
During the test execution cycles test reports are generated and distributed.
A final summary report is created after completion of all the test cycles. The structure of the final report is outlined as under
1) Introduction to the test project: The introduction to the test project section of the test report summarizes the purpose of the report according to the test plan. This section includes the following information:
a) Name of the project
b) Name of the software image
c) Revision history of this document
d) Terminology and definitions
e) Testing staff
f) Scope of the document
2) Summary of test results: The summary of test results section summarizes the test results for each test category identified in the test plan. In each test cycle, the test results
are summarized in the form of the numbers of test cases passed, failed, invalid, blocked, and untested and the reasons for not executing some test cases. A test case may not be tested because of several reasons, such as unavailability of equipment and difficulty in creating the test scenario using the available test bed setup. The latter may only be possible at a real beta-testing site. For each test cycle, the following data are given in this section:a) Start and completion dates
b) Number of defects filed
c) Number of defects in different states, such as irreproducible, FAD, closed, shelved, duplicate, postponed, assigned, and open
Problems still remaining in the system, including any shortcoming or likelihood of failure, are summarized in this section in terms of their levels of severity and their potential impact on the customers. For example, if the password is case insensitive, then it should be stated so. As another example, if an operating system puts a limit of 20 on the number of windows that can be simultaneously opened, then it should be stated that the user should not open more than 20 windows simultaneously; exceeding the limit may cause the system to crash.
3) Performance characteristics: In the performance characteristics section of the document, the system response time, throughput, resource utilization, and delay measurement results are described along with the test environments and tools used in the measurement process.
4) Scaling limitations: The limits of the system are stated in the scaling limitation section of the document based on the findings of scalability testing. Once again the facilities and tools that are used in scaling testing are described here.
5) Stability observations: The stability observation section states the following information:
a) Uptime in number of hours of the system
b) Number of crashes observed by different groups, such as software developer, ST, and SIT, in each week of the test cycles
c) Descriptions of symptoms of the defects causing system crash that are in the irreproducible state
6) Interoperability of the system: We state in the interoperability section of the document those third-party software and hardware systems against which the system interoperability tests were conducted. The list may include software and hardware against which the system will not interoperate.
7) Hardware/software compatible matrix: A table shows what software versions are compatible with what hardware revisions in the hardware/software compatible matrix section. Generation of this table is based on the different kinds of hardware systems and different versions of the software used during system test cycles.
8) Compliance requirement status: Finally, in the compliance requirement status section of the document compliance statistics are summarized by generating them from the requirements database. Creating appropriate queries on the database generates the following reports:
a) Requirements that are in compliance, partial compliance, or noncompliance with the software image
b) Requirements that are verified by testing, analysis, demonstration, and inspection
c) Requirements that are covered by each test case of the test suite along with the status of the test case
Ref: Notes from an excellent book – Software Testing & QA: By Sagar Naik & Piyu Tripathy