All About System Testing an Important Part of Our Software Testing Effort
The objective of system testing is to find out the defects in features of the system in comparison with the way it has been defined in the software system requirements (SRS). In principle in system testing we test to get information about the fulfillment of both the functional & the nonfunctional requirements. The functional requirements describe what the system shall be able to perform the functionality. The non-functional requirements describe how the functionality presents itself & behaves.
The execution of system test follows the completion of the entire component integration testing. Meaning thereby – the test object is the fully integrated system.
The better the component testing & the component integration testing performed prior to the system testing, the more effective is the system testing. Generally the hindrances in the process of system testing are poor or missing component & component integration testing.
Process of Conducting System Testing
System testing is carried out by the software testing engineers against both functional & nonfunctional requirements.
In some situations, several quality attributes like configuration, performance, security, stress, reliability, recovery, & usability are to be tested. However, since not all types of systems undergo such thorough testing, this article covers 3-categories of the most frequently conducted system testing:
1) Functional Testing:
The objective of system functional testing is to verify the application functional requirements at the highest level of abstraction. System functional tests generally correspond directly to the application use cases, & are designed as early as use cases are created.
2) Performance & Load Stress Testing:
The objective of the performance & load stress testing is to verify whether the system continues to function correctly under nominal & extreme loads, & whether the response time under those loads is satisfactory. The load is the amount of work performed by a system, & in testing it is expressed as a series of inputs to execute a group of transactions.
Performance testing focuses on fine tuning system response time & throughput under its nominal loads. Load stress testing, on the other hand, concentrates on creating extreme conditions that would break the system due to the resource exhaustion.
3) Security Testing:
Security testing involves designing & executing tests to expose possible application vulnerabilities to intentional & unintentional system security violations. The focus is on those segments of code that are responsible for protecting access to the system & where sensitive data is stored.
Why should we do the system testing?
System testing is necessary to verify both the functional & quality requirements of the application before it undergoes acceptance testing &, more importantly, before it is deployed. Since system testing requires a large amount of resources, this might be the first time that all system components are tested together & on various target platforms. Problems like resource limitations, data overflows, & other external constraints might not come to the notice of software testing engineers until this phase, & hence they should be a focal point of system tests.
Who should do the system testing?
Software testing engineers remain the preferred choice to perform the system testing.
How do conduct the system testing?
System functional testing takes place after successful integration testing has been completed & all the application modules & components have been connected. System functional tests are black box tests corresponding to the application use cases.
For instance, if an online library catalog must be able to add new users, login existing users using user name & password, & perform various types of searches, all these features should be exercised during system testing. All functional requirements should be verified for both valid & invalid inputs. Black box testing techniques like equivalence class partitioning & boundary value analysis should be used to narrow down the number of required tests.
To properly conduct performance testing, its requirements have to be articulated explicitly & they should be quantifiable. For instance, the requirements could state: �The system should be able to handle at least 15 transactions per second,� or �An average response time of a search shall be less than 2 seconds.� In order to verify these requirements, performance is essentially measured under nominal system conditions.
Good software testing engineer tries to probe the answers of following 5 questions to define the nominal conditions of the system.
Q.1: How many users should be able to access the application simultaneously?
Q.2: How much data is expected to be processed?
Q.3: Will the load be distributed among multiple systems?
Q.4: How much will the data grow?
Q.5: Is the application able to cope up with short periods of over activity?
In stress testing, the loads should be extreme (e.g., 2 times larger than under nominal conditions). Performance & stress testing is typically performed on the same staging system as is the integration testing. However, in some cases like with QA, it might not be possible to construct a realistic staging system in order to perform load & performance testing. In such cases, performance & load stress testing need to be conducted on the live application.
Security tests should cover equivalence class domains for valid & invalid password inputs, test for known encryption weaknesses, & check for trap doors & Trojan horses. For authorized access, security tests should also verify the levels of authorization for protected data. For instance, in a hospital patient record system, a staff member responsible for making appointments should be able to take out the patient�s visit history but not medical records. Additionally, security tests should look for ungraceful system degradations due to resource constraints, like a limited number of memory or network connections. Such ungraceful degradation might provide an application entry to unwanted intruders or hackers.
If a failure is observed during system testing, it becomes the first & foremost duty of the software testing engineers to pay an immediate attention to it & assess the severity of the problem.
What should be measured during system testing?
Software testing engineers essentially calculate / measure the following:
1) Calculation of total number of system functional tests & the number of tests that passed & failed per use case.
2) Calculation of the ratio of the passed tests to the total number of tests. With the exception of noise, which reflects false positive results due to, for instance, obsolete data in a tested record, the system functional testing pass rate should approach 100%, & this value is essentially used as a test progression criterion to move to acceptance testing.
3) Measurement of average response time & throughput under system nominal conditions. Stress test results essentially provide information about the maximum loads that the system can handle & continue to function properly.
4) Measurement of the system throughput under stress loads. For a new system, the load & performance test criteria can be passed after a sufficient number & range of tests that exercise realistic usage for the entire application have been developed & executed producing acceptable response times, without negatively affecting system functionality.
5) Measurement of the security test pass rate & unless its value is 100%, testing should not move to the acceptance level.
How best the tracking of performance tests is done?
In addition to tracking the pass test rates during system testing, tracking of performance & load stress testing should be started as soon as the corresponding modules are completed.
If the results of these tests are collected over an extended time & compared against a baseline, undesired performance drops can be easily detected & eliminated as soon as they are introduced. For instance, before adding the security layer to a module, performance testing should be conducted & the results should be saved as a baseline.
After adding the security layer, performance testing should be repeated, & the results compared with the baseline to see whether the added functionality degrades the performance. If the performance degradation is unacceptable, the security layer may need to be modified.
How should we automate the performance tests?
System functional & security tests can be automatically generated using automation techniques on white box, black box, & integration testing.
Performance & load stress testing can be automated by using load generators. Load generators simulate system interactions by using patterns similar to realistic use cases. They continuously access the application & automatically record system response as a function of a tested load.
Many tools support system testing. Capture/replay tools & test management tools are quite useful in supporting the system testing.
1) A comprehensive system test plan & system test specification must be prepared beforehand.
2) The functional testing techniques should be supplemented with experience-based techniques like exploratory testing etc. depending on the completion criteria defined in the plan. However experience-based software testing techniques should never be the only techniques used in the system testing.
3) It is better to perform a static test on the requirements specification & on the system test specification before execution starts.
4) Collect all measures of time spent on the testing, on faults found & corrected, & on coverage.
5) Stop the system testing when the completion criteria specified in the plan have been met.
6) A system test report must be produced on the completion of the system testing.