situations, several quality attributes like configuration, performance,
security, stress, reliability, recovery, & usability are to be tested.
However, since not all types of systems undergo such thorough testing, this
article covers 3-categories of the most frequently conducted system testing:
1) Functional Testing:
objective of system functional testing is to verify the application functional
requirements at the highest level of abstraction. System functional tests
generally correspond directly to the application use cases, & are designed
as early as use cases are created.
2) Performance & Load Stress Testing:
The objective of the performance & load stress testing
is to verify whether the system continues to function correctly under nominal
& extreme loads, & whether the response time under those loads is
satisfactory. The load is the amount of work performed by a system, & in
testing it is expressed as a series of inputs to execute a group of
Performance testing focuses on fine tuning system response time &
throughput under its nominal loads. Load stress testing, on the other hand,
concentrates on creating extreme conditions that would break the system due to
the resource exhaustion.
3) Security Testing:
testing involves designing & executing tests to expose possible application
vulnerabilities to intentional & unintentional system security violations.
The focus is on those segments of code that are responsible for protecting
access to the system & where sensitive data is stored.
Why should we do the system testing?
System testing is necessary to verify both the functional
& quality requirements of the application before it undergoes acceptance
testing &, more importantly, before it is deployed. Since system testing
requires a large amount of resources, this might be the first time that all
system components are tested together & on various target platforms.
Problems like resource limitations, data overflows, & other external
constraints might not come to the notice of software testing engineers until
this phase, & hence they should be a focal point of system tests.
Who should do the system testing?
Software testing engineers remain the preferred choice to
perform the system testing.
How do conduct the system testing?
System functional testing takes place after successful
integration testing has been completed & all the application modules &
components have been connected. System functional tests are black box tests
corresponding to the application use cases.
For instance, if an online library catalog must be able to add new users,
login existing users using user name & password, & perform various types
of searches, all these features should be exercised during system testing. All
functional requirements should be verified for both valid & invalid inputs.
Black box testing techniques like equivalence class partitioning & boundary
value analysis should be used to narrow down the number of required tests.
To properly conduct performance testing, its requirements have to be
articulated explicitly & they should be quantifiable. For instance, the
requirements could state: “The system should be able to handle at least 15
transactions per second,” or “An average response time of a search shall be less
than 2 seconds.” In order to verify these requirements, performance is
essentially measured under nominal system conditions.
Good software testing engineer tries to probe the answers of
following 5 questions to define the nominal conditions of the
Q.1: How many users should be able to access the application
Q.2: How much data is expected to be processed?
Q.3: Will the load be distributed among multiple
Q.4: How much will the data grow?
Q.5: Is the application able to cope up with short periods
of over activity?
In stress testing, the loads should be extreme (e.g., 2 times larger than
under nominal conditions). Performance & stress testing is typically
performed on the same staging system as is the integration testing. However, in
some cases like with QA, it might not be possible to construct a realistic
staging system in order to perform load & performance testing. In such
cases, performance & load stress testing need to be conducted on the live
Security tests should cover equivalence class domains for valid & invalid
password inputs, test for known encryption weaknesses, & check for trap
doors & Trojan horses. For authorized access, security tests should also
verify the levels of authorization for protected data. For instance, in a
hospital patient record system, a staff member responsible for making
appointments should be able to take out the patient’s visit history but not
medical records. Additionally, security tests should look for ungraceful system
degradations due to resource constraints, like a limited number of memory or
network connections. Such ungraceful degradation might provide an application
entry to unwanted intruders or hackers.
If a failure is observed during system testing, it becomes the first &
foremost duty of the software testing engineers to pay an immediate attention to
it & assess the severity of the problem.
What should be measured during system
Software testing engineers essentially calculate /
measure the following:
1) Calculation of total number of system functional tests & the number of
tests that passed & failed per use case.
2) Calculation of the ratio of the passed tests to the total number of tests.
With the exception of noise, which reflects false positive results due to, for
instance, obsolete data in a tested record, the system functional testing pass
rate should approach 100%, & this value is essentially used as a test
progression criterion to move to acceptance testing.
3) Measurement of average response time & throughput under system nominal
conditions. Stress test results essentially provide information about the
maximum loads that the system can handle & continue to function properly.
4) Measurement of the system throughput under stress loads. For a new system,
the load & performance test criteria can be passed after a sufficient number
& range of tests that exercise realistic usage for the entire application
have been developed & executed producing acceptable response times, without
negatively affecting system functionality.
5) Measurement of the security test pass rate & unless its value is 100%,
testing should not move to the acceptance level.
How best the tracking of performance tests is
In addition to tracking the pass test rates during
system testing, tracking of performance & load stress testing should be
started as soon as the corresponding modules are completed.
If the results of these tests are collected over an extended time &
compared against a baseline, undesired performance drops can be easily detected
& eliminated as soon as they are introduced. For instance, before adding the
security layer to a module, performance testing should be conducted & the
results should be saved as a baseline.
After adding the security layer, performance testing should be repeated,
& the results compared with the baseline to see whether the added
functionality degrades the performance. If the performance degradation is
unacceptable, the security layer may need to be modified.
How should we automate the performance
System functional & security tests can be
automatically generated using automation techniques on white box, black box,
& integration testing.
Performance & load stress testing can be automated by using load
generators. Load generators simulate system interactions by using patterns
similar to realistic use cases. They continuously access the application &
automatically record system response as a function of a tested load.
Many tools support system testing. Capture/replay tools & test management
tools are quite useful in supporting the system testing.
1) A comprehensive
system test plan & system test specification must be prepared
2) The functional testing techniques should be supplemented with
experience-based techniques like exploratory testing etc. depending on the
completion criteria defined in the plan. However experience-based software
testing techniques should never be the only techniques used in the system
3) It is better to perform a static test on the requirements specification
& on the system test specification before execution starts.
4) Collect all measures of time spent on the testing, on faults found &
corrected, & on coverage.
5) Stop the system testing when the completion criteria specified in the plan
have been met.
6) A system test report must be produced on the completion of the system