Understand the Ins and Outs of Exploratory Testing
Term “exploratory testing” is the brainchild of the great testing expert Cem Kaner.
How different experts define “Exploratory Testing”
1) By Cem Kaner: It is an approach to testing which is a sequential examination of requirements, followed by the design and documentation of test cases, followed by the execution of those test cases.
2) By James Bach: It is “simultaneous learning, test design, and test execution.” The tester designs and executes tests while exploring the product.
How Exploratory Testing differs from Ad hoc Testing:
Ad hoc testing generally reflects unfocused, random, and unskilled testing, which can be performed by anyone, irrespective his/her skill level or practical experience. Ad hoc testing remains quite ineffective against majority of defective systems, remains unable to uncover substantial amount of defects.Whereas exploratory testing is comparatively more systematic. Here the tester controls the design of test cases as they are performed much before. In addition, the information the tester gains from executing a set of tests then guides the tester in designing and executing the next set of tests.
Common steps of exploratory testing process
Step-1: Creation of a mental model of the proper functioning of the system
Step-2: Design of one or more tests that would disprove the above mental model
Step-3: Execution of these tests and observing the outcomes
Step-4: Evaluation of the outcomes against the mental model
Step-5: Repeating the process until the mental model is proved or disproved
Alternatively exploratory testing process can involve simple exploration and learning before creating any mental models of the right behavior of the system.
Exploratory testing is generally carried out within an uninterrupted time slot of 1-2 hours exclusively devoted to the testing. This time slot of 1-2 hours is sufficient to perform good amount of testing but does not offer sufficient time to the tester to apply his/her mind elsewhere. The main advantage of having a short time slot of 1-2 hours is that it is quite easy to schedule, easy to control and easy to report as well.
Exploratory testing offers a close feedback loop between test design and test execution. In also provides close feedback between testers and developers as regards the quality of the product being tested is concerned.
Exploratory Testing is useful in following situations
1) When selection of the next test case to be run cannot be done in advance, but it is based on previous tests and their results.
2) When there is an urgent need to provide quick feedback on a product’s quality on a short notice. Or when the requirements are vague or even they don�t exist. Or when the system is quite unstable while being in the initial stages of development.
3) When a defect is detected, but there is a need to explore the size, scope, and variations of that defect for providing better feedback to the developers.
4) When we are not able to detect many errors by the scripted testing process.
Chartered exploratory testing � refined variant of exploratory testing
This type of testing involves the use of a charter which is nothing but a set of intellectually rich guidelines as against the use of scripts. The charter defines a clear mission for the testing session. Due to this approach, exploratory testing fully utilizes the skills of the testers.
Creation of a charter is the first step in doing “chartered exploratory testing”. This charter guides the tester within the defined time slot.
Charter usually defines the following:
1) What is to be tested.
2) What types of documents are available to the tester.
3) What tactics are to be used.
4) What types of defects are to be explored.
5) What types of risks are involved
A charter usually helps in doing the following:
1) Thorough investigation of particular function of the system
2) Defining followed by examining the workflow of the system
3) Identification & verification of all the claims mentioned in the user manual
4) Understanding the performance characteristics of the software
5) Ensuring proper validation of all input fields
6) Forcing all error conditions to do verification of all error messages
7) Checking the design against the standards of user interface
When exploratory testing is done without a charter, it is known as “freestyle exploratory testing.” This involves the maximum possible use of the skills of the testers. In addition the testers simultaneously learn the product as well as design & execute the tests.
Limitations of Exploratory Testing: Exploratory testing suffer from following drawbacks.
1) It has no ability to prevent the defects. Because the design of scripted test cases begins during the requirements gathering and design phases, defects can be identified and corrected earlier.
2) There is no need to do exploratory testing when we are already aware of the tests that need to be executed & in a particular order. In this situation it is better to write down the scripted tests & execute them.
Attributes of good exploratory testers
Exploratory testers are skilled testers. The approach of exploratory testing largely depends upon these skills of such testers.
Few attributes of good exploratory testers are described here. Although these are not hard & fast requirements & anyone without these skills can still perform useful exploratory testing if he/she is properly trained.
1) They are able to create good mental models of the system and its proper behavior.
2) They are careful observers who are able to see, hear, read, and comprehend.
3) They are skilled test designers who have ability to select appropriate test design techniques for different situations.
4) They are able to evaluate the risk, which provides further guidance to their testing.
5) They are critical thinkers who are able to generate different ideas, integrate their observations, skills, and experiences to concurrently explore the product, design the tests, and execute the tests.
6) They are careful reporters who are able to rigorously and effectively report their observations to others.
7) They are self-managed and are able to take the lead in testing rather than execute a plan suggested by someone else.
8) They are mentally strong & don�t get distracted by small issues.
Testing Computer Software – Cem Kaner, Jack Falk, and Hung Q. Nguyen
Exploratory Testing and the Planning Myth – James Bach