Is it wise to Automate All Our Testing – How do we decide?
Usually ISTQB certified CTAL experts like “Technical Test Analysts” come across a typical situation where they need to take a trade-off decision in favor of manual testing or automation based upon time, money, and capability.
It is a known fact that not everything can be automated. Some types of testing do not lend themselves to automation, particularly those requiring a human assessment as in usability testing. We have to determine the efficiency of automating. It will likely be a waste of time to automate software that is rapidly changing because the maintenance of the automation scripts will be costly and time consuming.
Automation is an undertaking that must be carefully considered, and a decision should be made on whether to automate fully, partially, or not at all.
Testing experts have laid down many cautions like:
1) Automation should not be treated as a silver bullet to kill the werewolf of spiraling testing
efforts.
2) Automating chaos results in faster chaos.
3) Use a carefully drafted automation checklist before starting the automation effort.The process of deciding what to automate is illustrated in following figure and shows that a checklist can be a useful guide when deciding what to automate.
The automation checklist shown in the figure would typically contain questions like the ones given below:
1) How often is it likely that we will need to execute the test case?
According to the great testing guru � Cem Kaner, a very important “lesson learned” is described relating to the justifications that are sometimes made in favor of test automation based on the number of test executions performed.
Truly speaking, it’s not the number of test executions performed that’s important, it’s the number you actually need. If you would execute a test manually only three times, don’t justify the automation of that test by saying that you’re “saving money” by executing it six times.
2) Are there procedural aspects that cannot easily be automated?
There is a whole range of technical and organizational reasons why full automation will not work or is too costly. There may be manual steps performed within a business process that is only partially supported by your application. There may be aspects of results verification that are more effectively performed by a human. Think of these things before deciding to automate. A partial automation may be the right decision here.
3) Do we have the details?
Automating a test requires telling a tool precisely what to do and what to expect. Do we have that level of detail? If our test cases are described high level (or maybe not at all), we will first need to establish the fine details of those tests before we can automate. How much effort will be required to specify the test case to a sufficient level of detail such that it can be automated? In the opinion of many experts, inadequate attention to these details is one of the main reasons test automation runs into trouble.
4) Do we have an automation concept?
Sounds like an obvious question, doesn’t it? You’d be surprised, though, at how often those conducting automation projects set off to automate certain types of test and then end up trying to automate others. A classic example here is when we start out automating simple GUI tests and end up tackling the automation of complex end-to-end tests.
Do we have a concept for modularizing our automated test cases so that we can chain them together to handle complex business processes? If not, we’d better leave those complex sequences as manual tests, at least for the time being.
5) Should we automate the smoke test (sometimes called a build verification test)?
This test determines if basic functionality is available and working. It is used frequently and provides a high return because it lets us determine if a new release is testable. This is a relatively quick automation effort and results in a very visible win for the usefulness of automation. It’s a good idea to use the successful execution of this script as one of the entry criteria into test.
6) What about regression testing?
These are usually tests that will remain stable but need to be executed frequently. These are just examples of potential automation concepts. I would strongly advise not to contemplate automation until the automation concept has been formed.
7) How much change are we expecting?
If requirements are undefined or unstable, we will also need to change automation scripts quickly and efficiently. If this is not possible, we shouldn’t take on too much automation right up front. It’s probably a good idea to focus automation on the stable parts of the system and leave the rest for manual test execution.
When we’re making a decision for automation, we need to answer a fundamental question:
Why do we want to do this?
a) Do we want to enable a lot of tests to be executed in a short time frame? This might be desirable if we are automating test cases for maintenance testing, where the time available to conduct testing may be severely limited.
b) Do we want to support daily build and test cycles?
c) Are we trying to reduce costs?
d) Do we want to ensure precise test execution of complex sequences?
e) Do we need to exactly reproduce a test?
f) Are we automating because our test process is chaotic?
g) Are there quality characteristics that cannot be tested manually? Performance and load testing frequently fall into this area.
Conclusion:
We should proceed with determining the costs and benefits of test automation tools only after we have considered the questions in the automation checklist carefully and made a decision in favor of full or partial automation.
Full Study Material for ISTQB Test Analyst & Technical Test Analyst Exam
An expert on R&D, Online Training and Publishing. He is M.Tech. (Honours) and is a part of the STG team since inception.
If you want to automate a lot, hire a developer and give him tasks and that’s it.
If it contains “test” in the naming, it doesn’t mean its a testing activity
Thank you for sharing such nice and useful information with us.