Am I doing Too Little or Too Much Automation

Download Link for your Favorite E-Book is at the End of this Page
***********************************************************************************
Am I doing Too Little or Too Much Automation.

We shall be able to count a large number of possible test cases for any system although fact may be that we might be having too little a time to execute only a few of them. Even if this number may be small, still we would expect to detect majority of our bugs out of these whatever small number of test cases.

Hence identification of test cases to be created & to be executed is an important activity. Random selection of our test cases is not a good approach for testing. Hence for the development of good test cases, a carefully designed approach is necessary

Am I trying to do too much of Automation:
Testers usually commit this common mistake of doing too much of automation & that also too soon. Intuitively, testers would be tempted to feel that having more number of automated

tests better it would be. But this would be wrong. This belief would leave us with plenty of poorly automated tests, which shall become quite difficult & uneconomical to maintain.

If we go on adding more & more automated tests we shall land up with unwanted duplication, redundancy & increased cost of their maintenance. Hence it is better to start with small numbers of good but diversified tests and automate them. Here it is ideal to begin with say 10-15 tests involving 2-3 hours of interactive testing.

Am I automating the Wrong Tests:
It is not wise to attempt to automate every test case just because the benefit of automating some of the tests is outweighing the cost of automating them. In fact some of the test cases cannot be automated however if some dedicated tester pulls up his strings, can get success in automating them at a very high cost but with no tangible benefit.

However after gaining some reasonable experience of automation testing one can predict with reasonable accuracy the time it will take to automate a particular test. The decision as to which of the test cases to be automated and which of these to be automated first, is quite crucial & is based upon the likely pay back.
Attributes of test case, which can be likely candidate for automation, are as under:

• Any test which can be expected to run several times. It is quite natural that more the times a test case is made to usefully run the more beneficial an automated version of it will be. Regression tests, which are made to run every time a new version of software is created, are best suitable for automation.
• Input verification tests, for example checking that an edit box accepts values in a valid range only. Such tests are quite boring by nature & prone to error for manual operation.
• Tests, which are expensive to execute manually, for example multi-user tests, which take a long, time to run.
• Tests, which are difficult to execute manually, for example test cases which are timing critical or are complex to execute.
• Tests, which require persons having special knowledge, like business knowledge or system�s knowledge can be good candidates for automation.

Attributes of test case, which are unlikely to be candidate for automation, are as under:

• Any test which will not be expected to run many times. If a test case is not run many times just because there had been no need to run it often (rather than because it could be costly to run it often manually) then it should not be automated.
• Test cases, which are not important, will definitely not find important bugs. Naturally, If a bug is not important it is not wise to invest time & money in finding it, especially when such bugs are not going to be fixed.
• Usability test, which cannot be automated because usability, is an issue related to human interactions.
• Any test which is difficult to automate. Test cases, which would take a lot of effort to automate, are generally not worth automating.
• Any test which is expensive to maintain. Even if a test case is quite simple to automate but if it is vulnerable to changes in the software it would not be worth automating, because it would be very costly to maintain it.
• A test case which might have a lot of value in itself, but if it duplicates some or all of the value of an existing test case then the value of the new one gets reduced to almost zero. Such test cases are also not worth automating.

Best Lessons learnt:

1) Wherever complete automation is not desirable, we can think of partial automation in our project.

For example, we may find that automation of execution of a particular complex test case is problematic, however it is feasible & viable to automate the relative comparison of few of the test case results with the expected ones. Alternatively, when due to some odd reasons, may be due to some technical ones, it is not possible to automate the execution of a particular test case, we can still draw many beneficial results by automating few parts of the subject test case, for example, the preparation of data and clear-up etc.

2) There is a great learning curve in the beginning, hence it is preferred not to automate so many test cases to start with because these may not be as good & fit for automation, as compared to the ones we could automate at a later stage.

3) It is preferred to focus our attention on comparatively few numbers of tests, by exploring strengths and weaknesses of a smaller group of tests under automation. We can then extend our exploration to automation of a larger numbers of tests at a later date after gaining experience from the smaller group set.

Remember that many case histories have made startling revelation, that over 70% of bugs are found by manual testing & not by automated testing in-spite of the fact that the automated tests had been designed & developed with great efforts of many years. Hence think well before automating your tests.

Ref.: Article Published with the persmission of its author – Expert QTP

Many More Articles on Test Automation

DownLoad Link To Your Favorite EBook:

Easy Steps to Learn IBM Rational Functional Tester (1357 Kb)