Reasons of Slow Death of Automated Regression Tests
The biggest problem facing automated regression tests is decay and premature death. You design regression tests to detect problems in features you’ve already tested – problems due to programmers’ mistakes when fixing bugs or adding new features.
Regression tests die over time due to following reasons:
Change of the user interface or output formats: These changes are the major cause of decay. Because of them, a large set of tests that used to pass will fail even though nothing obvious has been done to affect the features that the tests are aimed at. The changes may be so minor that manual regression testers would barely take notice. Automated tests, however, are sensitive and brittle, unable to differentiate between improvements and bugs.
Designed-in assumptions about the test environment:
Test suites may break when moved to different machines or when necessary resources are relocated.
Errors in maintenance: Automation engineers repairing tests make mistakes, introducing bugs into the test suites. Regression test suites thus develop regression bugs themselves.
Changing operators: Test suites may require special skills and knowledge to operate and maintain. When a test automation engineer is reassigned, valuable knowledge can walk away. For instance, person – A disables the disk I/O tests temporarily, due to a particular bug in the product, then leaves to take that programming job in some other place. Person � A’s replacement assumes that the tests are entirely obsolete and moves them to an archive, never to be seen again.
Testers invest in regression automation with the expectation that the investment will pay off in due time. Sadly, they often find that the tests stop working sooner than expected. The tests are out of sync with the product. They demand repair. They’re no longer helping find bugs. Testers reasonably respond by updating the tests. But this can be harder than expected. Many automation engineers have found themselves spending too much time diagnosing test failures and repairing decayed tests.
Exercise intelligence when selecting tests to automate, because anything you automate, you’ll have to maintain or abandon. Uncontrolled maintenance costs are probably the most common problem that automated regression test efforts face.
Ref: Notes from Lessons Learned in Software Testing – By Great Testing Guru – Cem Kaner
Many More Articles on Test Automation