Importance of doing a Pilot Project before Full Scale Automation Tool Roll Out
Before delving on the intricacies of going into a pilot project, I would like to describe few common pitfalls involved in the large-scale implementation of automation tools.
1) “Ease of use” of course being one of the key consideration in selecting an automation tool, can never overlook the prime requirement of good functionality. A tool can easily be misused while it may fail to do few simple things.
2) In the absence of a structured assignment of roles and responsibilities towards centralized methodology of using the tool, the users tend to get absorbed In several standardization attempts that land up evolving equal number of confusing approaches. Different users of the tool try to use it in different, ways as per their own convenience leading to collapse of the standardization efforts.
3) True value of a test automation tool can be better understood by carrying out a small-scale pilot project before a mass roll out.
Due to this reason, in small sized organizations, the test managers lay greater stress on the pilot compared to selling their concept. They believe in avoiding doing anything new on a large scale.
Why go in for a pilot evaluation project?
The objective of pilot project is to provide a safer opportunity to the organization to test the new automation tool, demonstrate its true value and discover deficiencies if any, well before spending significant amount of money, time and other resources on a full-scale project. Working on a small-scale project would help in ironing out the problems coming across small group of users. It is helpful in understanding the effect of the tool on the way we would be doing our testing. With this the testing managers will get good idea on revising the existing standards and procedure�s to get best out of the new testing toolGenerally, a pilot project is kick started by preparing a business case describing the objectives of the project and outlining the methodology how the project shall be carried out. A realistic time plan along with metrics for the determination of the success is an essential part of the business case. For instance, the testing engineer may like to reduce the time to run the regression tests from a week to a day. Actually, applying the ‘don’t be overly optimistic’ rule, it may be better to set a target such as reducing the time for 20% of the tests by a factor of 50%. This may result in a five-day regression test taking four and a half days, but might be a much easier target to hit.
The pilot project must not be either too short or too long, may be from 2 to 3 months. Subsequent phases of the pilot project could extend this time beyond 3 months but each phase should have measurable objectives. If the pilot stretches for a longer period without significant results, it would cast a shadow of doubts on the viability of the overall test automation. It is better to accrue smaller benefits in the beginning may be in bits and pieces, being less risky rather than gaining much larger benefits that are projected later.
Assessment of changes in the testing processes:
It is quite possible that the new testing tool could lead to unexpected changes in your testing procedures that may not be desirable in the longer run. This effect can be better understood through the following analogy of buying a dishwashing machine aimed to offer you some relief from otherwise time consuming and strenuous manual activity.
The activity began with the consideration of an alternative option of hiring a domestic help and subsequently you plan to go in for a dishwashing machine. You consolidated your purchase decision after evaluating various essential & desirable features of many products from different brands. Finally after due deliberations the equipment that best appeared to be meeting the maximum requirements was installed.
After around four weeks, you realize that the new equipment had drastically changed the entire routine of your household. You amazingly realized that despite considerable reduction in the time of washing of dishes from 45 minutes to 15 minutes after every meal; you had to spend more than 10 minutes of time in unloading of the equipment every morning. Whereas earlier the dishes were being put aside simultaneously while washing them & this had become an additional activity now.
At the same time you had been losing the valuable social time that you earlier spent chatting with your spouse, while washing the dishes manually. Now you realized that due to the new sophisticated equipment, you had lost something more precious than a saving of little time.
In conclusion, the new equipment although packed with lots of desirable features, unexpectedly changed the daily domestic routine that you never relished.
The deployment of a new software testing tool too can have similar unexpected results that you may seem to be a liability. For instance, with the help of a test execution tool, debugging may appear cumbersome compared to the ease of the manual process earlier. Whereas when you use the automation tool; you come to know of the problem later. And you need to spend time for generating the context of the bug before finding it. Meaning thereby, it is an additional activity that was not there earlier.
You may realize that while using the tool, something of significant value has been lost. For instance while manually executing the tests, the software testing engineers generally keep on thinking of many other things for testing while they were actually supposed to be after a script. Earlier they could get little diversion from the script while simultaneously creating notes for some new tests, which can be added. Such a mental inspiration for writing new tests generates more tests than even the planned numbers. However with the automation tool, you can�t be assured of any diversion from the planned script and you cant expect any new tests.
Though testing tools are there to make the life of testing engineers easy, yet they need to ensure that many such adverse effects of the tool, whether anticipated or not are not outweigh the perceived benefits of implementing the automation tool.
Trial of the automated setup:
With the pilot project on a small-scale, the software testing engineers can carryout the experimentation to understand the process of building automated Test Suites, which shall be expected to be comfortable to maintain in the real situations. The main objective of the pilot is to verify the functionality of the script structure, naming conventions, data organization etc. prior to rolling them out among larger group of the users of the tool.
Pilot result evaluation:
Alter the satisfactory completion of the pilot project its results are compared with the requirements of the business case. If the findings are satisfactory on a smaller scale, the tool can be believed to be acceptable & can be scaled up safely. The experience gained out of the completion of the pilot project is greatly helpful in churning out much more benefits out of the next project.
In case due to any reason whatsoever, the planned objectives are unable to be achieved, it can be concluded that;
1) Either the tool is not appropriate for the application, or
2) Tool is not being deployed properly, or
3) Expectations out of the tool were too optimistic
Before discarding the tool altogether, why-why analysis must be done to understand the reasons of not getting success out of the tool and to make a comprehensive plan to decide the further course of action.
No attempt should be made to deploy the tool on a larger scale unless you are able to justify the reasons of failure of the tool on a smaller scale. Before scaling up the deployment, the testing managers need to move extremely cautiously and if need be the tool usage can be increased slowly, may be in the increments of one project group at a time.
During the later stages of the pilot project, the benefits of the project can be monitored by the testing managers by using metrics like Maintainability, Efficiency, Reliability, Flexibility, Usability, Robustness, Portability etc.
If test creation time seems to be a problem in early attempts to automate, the pilot can try to shorten this time, while keeping the test maintenance costs or time under control.
Planned roll-out in phases:
If the pilot project happens to be successful, the tool is deployed in the remaining part of the organization after a careful planning. The plans must be flexible enough to react to the problems encountered..
Activities involved in the rollout process are;
1) Give a wide publicity to the success of the pilot project;
2) Company policies and strategies may call for modifications to include test automation as a part & parcel;
3) Ensure that the project managers take due cognizance of test automation issues in their project plans, quality plans, and test plans;
4) Go on building sound infrastructure for the test regime;
5) Formulate a company wide goal to make constant use of the tool than to do manual testing;
6) Involve maximum groups in test automation;
7) Continuously monitor the efficiency of test automation;
8) Provide training to the direct and indirect users of the test automation;
The change management team can function as internal consultants for the users of the new tool within the organization for drawing greater benefits.
Tips for the success of the pilot project:
1) Ensure the active involvement of the developers with sound coding experience;
2) Management must have realistic expectations from the project teams;
3) Ensure effective communication between the testing managers and the management;
4) Testing team must set up the project with full optimism of getting success in the one go;