Extent of Customer Involvement in Software Testing of a New Software
As a software-testing engineer, you might have done your tiring best by successfully using various means to bring the software application to the current level, and might have helped in getting it ready for its customers. You might have even helped its customers in having their first experiences with the software. But the fact remains that – your job is still not over.
No matter how much testing or how many test phases new software is put through, it can always benefit from exposure to additional environments and users. At the point in the development cycle when the new software has stabilized, a controlled release to some customers certainly provides coverage not easily performed in the development labs. It is assumed that these beta tests are put in place to augment internal testing to see what bugs are reported by the early users.
These beta tests are the fastest way to get new solutions into the hands of the customers most interested in them. They�ll provide a quick feedback, and testers can learn a lot from them. All testers must be deeply involved in many aspects of beta tests in order to investigate problems that escaped your software testing effort and to become more familiar with your
customers. It offers an opportunity for the software testing engineers to work closely with the customers.
The advantage customer gets out of beta is that he gets the product early (and possibly at a lesser price) in return for accepting a certain amount of immaturity and the responsibility for reporting the defects.
What are the objectives of a controlled Beta?
The basic objective of a controlled beta test is to determine that the software is ready to handle the strenuous demands of complex production environments.
We can consider following seven proof points that may be demonstrated by the software.
1) What is the accuracy of the ordering process?
2) What is the ease of the installation process?
3) Whether it has the ability to handle multiple environments?
4) Whether the solutions are validated?
5) Whether sufficient regression testing has been done?
6) Can the software be serviced?
7) Whether any defects have been missed?
How customer acts on getting new software?
Once a customer has their hands on the software, regardless of whether it is a beta or not, it is unlikely they will trust it enough to throw it blindly into production. Nor should they. A customer will first want to spend some time assessing its behavior in their unique environment. Software customers have an explicit role in its testing.
Why customers should test?
Customers detect problems in their production environments. They go to great lengths to avoid them. Yet their environments are growing in complexity as each day passes. Before changes can be introduced they require some amount of validation. A publicly held company has an unwritten obligation to its shareholders to mitigate any risk to the business that such changes might impose. In other words, they need to test. But are these tests a duplication of effort with the internal testing of software suppliers? No, the focuses are different.
The target of customer testing is the integration of all the separate pieces that make up the production environment. The customer can examine the effects of the proposed changes on the total environment, with particular attention paid to their unique components.
The areas under the focus of the customer can be the following.
1) User Exits and Local Customization:
It is not unusual for software to provide various ways for users to alter the default behavior of a subset of its functions. One approach that has existed for many years is the notion of exit points. At very strategically placed spots in the software, the ability to pass control to an exit is provided that optionally allows the customer to alter how the software works. This exit is a program written by the user. The exit may have to follow some strict rules, but it offers users a degree of freedom for customizing the software�s processing to meet their needs. However, whenever the software containing the exit points is upgraded, any exits tied to those exit points need to be tested to ensure they still work without any undesired side effects.
Of course, there�s more to customization than just exit points. Most software offers many other ways for users to change the default behavior. Many knobs and switches may be provided. A large number of adjustments by any customer may create a combination that forces the execution of unique code paths. Experimenting with that combination of adjustments first in a test environment is another step customers can take to reduce the risk of change.
2) Operational Automation:
The speed and volume of processing that current technology allows makes it very difficult to manage and operate within human reaction times. To address this challenge, customers generally automate many operational tasks. Several vendors offer software products that allow a user to specify system events to watch for, and actions to take in response to each event. The automation works in the background, monitoring system activity. When it detects a targeted event, such as a log file filling up, it responds with an appropriate action, eliminating the need for human intervention.
This technique is very effective, but it quickly creates a dependence on correct operation of the automation, since automation mistakes in the production environment can be very disruptive and difficult to work around. As a result, any proposed changes that may effect the automation processing, including adding or upgrading software, should be validated. For example, many automation packages are triggered by messages that the software presents. So software upgrades that might have changed message content or format demand special attention.
3) Correct Operation of System Monitors:
In addition to automation, programmatic monitoring of systems is a major part of production. The monitors become the eyes and ears of the system administrators struggling with massive amounts of processing. Many times, it is a monitor�s alerts that give the system�s human operators their first indication that there is an issue that requires action. As with automation, reliance on the correct operation of monitors is widespread.
Where do the monitors get their information and data? Normally, it is from other components of the system. Some monitors even are dependent on the internal control structures of other software – structures which of course may change with a new release. Customers will desire to demonstrate correct operation of monitors after changes are applied to the system, but before introducing those changes into production. Hence, they will need to test.
4) Importance of Applications:
Although lot of attention is paid to a production environment�s infrastructure, it is the application that provide the most visible value to the business. However, majority of the applications rely heavily on the services of the infrastructure. Also, production environments do not simply include one application, but many. And, it is typical for them to have dependencies on, interface with, or pass data among each other. The combination of all these applications is most likely unique to the enterprise that runs them.
It cannot be assumed that a software supplier or vendor has experimented with this exact combination. Changes to any of these parts should be validated prior to putting them in a position that may hurt the enterprise.
5) Servicing the end users:
Servicing the end users of production environments is almost always the main reason those environments exist. End users may range from the CEO of the company, to the warehouse worker filling orders, to the busy housewife browsing the Web. A business cannot afford to let its end users be let down by changes introduced to production. If the warehouse must delay filling orders because the inventory system is down, or the busy housewife can not place an order and so jumps to a different Web site, it is a direct hit to the bottom line.
Validating the changes in a test environment from the view of an end user is an important step in software testing. One way this can be accomplished is to have the application development and system support communities act as the end users. As a group they can simultaneously invoke the most important or often used areas of the applications. This can be done on a robust test environment if it exists, or during the change window in production prior to when the bulk of the real users start using the system. It is a relatively cheap way to do some functional validation, but may not drive the same amount of activity as will occur during peak times in production. Customers concerned about this can invest in the same load driving, user simulation test tools used by software testing cum development organizations.
Ultimate Challenge – Smooth Production Roll Out
The objective of the software testing by customers is to ensure that the new software works according to their expectations, in their environment, with their complete software pack. It helps ensure a very smooth introduction of those changes into the production environment. Many times it can be a thankless job. If it is done well, no one may notice.
In fact, the goal is to create a nonevent. It may take some time for the advantages provided by new software or other changes to be generally recognized by the user community. But it won�t take long for problems to be noticed.
Advice from the Testing Gurus:
1) Before providing software to external customers, you should run an internal beta under controlled conditions on your company�s own systems.
2) Some software bugs, even ones with devastating consequences, will only surface under specific environmental conditions; hence choose your beta test environments, that offer the greatest exposure to diverse hardware and other software to increase the odds of finding even the minute defects.
Many More Articles in Startup Kit for Software Testing
An expert on R&D, Online Training and Publishing. He is M.Tech. (Honours) and is a part of the STG team since inception.