End To End Testing
What is End To End (E2E) Testing?
Many subsystems are tested in isolation. This is fine for many scenarios that are local to that subsystem, but there are some scenarios that cross the boundary of several subsystems. This is good because a lot of testing can be done in isolation but at some point a complete end to end test needs to be carried out on the whole system.
Problems and Issues:
Unfortunately a lot of testing of complex systems is carried out manually because the level of effort to automate the E2E test is considered to be too expensive. This is primarily due to the fact that testers take a user interface centric approach to testing and do not have the necessary development skills to produce the tools needed to automate testing.
Often they rely on tools such as QTP / RFT / WinRunner to drive the user interface.
Manual testing is unfortunate because as we all know systems are not static, they evolve. Sometimes this evolution is minor and does not impact other systems, or so we think. At other times the impact is greater and the E2E tests need to be rerun. At this point we take the hit re-testing system with the inevitable delay to delivery. But what if a test fails? Do we restart and rerun all our tests after the fix is made. In most cases this is unlikely. The failed test is simply rerun and the hope is there are no side affects.
This haphazard way of testing may be fine when the risk to life or loss of money is low, but it becomes increasingly difficult to justify when the risks are high. A better approach is to automate end to end testing so quality is guaranteed.
Given that more business are trying to become agile in order to respond quickly to changing business needs the need for automated E2E becomes stronger, for without it testing cycles can significantly delay the introduction of new functionality. I’d go as far as saying you won’t be truly agile until you’ve automated E2E testing.
The objective of E2E is to look at the state of the system as well as its behavior. A realization needs to be made that the user interface is not what we are primarily interested in as it simply provides a representation of the underlying data and displays some of its behavior. In E2E we should be more interested in the underlying data and scenarios across all subsystems.
Clearly fully automated E2E testing can be very time consuming and to get the most value a 80:20 perspective is useful, Looking at the scenarios of a typical system only about 20% of the functionality is used on a regular basis. So the initial objective should be to automatically test the 20% first and gradually adding the lesser used functionality.
Given that we need to look at the data and the system behavior and not just the user interface a different set of skills are needed to automate E2E as specialized tools will almost certainly be required to drive the system as well as possible instrumentation of a subsystem to extract the data necessary.
In addition to development skills analytical skills are necessary to determine the 20% of functionality that is really worth testing.
Finally as with all projects, good project management is mandatory. Thus an agile approach is most successful, that is, short value driven iterations delivering useful tests as quickly as possible to get feedback as early as possible on the effectiveness and potential cost of automating E2E.