The Future of Test Case Reuse (Using Amazon EC2)

James Whittaker recently blogged about his view of the future of test case reuse. He points out that reusable tests (usually automated ones) are heavily dependent on the local machine environment. That is: a reusable test might work on Sally’s computer, but not on Greg’s.

He argues that in the future, this problem will be solved:

In the future we will solve this problem with a concept I call environment-carrying tests (nod to Brent Jensen). Test cases of the future will be written in such a way that they will encapsulate their environment needs within the test case using virtualization. Test cases will be written within virtual capsules that embed all the necessary environmental dependencies so that the test case can run on whatever machine you need it to run on.

I agree, though I think there is an added layer that is important to discuss here. Virtualization will indeed help, but the vision of fixing state to a test case can only really happen when two things are solved:

1. Ease of use: it needs to be very easy for a tester to capture the state of the application under test. In some cases, this is as easy as clicking the “take snapshot” button in VMware. But sometimes applications have very complex state definitions/requirements, such as the time on the clock, external forces (daily stock prices, etc), and multiple machines.
2. Speed: even if it’s easy to associate test cases with virtualized snapshots of the environment, it is currently too slow to be practical to reset a virtualized environment before each test case needs to run. Assume you have 100 automated tests, each one takes an average of 1 minute to run, and it takes 2 minutes to reset state. This means execution time when from less than two hours (1 * 100) to five hours (1*100 + 2 * 100).

Fortunately, the solution to issue #2 (speed) is already available: Amazon EC2. Assuming that the state for each test case has already been stored in Amazon EC2, the speed problem can be solved today as long as you are willing to pay for it. EC2 would allow you to take those 100 tests and run them all in parallel, reducing the overall time test suite time from five hours to three minutes, making it 33X faster than even the traditional way of running the tests in sequence but without virtualization.

In many ways, this is similar to what we’re doing at BrowserMob. Just as we massively parallelize the end user’s browser environment, this technique massively parallelizes the server’s environment. Our approach of using massive amounts of machines to create real end user environments (with real browsers) means that you don’t have to deal with complex scripting designed to compensate for that lack of hardware resources.

Similarly, when we solve the “ease of use” problem (taking snapshots on EC2 is still too hard), writing functional test cases will become that much easier as well. Instead of trying to compensate for that fact that script X might have changed the state for script Y when X and Y are run back-to-back on the same environment, QA professionals will be able to write each test case independently of each other, drastically simplifying them.

In both cases, it dramatically changes what is possible. For us, we’ve redefined how load testing is done, removing the traditional barriers that were only put up due to lack of massive machine resources. Likewise, in the future of testing, we won’t have to wait until the end of the day to run long-running test suites that verify quality. This will have drastic impact on how products are built, as quality metrics will be available within minutes or seconds rather than hours or days.