James Whittaker recently blogged about his view of the future of test case reuse. He points out that reusable tests (usually automated ones) are heavily dependent on the local machine environment. That is: a reusable test might work on Sally’s computer, but not on Greg’s.

He argues that in the future, this problem will be solved:

In the future we will solve this problem with a concept I call environment-carrying tests (nod to Brent Jensen). Test cases of the future will be written in such a way that they will encapsulate their environment needs within the test case using virtualization. Test cases will be written within virtual capsules that embed all the necessary environmental dependencies so that the test case can run on whatever machine you need it to run on.

I agree, though I think there is an added layer that is important to discuss here. Virtualization will indeed help, but the vision of fixing state to a test case can only really happen when two things are solved:

1. Ease of use: it needs to be very easy for a tester to capture the state of the application under test. In some cases, this is as easy as clicking the “take snapshot” button in VMware. But sometimes applications have very complex state definitions/requirements, such as the time on the clock, external forces (daily stock prices, etc), and multiple machines.
2. Speed: even if it’s easy to associate test cases with virtualized snapshots of the environment, it is currently too slow to be practical to reset a virtualized environment before each test case needs to run. Assume you have 100 automated tests, each one takes an average of 1 minute to run, and it takes 2 minutes to reset state. This means execution time when from less than two hours (1 * 100) to five hours (1*100 + 2 * 100).

Fortunately, the solution to issue #2 (speed) is already available: Amazon EC2. Assuming that the state for each test case has already been stored in Amazon EC2, the speed problem can be solved today as long as you are willing to pay for it. EC2 would allow you to take those 100 tests and run them all in parallel, reducing the overall time test suite time from five hours to three minutes, making it 33X faster than even the traditional way of running the tests in sequence but without virtualization.

In many ways, this is similar to what we’re doing at BrowserMob. Just as we massively parallelize the end user’s browser environment, this technique massively parallelizes the server’s environment. Our approach of using massive amounts of machines to create real end user environments (with real browsers) means that you don’t have to deal with complex scripting designed to compensate for that lack of hardware resources.

Similarly, when we solve the “ease of use” problem (taking snapshots on EC2 is still too hard), writing functional test cases will become that much easier as well. Instead of trying to compensate for that fact that script X might have changed the state for script Y when X and Y are run back-to-back on the same environment, QA professionals will be able to write each test case independently of each other, drastically simplifying them.

In both cases, it dramatically changes what is possible. For us, we’ve redefined how load testing is done, removing the traditional barriers that were only put up due to lack of massive machine resources. Likewise, in the future of testing, we won’t have to wait until the end of the day to run long-running test suites that verify quality. This will have drastic impact on how products are built, as quality metrics will be available within minutes or seconds rather than hours or days.

Tweet This Post 

No tags

We recently published an article on Ajaxian.com titled Why Load Testing Ajax is Hard. This is an excellent overview of the challenges that face testers when working on performance testing for today’s modern web applications. Please check it out and let us know what you think.

Tweet This Post 

No tags

Today we’re pleased to announce that BrowserMob now supports generating load from either the Washington, DC area (US east coast) or from Dublin, Ireland (western Europe). This means BrowserMob customers can now choose where the browsers that participate in their load tests will run from, resulting in an even more accurate end-user simulation. This new capability is possible thanks to recent expansion by the Amazon EC2 service in to Europe.

Selecting which geographic source the load comes from is as simple as selecting a drop-down box when scheduling your test, as seen in this screenshot:

For US-based companies and websites, we recommend continuing to use the Washington, DC source. For companies and websites based in Europe, generating load from Dublin, Ireland is most likely a better option. And for companies that have presence in both regions, BrowserMob is fully capable of running two simultaneous tests from both locations.

We expect to continue to add additional geographic locations as we are able to. If you have any questions about our physical locations, network access, or other infrastructure technology, you’re always more than welcome to contact us.

Tweet This Post 

No tags

Because BrowserMob relies on Selenium to record and play back traffic, we wanted to take a moment to share the latest news in the Selenium world with our BrowserMob customers. On the Selenium blog I just posted the following, which discusses our short-term goals for 2009:

Our two week survey is ending today and the vast majority of you have told us you like the new website. Thank you for the positive support!

I want to take a moment to let you know that we’re not standing still in the Selenium community. We know there are still many problems with the website and with Selenium in general. For example, we are aware of the broken links and poor documentation for all the Selenium projects. We also know that we haven’t put out a release in almost a year and that many of you are resorting to hacks or nightly builds to get things working for you (ie: Firefox 3 support).

Please know we’re working hard to solve all of these. Currently there is a team of volunteers (most of them previously users and not developers) working on the documentation. In parallel the developers are working to get the build for Selenium working on all the supported browsers and operating systems. We’re almost there. When the docs are done and all our tests are working, we will be releasing a 1.0 version for all the major projects (ie: IDE and RC).

If you’re interested in helping with coding, documentation, or general community management (we need moderators!) please let us know in the Selenium Developers forum. Thanks and happy new year!

Patrick & the entire Selenium development team

While it is true that the Selenium documentation is a bit disorganized right now, BrowserMob customers should rest assured that support that is bundled with any BrowserMob purchase covers not only load testing topics, but also any Selenium topic. This means that while the community documentation is evolving, you can rest assured that you still have direct access to Selenium experts.

Selenium 2.0: Just around the corner

Beyond these short term goals of finishing the documentation and getting a 1.0 release out the door, the Selenium team has even bigger goals for 2009. While it has taken several years to get to a 1.0 release (typical in open source), we hope to have a 2.0 release also happen this year.

Selenium 2.0 will focus on the integration of WebDriver, which has already begun. WebDriver is another great open source browser automation technology that is focussing on native browser driver technologies. Selenium, on the other hand, provides a general cross-browser, JavaScript-backed driver that powers many browsers. By merging the two projects together, we hope Selenium 2.0 will be a “best of both worlds”.

What this means for users is that they should see increased stability on the most popular browsers due to new native code hooks. They should also see options to test previously unavailable parts of the browser, such as uploading/downloading files, clicking on security dialog boxes, etc.

There is a lot more we hope to do, but we’re trying to first stay focussed and get 1.0 out the door.

Tweet This Post 

No tags