Because both our load testing and website monitoring services are based on Selenium, we have a unique ability to measure the performance of things like page load times, AJAX timings, and other in-browser interactions.

Selenium has both a setTimeout command and a waitForPageToLoad command. Both can be given a timeout value, which will control how long Selenium waits for a given page to load or element to appear. When it comes to using our services, most people stick with the default time of 30 seconds. If the timeout is reached, an error is thrown, the script aborts, and the transaction is recorded.

However, sometimes people want to know when pages take more than X seconds to load, but don’t want to necessarily interrupt the flow of the script. In fact, just last week we got this request from a customer:

Ability to trigger an alert based on a set threshold (by the user) – not using the timeout. This basically came from the performance issues we are experiencing. Here’s the scenario:

• We have a specific page that takes 2+ minutes to load.
• The page load timeout in the Selenium script was set to 60s.
• BMob properly reported the “timeout” error but when this error happens, BMob quits the script.
• This is not ideal for me since I want to still be able to see how long the page takes to load.
• Increasing the page load timeout for the page in question works, but now I don’t have a way (that I know of) to still trigger an alert after 60s.

I should be able to set a threshold for any page I choose that would then send a notification alert.

In other words, this customer wanted for a way to still report a transaction as a failure and receive an alert, but also still allow the script to continue. Fortunately, our support for JavaScript as a scripting language provides the answer:

var timeout = 90000;
...
// start of script
...
var start = new Date().getTime();
selenium.waitForPageToLoad(timeout);
var end = new Date().getTime();
...
// rest of script
...
if ((end - start) > 45000) {
    throw "An important page took longer than 45 seconds to load";
}

What this script does is sets the timeout to a very long amount (1.5 minutes) but will still report an error if a specific page takes longer than 45 seconds. This allows the remainder of the script to execute even when the page takes more than a specified 45-second threshold.

The only problem with this script is that if the page takes longer than 90 seconds, then the rest of the script will still not run because waitForPageToLoad will throw an exception. You can solve that too with a little code:

var start = new Date().getTime();
try {
    selenium.waitForPageToLoad(timeout);
} catch (e) {
    // this will happen after 90 seconds
    // todo: recover and send the browser to the the next URL
}
var end = new Date().getTime();

The only thing that is important to remember with this use of try/catch is that you’ll need to properly recover from the error. Simply catching the error and trying to continue may not work. For example, if the next Selenium command requires clicking on a button that should have loaded from the last page, there may be no way to recover. However, if the next step is simply visiting a new URL, you could possibly get away with a simple open() command in the catch block.

Tweet This Post 

Scripting · Selenium · Timeouts

dynaTrace just posted a really nice tutorial showing how to track client-side performance (JavaScript, DOM, render times, etc) automatically using Selenium. They do this by using their super-cool free AJAX edition of their product.

Tweet This Post 

dynaTrace · Selenium

We’re a few weeks late to the story, but we think it’s still important to call attention to: Watir and Selenium are joining forces. For those unfamiliar with Watir or Selenium, both of them are open source browser automation frameworks. For years they have been “competing” in the open source community, with Watir winning favor among the Ruby community and those who needed strong IE support, while Selenium won favor among the Java and C# crowds and those who really valued cross-browser support.

While both projects had their pluses and minuses, it’s great to see them finally working together. On behalf of BrowserMob (which builds on top of Selenium) and as a Selenium contributor, it pleases me to know end to know that the Selenium community will now gain two huge contributions: a first-class Ruby API and all the benefits of Watir’s fantastic IE support.

Note: this is not a merger. Watir will continue to run as an independent project and it will even still work with it’s own browser backend. The important thing here is that Watir will now offer an option to wrap around Selenium WebDriver, the core cross-browser automation library that all the other Selenium projects also wrap around. This means that while the projects will remain independent there will likely be a lot more cooperation moving forward.

Tweet This Post 

Selenium · Watir · WebDriver

We’re happy to report that Selenium IDE 1.0.4 has been released. You can download it here and you can find the release notes here.

While this release doesn’t have many new user-facing features, it does clean up several bugs. More importantly, however, is that Adam Goucher and Jérémy Hérault did some amazing work to lay the foundation for a plugin framework. This means that soon you’ll see Selenium IDE plugins that further expand the Selenium IDE capability.

Jérémy is working on one such plugin, called Helenium (see proof of concept in action), that will allow you to do text matching against images and PDF files using optical character recognition (OCR). We’re also working on a plugin that will make it easier to upload scripts from the IDE to BrowserMob.

If you’re interested in how to build your own plugin, I recommend reading Adam’s blog, which has a series of recent posts on how the plugin framework works.

Tweet This Post 

Selenium · Selenium IDE

We’re always working hard to improve our BrowserMob monitoring and load testing services. Over the last few weeks, we’ve pushed pushed out a bunch of improvements:

New Monitoring and Load Testing Location

Hot off the heels of Amazon’s announcement of a new US West Coast cloud data center, we are happy to report that you can now schedule load tests and monitoring jobs from this new location. Simply select the “San Jose, CA” location when scheduling tests.

Selenium 2.0 Support

In December, Selenium 2.0 alpha 1 was released. This release dramatically improves the realism and reliability of Selenium scripts. We’re proud to say that you can try out Selenium 2.0 support (but keep in mind it’s still in alpha) by simply changing your selenium script from this:

var selenium = browserMob.openBrowser();

To this:

var selenium = browserMob.openBrowser(true);

We will continue to keep BrowserMob up-to-date with all the latest happenings in the Selenium world, as well as donate our time and code back to the Selenium project. We also upgraded all the BrowserMob browsers to have the latest version of Firefox (3.5.7) and Flash.

Schedule Load Test UI Improvements

We’ve also made scheduling a load test a lot easier. We now give you a realtime estimate of what the test will cost you, changing dynamically based on your test configuration. We also display tooltips explaining things like “location”, “ramp”, and “constant”. Finally, we’re really excited to have rolled out a “Run ASAP” option that will kick off the test as quickly as it can, usually within 10 minutes.

New Script Editor

Our users love that they can write their scripts using JavaScript, so we decided to make working on that JavaScript code even easier. By utilizing the Bespin open source project from Mozilla, you now will see a rich text editor with code syntax highlighting. If for some reason you’re having trouble with it, you can always switch back to the plain text editor.

New Load Test Charts

We’ve always been proud of our realtime load test charts, but some users had recently complained that they were too heavy-weight and were slowing down their browser. Of course, this is a perfect example of why performance in the browser is starting to matter just as much as performance on the server.

Responding to this complaint, we rewrote the charts from scratch, moving from YUI Charts (Flash-based) to Flot (Canvas-based). We hope you like them!

New Scripting API Improvements

If you do advanced scripting, especially with virtual users, you’ll definitely want to take a look out the BrowserMob scripting API. We added a whole bunch of useful functions, including:

• setFollowRedirect(true) now logs all intermediate HTTP requests
• You can now automatically verify response codes
• If a 3xx response code is returned when you expected something else, the Location header is logged
• You can tie in “interceptors” for both HTTP requests and HTTP responses, allowing very advanced scripting techniques

Tweet This Post 

API · Bespin · Flot · Release Notes · Selenium · Selenium 2 · YUI Charts

We get a lot of customer requests about Flash automation, Selenium, and BrowserMob. Because our load testing and website monitoring services uses real browsers, complete with Flash 10, we can do things no one else can do, like run a load test with hundreds of Flash runtime environments driving traffic on your site.

However, while our infrastructure allows for extremely unique Flash testing support, it’s not perfectly streamlined (yet). While we are hard at work on making Flash support for Selenium and BrowserMob significantly better, at this time some work is required by Selenium users before you can get started.

Specifically, the first question that must be answered is: what automation technique do I want to employ? There are two distinct ways to go:

• Native mouse & keyboard integration – good for situations where there is minimal Flash interaction required
• API-level integration – good for situations where there is a lot of Flash interaction required

Native mouse & keyboard integration

Let’s talk briefly about native integration. This is where a mouse movement, mouse click, or keyboard action is simulated at the operating system level. It reproduces the most realistic user simulation. However, it can be very brittle, as it has limited ways to “read” from the screen and validate that functionality worked correctly.

Presently there is no great Selenium-only solution here. There are commercial functional testing products such as eggPlant, but nothing on the load side of things. BrowserMob does have, however, the ability to interact with the operating system using native key events and coordinate-based mouse interaction.

This feature is perfect for those who want to fire off one or two simple interactions, such as clicking on a movie’s “play” button or interacting with a confirmation dialog. If this is all you need for your Flash or Flex integration, please contact our support team and we’ll gladly help you get your script working using this unique API.

API-level integration

For applications that have much more complex Flash/Flex user interfaces or applications that are 100% Flash-based, we don’t recommend native events as they can be brittle and difficult to confirm that the desired functionality worked. Instead, we recommend interacting with your Flash applications using API-level integration.

Important: to do this type of automation you must be able to recompile/modify the underlying Flash object(s). If you cannot do this and require full “black box” style automation, we suggest you consider native automation or you consult with the authors of the original Flash component.

How you proceed from here depends on whether you are using Flash or Flex. The reason is that Flex exposes some automation APIs that you will otherwise have to reproduce if you are using Flash-only.

Flash automation with ExternalInterface

If you are using low-level Flash, your automation options are a little more limited:

• Using ExternalInterface directly – no 3rd parties, but might requires the most programming
• Using FlashSelenium – an open source library to make ExternalInterface callbacks easier to call

We recommend you first start off with ExternalInterface directly, as FlashSelenium is really just a thin wrapper around ExternalInterface. While it’s convenient, we think doing it by yourself once is an important learning experience.

To get started, we recommend reading the ExternalInterface documentation by Adobe. The basic idea is that within your Flash component you execute this code:

ExternalInterface.addCallback("doSomething", doSomething);

This exposes the function doSomething such that it can be executed by JavaScript. Fortunately, because Selenium can execute JavaScript using the getEval, storeEval, assertEval, etc commands, we’re in luck:

var jsExpr = "window.document.getElementById('myFlashObjId').doSomething()";
var something = selenium.getEval(jsExpr);
if (something == 'blah') {
   // throw an error, we weren't expecting 'blah'!
}

Using this technique, you can begin to expose ActionScript functions that can be used to automate the Flash component and validate that the right behavior took place.

Flex automation with Selenium-Flex

If you’re using Flex, automation should be quite a bit easier. Thanks to Selenium-Flex, you don’t have to spend time exposing dozens of functions via ExternalInterface. Instead, just compile your Flex application using the Selenium-Flex library:

1. Copy the sfapi.swc file in to your project (ie: the lib directory)
2. In your IDE/build system, ensure that the library is included (ie: -include-libraries)

That’s it! Now the next time you build your Flex app it’ll automatically have a whole bunch of functions already created. Even better, you don’t have to do all that getEval mumbo-jumbo we did for Flash testing. Instead, thanks to another open source project called FlexUISelenium (which works in concert with Selenium-Flex), the getEval stuff is abstracted away so that your tests look like this:

flex.type("2").at("arg1");
flex.type("3").at("arg2");
flex.click("submit");

If you’re doing functional programming using Java or some other full-blown programming language, you can get started with FlexUISelenium today. If you’re looking to load test or do website monitoring with Selenium, we are in the middle of adding FlexUISelenium support to the BrowserMob API. We are looking for beta participants right now, so contact us if you’re interested.

A note about recording support

Currently, none of the open source solutions referenced here support recording interactions from within Selenium IDE. While not Selenium-compatible (yet), FlexMonkey is a very good open source Flex record and playback functional testing framework. We have plans to integrate Selenium and FlexMonkey in the future, including record support through Selenium IDE.

In the meantime, you will need to write each Selenium command by hand. This means that you’ll need to be aware of the IDs of all the elements you wish to interact with. Once you know those IDs, you can write the code that issues clicks or types on the component.

Tweet This Post 

Flash · Flex · Selenium