Because both our load testing and website monitoring services are based on Selenium, we have a unique ability to measure the performance of things like page load times, AJAX timings, and other in-browser interactions.

Selenium has both a setTimeout command and a waitForPageToLoad command. Both can be given a timeout value, which will control how long Selenium waits for a given page to load or element to appear. When it comes to using our services, most people stick with the default time of 30 seconds. If the timeout is reached, an error is thrown, the script aborts, and the transaction is recorded.

However, sometimes people want to know when pages take more than X seconds to load, but don’t want to necessarily interrupt the flow of the script. In fact, just last week we got this request from a customer:

Ability to trigger an alert based on a set threshold (by the user) – not using the timeout. This basically came from the performance issues we are experiencing. Here’s the scenario:

• We have a specific page that takes 2+ minutes to load.
• The page load timeout in the Selenium script was set to 60s.
• BMob properly reported the “timeout” error but when this error happens, BMob quits the script.
• This is not ideal for me since I want to still be able to see how long the page takes to load.
• Increasing the page load timeout for the page in question works, but now I don’t have a way (that I know of) to still trigger an alert after 60s.

I should be able to set a threshold for any page I choose that would then send a notification alert.

In other words, this customer wanted for a way to still report a transaction as a failure and receive an alert, but also still allow the script to continue. Fortunately, our support for JavaScript as a scripting language provides the answer:

var timeout = 90000;
...
// start of script
...
var start = new Date().getTime();
selenium.waitForPageToLoad(timeout);
var end = new Date().getTime();
...
// rest of script
...
if ((end - start) > 45000) {
    throw "An important page took longer than 45 seconds to load";
}

What this script does is sets the timeout to a very long amount (1.5 minutes) but will still report an error if a specific page takes longer than 45 seconds. This allows the remainder of the script to execute even when the page takes more than a specified 45-second threshold.

The only problem with this script is that if the page takes longer than 90 seconds, then the rest of the script will still not run because waitForPageToLoad will throw an exception. You can solve that too with a little code:

var start = new Date().getTime();
try {
    selenium.waitForPageToLoad(timeout);
} catch (e) {
    // this will happen after 90 seconds
    // todo: recover and send the browser to the the next URL
}
var end = new Date().getTime();

The only thing that is important to remember with this use of try/catch is that you’ll need to properly recover from the error. Simply catching the error and trying to continue may not work. For example, if the next Selenium command requires clicking on a button that should have loaded from the last page, there may be no way to recover. However, if the next step is simply visiting a new URL, you could possibly get away with a simple open() command in the catch block.

Tweet This Post 

Scripting · Selenium · Timeouts

Our use of a real web browsers makes it really easy to do website load testing and monitoring. By default a page is considered to have timed out if the page “onload event” can’t fire within 30 seconds. However, sometimes the page appears to be fully loaded to a typical user and shouldn’t be counted as a timeout error in your tests.

We call these errors “soft timeouts”, and they require a little bit of code to ignore them:

var selenium = browserMob.openBrowser();
 
selenium.setTimeout(2500);
 
function ignoreIfSafeTimeout(e, step, minObjects, acceptableThreshold) {
    if (e.message.indexOf("Timed out after ") == -1) {
        throw e; // this isn't a timeout, so throw it
    }
 
    var objects = step.getObjects();
    var objectCount = objects.size();
 
    if (objectCount < minObjects) {
        throw "Expected at least " + minObjects + 
              " objects but only saw " + objectCount;
    }
 
    var errors = 0;
    for (var i = 0; i &lt; objectCount; i++) {
        var object = objects.get(i);
        var code = object.getStatusCode();
        if (code < 200 || code >= 400) {
            errors++;
        }
    }
 
    var errorRatio = errors / objectCount;
    if (errorRatio > acceptableThreshold) {
        throw "Expected no more than " + acceptableThreshold + 
              " error ratio but saw " + errorRatio;
    }
}
 
browserMob.beginTransaction();
 
var step1 = browserMob.beginStep("Home Page");
try {
    selenium.open("http://marketwatch.com");
} catch (e) {
    ignoreIfSafeTimeout(e, step1, 130, 0.1);
}
 
browserMob.endStep();
 
browserMob.endTransaction();

What this code does is catch the automatic timeout exception thrown when visiting a website (in this case http://marketwatch.com) and decide whether it ignore it or not enough of the page content had loaded. There are a few things to note:

• The timeout for the page load is set to 2.5 seconds. This is set artificially low to simulate a page timeout, which is bound to happen on a heavy page like MarketWatch where there are 100+ objects.
• A custom function, ignoreIfSafeTimeout , is defined that takes several arguments:
  • e – the exception that gets thrown when opening the page
  • step – the step that holds a list of all objects requested
  • minObjects – the minimum number of objects that should be associated with the step
  • acceptableThreshold – an acceptable ratio of errors-to-successes in the objects downloaded
• When browserMob.beginStep() is called, we save a reference to it (”step1″) that gets used to call ignoreIfSafeTimeout if an exception is thrown.
• The selenium.open() command is wrapped with a try/catch block.

The most important thing in here is the call to ignoreIfSafeTimeout() in the catch block. When called, we pass in the exception that we just caught along with the step object. The last two arguments are the important ones.

The third argument, 130, means that if there is a timeout and less than 130 objects have been requested, an error will be thrown. The fourth argument, 0.1, means that if more than 10% of the requests are error codes, then an error will be thrown.

Feel free to use this code as a starting point for reducing the number of “soft timeouts” in your load tests and monitoring jobs.

Tweet This Post 

Timeouts