Tim Koopmans, one of the partners at Altentee, a partner of ours, recently wrote up some tips on how to emulate think time and pacing of scripts in BrowserMob, specifically for load testing. Tim provides some very handy code samples, so be sure to check them out, especially if you’re concerned about pacing of transactions/sec.

Tweet This Post 

Altentee · Think Time

I just recently came across a great article titled How tracking scripts affect page loads… can Google Analytics kill my web app?:

This post explains script blocking, and then shows how to safely setup a tracking script or any external script, such as Google Analytics or Quantcast, to not block page loads or other javascript handlers on your site.

It’s a great read and provides a clear explanation on how third party components (everything from advertisements to analytics to content widgets) can slow your site down. JavaScript includes have been commonly used in websites for coming on a decade, but there are still many people who don’t understand the destruction a rogue script can cause to user experience.

One such example of that kind of destruction recently happened to customer of ours who ran an advertisement during the Super Bowl. In the weeks and months leading to the event, they had done massive amounts of tests and verified that their data center could easily handle 2.5 gigabits per second or more.

They were more than prepared and, overall, the event went off without a hitch. However, there was one minor problem that did plague their launch: a third party that served a component on their site couldn’t keep up with the load.

It turns out that one page (unfortunately, an important one that allowed visitors to search for their products) included an inline JavaScript file hosted externally. This included file was added at the last minute and didn’t get subject to the intensive testing that the rest of the site had gone through.

When the external site went down, visitors’ browsers would timeout after 1-2 minutes of trying to fetch that content before finally firing the important onload event in the browser. Nothing hosted by the company had a problem, but yet it still affected the user experience.

To make matters worse, the customer did have certain JavaScript that fired on the onload event. This JavaScript added rich functionality, such as making a calendar pop up when clicking on a date field. Because of the 1-2 minute delay, this functionality wasn’t available immediately, causing the experience to deteriorate.

In addition to all the guidance given in the aforementioned article, there are two things they can do to avoid this next time:

• Remove the third party reference entirely. In this case the reference was to some JavaScript that in turn produced a “Follow me on Twitter” image. That image could have easily been hosted in their own data center, which had been tested thoroughly.
• Take advantage of the “on content ready” psuedo event. Most modern JavaScript frameworks support a non-standard event that will fire before onload. Most of the time, this is much more desirable as it means that the JavaScript will fire closer to when the user sees content in their browser – not just when all images and JavaScript has been downloaded.

The “on content ready” psuedo-event doesn’t have an official name, but a lot of people also refer to it as “on DOM ready”. YUI, Dojo, Prototype, jQuery, and many others support it, so be on the lookout.

Tweet This Post 

3rd party widgets · Performance

Our use of a real web browsers makes it really easy to do website load testing and monitoring. By default a page is considered to have timed out if the page “onload event” can’t fire within 30 seconds. However, sometimes the page appears to be fully loaded to a typical user and shouldn’t be counted as a timeout error in your tests.

We call these errors “soft timeouts”, and they require a little bit of code to ignore them:

var selenium = browserMob.openBrowser();
 
selenium.setTimeout(2500);
 
function ignoreIfSafeTimeout(e, step, minObjects, acceptableThreshold) {
    if (e.message.indexOf("Timed out after ") == -1) {
        throw e; // this isn't a timeout, so throw it
    }
 
    var objects = step.getObjects();
    var objectCount = objects.size();
 
    if (objectCount < minObjects) {
        throw "Expected at least " + minObjects + 
              " objects but only saw " + objectCount;
    }
 
    var errors = 0;
    for (var i = 0; i < objectCount; i++) {
        var object = objects.get(i);
        var code = object.getStatusCode();
        if (code < 200 || code >= 400) {
            errors++;
        }
    }
 
    var errorRatio = errors / objectCount;
    if (errorRatio > acceptableThreshold) {
        throw "Expected no more than " + acceptableThreshold + 
              " error ratio but saw " + errorRatio;
    }
}
 
browserMob.beginTransaction();
 
var step1 = browserMob.beginStep("Home Page");
try {
    selenium.open("http://marketwatch.com");
} catch (e) {
    ignoreIfSafeTimeout(e, step1, 130, 0.1);
}
 
browserMob.endStep();
 
browserMob.endTransaction();

What this code does is catch the automatic timeout exception thrown when visiting a website (in this case http://marketwatch.com) and decide whether it ignore it or not enough of the page content had loaded. There are a few things to note:

• The timeout for the page load is set to 2.5 seconds. This is set artificially low to simulate a page timeout, which is bound to happen on a heavy page like MarketWatch where there are 100+ objects.
• A custom function, ignoreIfSafeTimeout , is defined that takes several arguments:
  • e – the exception that gets thrown when opening the page
  • step – the step that holds a list of all objects requested
  • minObjects – the minimum number of objects that should be associated with the step
  • acceptableThreshold – an acceptable ratio of errors-to-successes in the objects downloaded
• When browserMob.beginStep() is called, we save a reference to it (”step1″) that gets used to call ignoreIfSafeTimeout if an exception is thrown.
• The selenium.open() command is wrapped with a try/catch block.

The most important thing in here is the call to ignoreIfSafeTimeout() in the catch block. When called, we pass in the exception that we just caught along with the step object. The last two arguments are the important ones.

The third argument, 130, means that if there is a timeout and less than 130 objects have been requested, an error will be thrown. The fourth argument, 0.1, means that if more than 10% of the requests are error codes, then an error will be thrown.

Feel free to use this code as a starting point for reducing the number of “soft timeouts” in your load tests and monitoring jobs.

Tweet This Post 

Timeouts

We often have customers ask us why their site appeared to slow down significantly, despite the fact that their CPU, RAM, and disk utilization did not rise in utilization significantly. While those three metrics are often good indicators of why systems can “slow down”, there are many other causes of performance problems. Today, we’re going to discuss one common root cause for slow websites that often gets overlooked: connection management.

Until very recently, most web browsers would only issue a maximum of two connections per host, as per the recommendation by the original HTTP/1.1 specification. This meant that if 1000 users all hit your home page at the same time, you could expect ~2000 open connections to your server. Let’s suppose that each connection consumes, on average, 0.01% of the server’s CPU and no significant RAM or disk activity.

That would mean that 2000 connections should be consuming 20% of the CPU, leaving a full 80% ready to handle additional load – or that the server should be able to handle another 4X load (4000 more users). However, this type of analysis fails to account for many other variables, most importantly the web server’s connection management settings.

Just about every web server available today (Apache, IIS, nginx, lighthttpd, etc) has one or more settings that control how connections are handle. This includes connection pooling, maximum allowed connections, Keep-Alive timeout values, etc. They all work basically the same way:

• When a request (connection) comes in to the server, the server will look at the maximum active connections setting (ie: MaxClients in Apache) and decide if it can handle the request.
• If it can, the request is processed and the number of active connections is incremented by one.
• If it can’t, the request is placed in to a queue, where it will wait in line until it finally can be processed.
• If that queue is too long (also a configuration setting in the server), the request will be rejected outright, usually with a 503 response code.

It’s this queue that can make your site to appear to be slow, despite low server utilization. Say the server allows up to 256 concurrent requests and each request takes 1 second to complete. That means if 1000 users visited the site at the same time, causing 2000 requests, then the first 128 (256/2) users would get a 1 second response time, the second 128 users would get a 2 second response time, and the last user would get an EIGHT SECOND response time.

The simple solution is to raise the concurrent request limit. However, be careful here: if you raise it too high it’s possible your server won’t have enough CPU or RAM to handle all the requests, resulting in all users be affected (rather than just some of them, like in the last example).

Also remember that not all requests are equal: a request to a dynamic search result will be much more expensive than one to a static CSS file. This is why larger sites optimize their hosting to place static files on special web servers with different configurations, usually with host names like images.example.com, while leaving their more complex content to be handled by a larger quantity of servers with a fewer number of concurrent requests on each server.

So next time you’re wondering why your site is slow, take a look at more than just CPU and RAM. Find out how the server is processing the content and see if perhaps your web server is the bottleneck.

Tweet This Post 

No tags

After running the load test customer’s often download the database and run queries to determine their sites performance. Here are some common SQL queries to get you started:

# Analyze the avg, max, and min time for each step in a transaction
SELECT step, COUNT(*), AVG(time_active), MAX(time_active), MIN(time_active) 
FROM step GROUP BY step
 
# General query to look at loading of objects
SELECT path, COUNT(*), AVG(time_active) 
FROM object GROUP BY path 
ORDER BY AVG(time_active) DESC
 
# Get error msg grouped by error, host, status
SELECT err_msg, host, status_code, COUNT(*), AVG(time_active) 
FROM object GROUP BY err_msg, host, status_code

We hope that you find these a useful starting point for your load testing analysis.

Tweet This Post 

No tags

People often ask what are the differences between RBUs and VUs. Obviously RBUs are more realistic while VUs are the perfect vehicle for testing APIs, but here is a table giving you a breakdown.

Some things to take note of:

• There is a difference in the difficulty for scripting RBUs vs. VUs. RBU scripts are easier to create with the Selenium IDE tool and have validation built in. Because RBUs mimic human behavior the
  scripts include built-in validation concepts. For example an error will be thrown if you try to click on an element that doesn’t exist because the page being displayed is a “server too busy” page instead of the expected page. With VUs, you have to code in your own validation logic.
• VUs are the preferred way to test SOAP/REST and other “non-browser” HTTP requests.
• Often the right approach involves using a bit of both RBUs and VUs. The VUs can be used to lay down base load on “read-only” pages that don’t involve form interaction or AJAX. The 80/20 rule applies here, you can generate 80% of the read-only traffic with VUs and use RBUs for the remaining 20%.

We hope this give you a better understanding of the differences and relative strengths of RBUs and VUs.

Tweet This Post 

No tags

We are big fans of what DynaTrace is doing, they are a great compliment to testing with BrowserMob. They have written an nice article on integrating their performance testing monitor with Selenium and BrowserMob.

Selenium offers a feature to add additional HTTP Tags to each Web Request via DefaultSelenium.addCustomRequestHeader. This feature allows achieving the exact same result as I explained in my previous blog entry.
In order to make it easy for Selenium script developers to use this feature we (Patrick Lightbody from BrowserMob and I) came up with the following 2 requirements:

1. every test case should automatically be tagged with the test case name. This ensures that we can link each transaction to the test case
2. the scripter must have the option to change the tagged value within a test case to allow more granular linking to individual actions in a test case

The implementation for these requirements resulted in

1. a custom TestRunner class (DynaTraceSeleniumRunner) that sets the default value for the HTTP Header to the current test case name
2. a custom Selenium class (DynaTraceSelenium) that offers the scripter to change the value for the HTTP Header

Go to the full article to get the code snippets and integration examples.

Tweet This Post 

No tags

Today dynaTrace announced that it has intergrated support of Selenium and BrowserMob to allow customers to tie their web load testing to server performance.

In my recent blog post – Get more out of functional web testing: How to correlate test reports with server side log information? – I discussed the problem that testing results are usually not linked to the log and diagnostics information captured by the application under test. The blog entry offered a way to link the two sides using HTTP Tagging via an HTTP Proxy. Tagging individual Web Requests allows linking each individual request executed by the testing tool with the transactions that are executed on the server side. Your logging framework or diagnostics solution can then take this tag and link the transaction to the originating web request. (via dyanTrace Blog)

This is a great solutiong for in depth performance diagnostics.

Tweet This Post 

No tags

Steve Sauders, the master of building fast websites, wrote up a very nice summary how JavaScript is loaded by different browsers and how you can optimize your page performance by following a few techniques he outlines.

We have a lot of customers that either have 3rd part widgets on their site or provide them to other websites. In either case, they are often concerned that the JavaScript on the page isn’t slowing down the user experience.

If you’re in the business of selling or using widgets, read Steve’s article and double check that the widgets you’re working with use some of these techniques. Otherwise, your website visitors will be the ones suffering when there may be a very simple fix that shaves hundreds of milliseconds off page load times!

Tweet This Post 

No tags

Google recently posted an article on their testing blog: Survival techniques for acceptance tests of web applications (Part 1). In it they note the difference between various expressions used with Selenium to get access to an object on a web page:

For each method, we need to work out how to implement it in code. How could an automated test select the compose message icon? Do alternative ways exist? An understanding of HTML, CSS, and JavaScript will help you if you plan to use browser automation tools. All the visible elements of a web application are reflected in the Document Object Model (DOM) in HTML, and they can be addressed in various ways: the directions from the root of the document to the element using xpath; unique identifiers; or characteristics possessed by the elements, such as class names, attributes, or link text. Some examples of these addressing options are shown in the Navigation Options illustration below. (Notes: navigation using xpath is much slower than using IDs; and IDs should be unique.)

We agree that IDs are usually the best approach, but sometimes (especially with AJAX.NET applications) those IDs are actually really bad to use. The reason is that some frameworks (AJAX.NET, ExtJS, etc) use ID generation techniques that aren’t the same for every page view or every web app build.

As such, keep an eye out for suspicious IDs and be prepared to switch back to XPath or DOM locators when necessary. For example, if you see an ID like ctl00_ctl00_cp_cpTab_txtFirstName (generated from AJAX.NET) consider changing it to an XPath such as //input[contains(@id, 'txtFirstName')].

Related to that topic, they also note that scripts recorded in Selenium IDE should not be considered “done” and will likely need to be optimized and tweaked before they can be considered stable:

The open-source test automation tool Selenium (http://seleniumhq.org/) includes a simple IDE record and playback tool that runs in the Firefox browser. Recorded scripts can help bootstrap your automated tests. However, don’t be tempted to consider the recorded scripts as automated tests: they’re unlikely to be useful for long. Instead, plan to design and implement your test code properly, using good software design techniques.

The author goes on to suggest Firebug as a great tool for doing this kind of work, which we tend to agree with and have covered previously. But they missed one of the most important functions: the ability to easily test XPath expressions from within Firebug!

Very few Firebug users know about this, so take note: Firebug adds a $x function that can be called from the console. It takes an XPath in the form of a String and will spit out the HTML elements that it evaluates to directly back in the console. You can then mouse over the elements and check that the XPath is working correctly:

Armed with Selenium IDE, Firebug, and $x function, you should be able to build out solid Selenium scripts in no time!

Shameless plug: these types of tips (and more) are yours for free when you sign up for a BrowserMob performance and load testing account.

Tweet This Post 

No tags