We recently had a customer from a large clothing retailer ask us if there was any way to ensure that data, such as a username/password combination, could be restricted such that it was “checked out” and available only for a specific concurrent user. This is very common with logins, where systems often will prevent concurrent logins from multiple IP addresses.

While BrowserMob does not have a concept in which data rows can be “checked out”, some simple scripting can achieve the same results. The key is in creative use of the browserMob.getUserNum() and browserMob.getTxCount() APIs. You can learn more about them by reading up on the BrowserMob APIs.

The getUserNum function returns 0, 1, 2, etc based on the concurrent user in your load test. So if you have a 100 user test, getUserNum will return between 0 and 99. It’s important to understand that it will return the same value for the same user throughout the test.

The getTxCount function returns 1, 2, 3, etc based on the number of cycles for that specific user. This number will effectively be a counter of the unique number of transactions that that particular user has executed. So user 1 and user 100 will both have a getTxCount of 1 returned, but by the time user 100 sees it, user 1 might already be on transaction 50.

Now suppose you want to run a 1000 user test in which you never concurrently log in with the same user. All you need to do is pre-create 1000 user accounts and then write your script like so:

var userId = browserMob.getUserNum();
var username = "test-" + userId;
var password = "password";
selenium.type("username", username);
selenium.type("password", password);

This works great, but what if you want to use more than 1000 logins? Suppose you want to use up to 10,000 logins among the 1000 user test? This is where the getTxCount function comes in to:

var loginsPerUser = 10;
var userNum = browserMob.getUserNum();
var txCount = browserMob.getTxCount();
var userId = userNum * loginsPerUser + txCount % loginsPerUser;
var username = "test-" + userId;
var password = "password";
selenium.type("username", username);
selenium.type("password", password);

What this does is allocate 10 logins per concurrent user. So user 1 will get usernames test-0, test-1, …, test-9 while user 8 will get usernames test-80, test-81, …, test-89, etc. Because of the mod call (%) the ten usernames will simply wrap around once they’ve been used.

Tweet This Post 

API · Scripting

A couple weeks ago, New Relic launched a new hosted, on-demand Java profiling tool. We’re big fans of New Relic (great for Ruby or Java), dynaTrace (great for .NET or Java), and other application performance monitoring tools because they complement our website load testing and website monitoring services so well.

At BrowserMob, we’re a Java shop, and so it didn’t take long for us to have an opportunity to try out New Relic’s Java profiler. We were getting reports from some customers that viewing the response time charts of their monitoring data was occasionally slow (don’t worry, we didn’t miss the irony.)

To investigate the problem, we turned on New Relic’s profiler and selected 5 minutes of profiler time. You can choose between 1 and 10 minutes as the length of time you want to gather data. During that time, there is a small but potentially noticeable hit in performance on your site, but the upside is that the profiler is able to inspect deep in to our codebase and expose bottlenecks.

Five minutes later, we instantly saw the problem:

As you can see, during that 5 minute period, 55% of the time was spent on java.net.SocketInputStream.socketRead. Tracing back a bit, we were able to see that was from a call by MonitoringController.selectRangeForObject. It turned out we were calling a cloud-based data server that was having performance issues due to the way we constructed our API call. It didn’t take more than a few minutes to optimize the call and roll out the fix.

Such is the power of application performance monitoring tools. If you’re doing any sort of load testing, website monitoring, or page optimizations, you’re going to be very blind unless you have access to information like this. That’s why we jumped at the opportunity to partner with New Relic and dynaTrace and encourage all our customers to look at their products.

Tweet This Post 

.NET · dynaTrace · Java · Load Testing · Monitoring · New Relic · Profiler · Untitled

uTest, the world’s largest community of on-demand QA professionals, launched a major update to their customer portal over the weekend. The changes included a bunch of good stuff for both testers and end-users, including:

• A brand new Flash-based UI that loads much faster than before and is much easier to use.
• Backend optimizations to allow the uTest to continue to grow beyond their already impressive 25K+ community of testers
• A brand new API allowing for more advanced integration and partnerships.

We’re most excited about uTest’s new API support. While it requires APIs that are not yet available, we’re really excited about the prospects of being able to utilize the uTest community programatically. We hope that soon we’ll be able to offer a fully integrated service in which BrowserMob customers can simply click a checkbox to have dozens of professional QA engineers participate in a load test, verifying functionality and recording their experience as thousands of automated browsers slam the site.

We’re really excited about the power of combining the cloud and the crowd to facilitate modern website testing. And if you have a need to augment your QA team, you should definitely check out uTest, especially now that the portal looks so good!

Tweet This Post 

Crowd · uTest

Tim Koopmans, one of the partners at Altentee, a partner of ours, recently wrote up some tips on how to emulate think time and pacing of scripts in BrowserMob, specifically for load testing. Tim provides some very handy code samples, so be sure to check them out, especially if you’re concerned about pacing of transactions/sec.

Tweet This Post 

Altentee · Think Time

I just recently came across a great article titled How tracking scripts affect page loads… can Google Analytics kill my web app?:

This post explains script blocking, and then shows how to safely setup a tracking script or any external script, such as Google Analytics or Quantcast, to not block page loads or other javascript handlers on your site.

It’s a great read and provides a clear explanation on how third party components (everything from advertisements to analytics to content widgets) can slow your site down. JavaScript includes have been commonly used in websites for coming on a decade, but there are still many people who don’t understand the destruction a rogue script can cause to user experience.

One such example of that kind of destruction recently happened to customer of ours who ran an advertisement during the Super Bowl. In the weeks and months leading to the event, they had done massive amounts of tests and verified that their data center could easily handle 2.5 gigabits per second or more.

They were more than prepared and, overall, the event went off without a hitch. However, there was one minor problem that did plague their launch: a third party that served a component on their site couldn’t keep up with the load.

It turns out that one page (unfortunately, an important one that allowed visitors to search for their products) included an inline JavaScript file hosted externally. This included file was added at the last minute and didn’t get subject to the intensive testing that the rest of the site had gone through.

When the external site went down, visitors’ browsers would timeout after 1-2 minutes of trying to fetch that content before finally firing the important onload event in the browser. Nothing hosted by the company had a problem, but yet it still affected the user experience.

To make matters worse, the customer did have certain JavaScript that fired on the onload event. This JavaScript added rich functionality, such as making a calendar pop up when clicking on a date field. Because of the 1-2 minute delay, this functionality wasn’t available immediately, causing the experience to deteriorate.

In addition to all the guidance given in the aforementioned article, there are two things they can do to avoid this next time:

• Remove the third party reference entirely. In this case the reference was to some JavaScript that in turn produced a “Follow me on Twitter” image. That image could have easily been hosted in their own data center, which had been tested thoroughly.
• Take advantage of the “on content ready” psuedo event. Most modern JavaScript frameworks support a non-standard event that will fire before onload. Most of the time, this is much more desirable as it means that the JavaScript will fire closer to when the user sees content in their browser – not just when all images and JavaScript has been downloaded.

The “on content ready” psuedo-event doesn’t have an official name, but a lot of people also refer to it as “on DOM ready”. YUI, Dojo, Prototype, jQuery, and many others support it, so be on the lookout.

Tweet This Post 

3rd party widgets · Performance

Our use of a real web browsers makes it really easy to do website load testing and monitoring. By default a page is considered to have timed out if the page “onload event” can’t fire within 30 seconds. However, sometimes the page appears to be fully loaded to a typical user and shouldn’t be counted as a timeout error in your tests.

We call these errors “soft timeouts”, and they require a little bit of code to ignore them:

var selenium = browserMob.openBrowser();
 
selenium.setTimeout(2500);
 
function ignoreIfSafeTimeout(e, step, minObjects, acceptableThreshold) {
    if (e.message.indexOf("Timed out after ") == -1) {
        throw e; // this isn't a timeout, so throw it
    }
 
    var objects = step.getObjects();
    var objectCount = objects.size();
 
    if (objectCount < minObjects) {
        throw "Expected at least " + minObjects + 
              " objects but only saw " + objectCount;
    }
 
    var errors = 0;
    for (var i = 0; i &lt; objectCount; i++) {
        var object = objects.get(i);
        var code = object.getStatusCode();
        if (code < 200 || code >= 400) {
            errors++;
        }
    }
 
    var errorRatio = errors / objectCount;
    if (errorRatio > acceptableThreshold) {
        throw "Expected no more than " + acceptableThreshold + 
              " error ratio but saw " + errorRatio;
    }
}
 
browserMob.beginTransaction();
 
var step1 = browserMob.beginStep("Home Page");
try {
    selenium.open("http://marketwatch.com");
} catch (e) {
    ignoreIfSafeTimeout(e, step1, 130, 0.1);
}
 
browserMob.endStep();
 
browserMob.endTransaction();

What this code does is catch the automatic timeout exception thrown when visiting a website (in this case http://marketwatch.com) and decide whether it ignore it or not enough of the page content had loaded. There are a few things to note:

• The timeout for the page load is set to 2.5 seconds. This is set artificially low to simulate a page timeout, which is bound to happen on a heavy page like MarketWatch where there are 100+ objects.
• A custom function, ignoreIfSafeTimeout , is defined that takes several arguments:
  • e – the exception that gets thrown when opening the page
  • step – the step that holds a list of all objects requested
  • minObjects – the minimum number of objects that should be associated with the step
  • acceptableThreshold – an acceptable ratio of errors-to-successes in the objects downloaded
• When browserMob.beginStep() is called, we save a reference to it (”step1″) that gets used to call ignoreIfSafeTimeout if an exception is thrown.
• The selenium.open() command is wrapped with a try/catch block.

The most important thing in here is the call to ignoreIfSafeTimeout() in the catch block. When called, we pass in the exception that we just caught along with the step object. The last two arguments are the important ones.

The third argument, 130, means that if there is a timeout and less than 130 objects have been requested, an error will be thrown. The fourth argument, 0.1, means that if more than 10% of the requests are error codes, then an error will be thrown.

Feel free to use this code as a starting point for reducing the number of “soft timeouts” in your load tests and monitoring jobs.

Tweet This Post 

Timeouts

We often have customers ask us why their site appeared to slow down significantly, despite the fact that their CPU, RAM, and disk utilization did not rise in utilization significantly. While those three metrics are often good indicators of why systems can “slow down”, there are many other causes of performance problems. Today, we’re going to discuss one common root cause for slow websites that often gets overlooked: connection management.

Until very recently, most web browsers would only issue a maximum of two connections per host, as per the recommendation by the original HTTP/1.1 specification. This meant that if 1000 users all hit your home page at the same time, you could expect ~2000 open connections to your server. Let’s suppose that each connection consumes, on average, 0.01% of the server’s CPU and no significant RAM or disk activity.

That would mean that 2000 connections should be consuming 20% of the CPU, leaving a full 80% ready to handle additional load – or that the server should be able to handle another 4X load (4000 more users). However, this type of analysis fails to account for many other variables, most importantly the web server’s connection management settings.

Just about every web server available today (Apache, IIS, nginx, lighthttpd, etc) has one or more settings that control how connections are handle. This includes connection pooling, maximum allowed connections, Keep-Alive timeout values, etc. They all work basically the same way:

• When a request (connection) comes in to the server, the server will look at the maximum active connections setting (ie: MaxClients in Apache) and decide if it can handle the request.
• If it can, the request is processed and the number of active connections is incremented by one.
• If it can’t, the request is placed in to a queue, where it will wait in line until it finally can be processed.
• If that queue is too long (also a configuration setting in the server), the request will be rejected outright, usually with a 503 response code.

It’s this queue that can make your site to appear to be slow, despite low server utilization. Say the server allows up to 256 concurrent requests and each request takes 1 second to complete. That means if 1000 users visited the site at the same time, causing 2000 requests, then the first 128 (256/2) users would get a 1 second response time, the second 128 users would get a 2 second response time, and the last user would get an EIGHT SECOND response time.

The simple solution is to raise the concurrent request limit. However, be careful here: if you raise it too high it’s possible your server won’t have enough CPU or RAM to handle all the requests, resulting in all users be affected (rather than just some of them, like in the last example).

Also remember that not all requests are equal: a request to a dynamic search result will be much more expensive than one to a static CSS file. This is why larger sites optimize their hosting to place static files on special web servers with different configurations, usually with host names like images.example.com, while leaving their more complex content to be handled by a larger quantity of servers with a fewer number of concurrent requests on each server.

So next time you’re wondering why your site is slow, take a look at more than just CPU and RAM. Find out how the server is processing the content and see if perhaps your web server is the bottleneck.

Tweet This Post 

No tags

After running the load test customer’s often download the database and run queries to determine their sites performance. Here are some common SQL queries to get you started:

# Analyze the avg, max, and min time for each step in a transaction
SELECT step, COUNT(*), AVG(time_active), MAX(time_active), MIN(time_active) 
FROM step GROUP BY step
 
# General query to look at loading of objects
SELECT path, COUNT(*), AVG(time_active) 
FROM object GROUP BY path 
ORDER BY AVG(time_active) DESC
 
# Get error msg grouped by error, host, status
SELECT err_msg, host, status_code, COUNT(*), AVG(time_active) 
FROM object GROUP BY err_msg, host, status_code

We hope that you find these a useful starting point for your load testing analysis.

Tweet This Post 

No tags

People often ask what are the differences between RBUs and VUs. Obviously RBUs are more realistic while VUs are the perfect vehicle for testing APIs, but here is a table giving you a breakdown.

Some things to take note of:

• There is a difference in the difficulty for scripting RBUs vs. VUs. RBU scripts are easier to create with the Selenium IDE tool and have validation built in. Because RBUs mimic human behavior the
  scripts include built-in validation concepts. For example an error will be thrown if you try to click on an element that doesn’t exist because the page being displayed is a “server too busy” page instead of the expected page. With VUs, you have to code in your own validation logic.
• VUs are the preferred way to test SOAP/REST and other “non-browser” HTTP requests.
• Often the right approach involves using a bit of both RBUs and VUs. The VUs can be used to lay down base load on “read-only” pages that don’t involve form interaction or AJAX. The 80/20 rule applies here, you can generate 80% of the read-only traffic with VUs and use RBUs for the remaining 20%.

We hope this give you a better understanding of the differences and relative strengths of RBUs and VUs.

Tweet This Post 

No tags

We are big fans of what DynaTrace is doing, they are a great compliment to testing with BrowserMob. They have written an nice article on integrating their performance testing monitor with Selenium and BrowserMob.

Selenium offers a feature to add additional HTTP Tags to each Web Request via DefaultSelenium.addCustomRequestHeader. This feature allows achieving the exact same result as I explained in my previous blog entry.
In order to make it easy for Selenium script developers to use this feature we (Patrick Lightbody from BrowserMob and I) came up with the following 2 requirements:

1. every test case should automatically be tagged with the test case name. This ensures that we can link each transaction to the test case
2. the scripter must have the option to change the tagged value within a test case to allow more granular linking to individual actions in a test case

The implementation for these requirements resulted in

1. a custom TestRunner class (DynaTraceSeleniumRunner) that sets the default value for the HTTP Header to the current test case name
2. a custom Selenium class (DynaTraceSelenium) that offers the scripter to change the value for the HTTP Header

Go to the full article to get the code snippets and integration examples.

Tweet This Post 

No tags