At BrowserMob we have two different levels of load testing support: one for Real Browser Users (RBUs) and one for Virtual Users (VUs). Often it’s not immediately clear what the difference is and why you’d use one over the other.

RBU Benefits

The high level difference is this: an 100 RBU test means that there will be 100 browsers in the cloud all loading your site and interacting with it, while a 100 VU test means that there will be 100 “threads” issuing HTTP GET and POST requests to your website, but not actually running a browser. This means that RBUs:

• Check functionality – they don’t just make HTTP requests, they validate if the application is actually working as an end-user would expect it to.
• Provide better error reporting – because there is an actual browser on a desktop running, we can take a screenshot to show you what the failure looked like.
• Are more realistic – today’s browsers load up to 15 objects at a time and have very unique HTTP request patterns that are almost impossible to simulate without a real browser.
• Use a wide range of IP addresses – because RBUs need a lot of resources, we allocate between 1 and 6 browsers per IP.
• Can measure real user wait time – sometimes the UI can’t be used until multiple overlapping AJAX calls are finished. Only a real browser can tell you when the UI is ready for interaction.

RBU Cost

When it comes to load testing website applications, RBUs are superior in almost every way. However, with all that realism and IP address distribution comes a massive hardware requirement: for every RBU in a test we allocate a full CPU core and 1.5GB+ of RAM. That means that a 100 RBU test requires at least 100 CPU cores and 150GB of RAM.

Compared to traditional load testing, which only requires half a CPU core or less and 256MB of RAM, it’s very different. This resource-hungry nature of RBUs means that they cost 10X more than our Virtual User service. That means a 100 RBU test will cost the same as a 1000 VU test. As such, depending on your budget, it may make sense to use a mixture of RBUs and VUs.

Scripting Effort

While RBUs are more realistic but cost a bit more, it also helps to understand the level of effort to write and maintain RBY and VU scripts. Remember that RBU scripts interact at the application level, while VU scripts interact at the protocol level.

Consider this RBU script:

var selenium = browserMob.openBrowser();
 
selenium.open("http://google.com/");
selenium.type("q", "BrowserMob");
selenium.click("btnG");
 
selenium.waitForPageToLoad(30000);
if (!selenium.isTextPresent("BrowserMob")) {
    throw "BrowserMob text not found";
}

Written as a VU script, each and every HTTP request would need to be scripted:

var c = browserMob.openHttpClient();
 
// get the home page and verify we got a 200 OK
c.get("http://www.google.com/", 200);
c.get("http://www.google.com/intl/en_ALL/images/logo.gif");
c.get("http://www.google.com/extern_js/f/.../Y3ut8jJ-OXk.js");
c.get("http://www.google.com/extern_chrome/be5ffa94030d2d34.js");
c.get("http://clients1.google.com/generate_204");
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://www.google.com/csi?v=3&s=webhp&action=&e=...&rt=...");
 
// simulate the AJAX event for the Google auto-complete
// also verify a 200 OK response
c.get("http://clients1.google.com/complete/search?hl=en&client=hp&q=BrowserMob&cp=27", 200);
 
// simulate searching
// also verify a 200 OK response
var resp = c.get("http://www.google.com/search?hl=en&source=hp&q=BrowserMob&btnG=...", 200);
 
// get the other requests caused by the new page load
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://id.google.com/verify/EAAAAECIWFsnXJixiQogbBCZlzA.gif");
c.get("http://www.google.com/csi?v=3&s=web&action=&ei=slYdS_-MGpXKsAPngsX8BA&e=...");
c.get("http://clients1.google.com/generate_204");
 
// finally, validate the browsermob was in the main HTTP request
if (resp.getBody().indexOf("BrowserMob") == -1) {
    throw "BrowserMob text not found";
}

When to Use Virtual Users

As you can imagine, writing and maintaining VU scripts is a lot more work. However, that doesn’t mean VUs are always a bad idea. In fact, they are great for:

• Placing a large amount of low-cost “dumb” load on a website. For example, when trying to saturate simple hits again a home page.
• When testing a web service, such as REST or SOAP, that can’t be directly tested with a web browser.
• When budget constraints require VUs

As a general rule of thumb, if your own hourly cost to write a VU script is more than it would be to quickly run an RBU test, you should probably go with an RBU. However, if it’s an extremely large scale test, VUs may be worth considering for a good portion of the load.

Don’t Forget!

Finally, don’t forget that there is no reason you can’t run two load tests at the same time:

• A smaller RBU test that lets you measure what the real user experience is like
• A larger VU test that hammers at the site in a cost-effective way

This technique allows you to stay within your budget without giving up entirely on the benefits of RBU scripts.

Tweet This Post 

RBU · VU