Today we received an excellent question from a customer of ours:

We were wondering if you all have any information that says “X Unique visitors per day translates into Y simultaneous users at any given time.”

Essentially, we’re looking for a way to determine how many simultaneous users we should load test with if we know the sites normal daily traffic.

While every site is different, we recommend following this line of reasoning to help you find the answer. Suppose your site gets 100K unique visitors per day, with peak traffic in the mornings and afternoons. Assume that 40% of the traffic comes between 7AM and 11AM, 40% at 4PM to 9PM, and 20% at other times. This means during your peak hours (7AM to 11AM and 4PM to 9PM) you’ll get ~10% of your unique visitors per hour, or 10K uniques in our example.

Now that you know how many unique visitors you’ll get in an hour, you can start turning that in to concurrent users. To do that, it’s important to understand that a unique visitor is roughly equivalent of a transaction. So really you want to figure out how many users you need to reach 10K transactions in an hour.

Suppose your script (or scripts) take an average of 2 minutes to complete. That means a single user will execute 30 transactions in an hour. So to reach 10K transactions, you’d need 334 users (10K divided by 30). If you decide you want to create realistic scripts that include human think time, then the scripts will take that much longer and you’ll need that many more concurrent users. For example, if the script gets 5X think time added and now takes 10 minutes to run, then you’ll need 1,667 users (10K divided by 6).

Of course, this calculation will only get you the load on a typical day (assuming a single hour sees 10% of traffic). Your traffic patterns may vary, or you may want to prepare for a larger surge. For example, if you want to test what happens when 60% of the daily traffic visits in an hour, then you’d need 2,000 users (60K divided by 30).

Tweet This Post 

No tags

When setting up monitoring jobs, there are often predictable time periods in which you want to change the behavior of a script or prevent it from running at all, without having to manually stop/start the monitoring job each time. For instance, you might want to prevent errors and alert emails during routine maintenance windows, or perhaps you’re only interested in site performance on weekdays during regular business hours. We’ve come up with solutions to a few common situations that will help jump start your scripts.

Overview
At the core of all these examples is a small configuration object that sits at the top of the script for quick changes, and a single function that returns true or false depending on whether or not the current time is within a blackout period. The following is some pseudo-code outlining one way in which you might choose to implement these examples in your own scripts. Don’t forget to add in the appropriate maintenance object and isInMaintenance() function from the examples below to make this work.

// The configuration object for the maintenance window
var maintenance = {...}
 
// The maintenance function (plug in the appropriate function)
var isInMaintenance = function() {
    ...
}
 
var selenium = browserMob.openBrowser();
 
if(!browserMob.isValidation() && isInMaintenance()) {
    // Active maintenance window
    browserMob.log("Is in maintenance!");
} else {
    // Run normal script
    browserMob.beginTransaction();
 
    browserMob.beginStep("Step 1");
    selenium.open("http://google.com/search?q=website+monitoring");
    ...
    browserMob.endStep();
 
    browserMob.endTransaction();
}

Web Page Check
The most straight forward way to check for a maintenance window is to have Selenium load a web page and look for an element that’s only present during scheduled down time. You’ll want to be careful not to choose something that is also present when the site is really in trouble or you will never get your alerts!

var maintenance = {
    check: {
        url: "http://example.com/home",
        element: "//img[@title=\"maintenance\"]"
    }
}
 
var isInMaintenance = function() {
    if(maintenance.check && maintenance.check.url != null && maintenance.check.element != null) {
        selenium.open(maintenance.url);
        return selenium.isElementPresent(maintenance.element);
    }
 
    return false;
}

Multipe Date Ranges
One of the most common requests we get is for a way to handle scheduled maintenance windows. These windows tend to be one-off date ranges decided well in advance of the actual event. Given a set of start/end dates, you could prevent monitoring errors and email alerts during the blackout periods. When setting the dates, don’t forget to include the time zone abbreviation in the argument string or the script will use the local time of the machine it’s running on.

var maintenance = {
    dates: [
        {start: new Date("05/12/2010 23:00 PDT"), end: new Date("05/13/2010 02:30 PDT")},
        {start: new Date("05/24/2010 23:00 PDT"), end: new Date("05/25/2010 02:30 PDT")}
    ]
}
 
var isInMaintenance = function() {
	var now = new Date(),
	    nowTime = now.getTime(),
	    maintenanceWindows = [];
 
    // Check for maintenance date ranges
    if(maintenance.dates && maintenance.dates.length) {
        maintenanceWindows = maintenance.dates;
    }
 
    for(var i in maintenanceWindows) {
        var m = maintenanceWindows[i];
 
        if(nowTime >= m.start.getTime() && nowTime < m.end.getTime()) {
            return true;
        }
    }
 
    return false;
}

Days of the Week
Here’s an example for blacking out specific days of the week. This will allow you to create a script that, for example, monitors on Mondays, Wednesdays, and Fridays, or only on business days. Simply provide a time zone offset, which is the number of minutes your time zone deviates from coordinated universal time (a.k.a. UTC, GMT, Zulu time), and the days of the week you don’t want to run the script (from 0-6, where 0 is Sunday, 1 is Monday, etc.).

var maintenance = {
    timezoneOffset: 240, // UTC offset in minutes (eg: NYC (GMT-4) is 4 * 60
    days: [0, 6]    // Sunday = 0, Saturday = 6
}
 
var isInMaintenance = function(){
    var now = new Date(),
        nowTime = now.getTime(),
        timezoneOffset = maintenance.timezoneOffset || 0,
        localTimezoneOffset = now.getTimezoneOffset(),
        timezoneDiff = localTimezoneOffset - timezoneOffset;
 
    // Check for full blackout days (such as weekends)
    if(maintenance.days != null && maintenance.days.length) {
        var targetDate = new Date(nowTime + (timezoneDiff * 60 * 1000)),
            targetDay = targetDate.getDay();
 
        for(var i in maintenance.days) {
            if(maintenance.days[i] == targetDay) {
                return true;
            }
        }
    }
 
    return false;
}

Time of Day
This example shows how you can have a daily blackout range. This code will parse the start/end strings into a daily time range adjusted to the target time zone. Use this if you only want to monitor during banker’s hours, or if you want to prevent a script from running while your database is backing up.

var maintenance = {
    timezoneOffset: 240, // UTC offset in minutes (eg: NYC (GMT-4) is 4 * 60
    hours: {start: "23:30", end: "07:30"}
}
 
var isInMaintenance = function(startTime, endTime, offset) {
    var offset = maintenance.timezoneOffset || 0;
    var now = new Date();
    var nowTime = now.getTime();
    var maintenanceWindows = [];
 
    var toUtc = function(localTime) {
        var chunks = localTime.toString().split(":");
        var h = parseInt(chunks[0], 10);
        var m = parseInt(chunks[1], 10) || 0;
 
        var d = new Date();
        d.setHours(h);
        d.setMinutes(m + offset);
 
        return d.getHours() + ":" + d.getMinutes();
    }
 
    var sH, eH, sM, eM;
 
    startTime = toUtc(maintenance.hours.start);
    var chunks = startTime.toString().split(":");
    sH = parseInt(chunks[0], 10);
    sM = parseInt(chunks[1], 10) || 0;
 
    endTime = toUtc(maintenance.hours.end);
    chunks = endTime.toString().split(":");
    eH = parseInt(chunks[0], 10);
    eM = parseInt(chunks[1], 10) || 0;
 
    var startDate = new Date(now.getFullYear(), now.getMonth(), now.getDate(), sH, sM, 0, 0);
    var endDate = new Date(now.getFullYear(), now.getMonth(), now.getDate(), eH, eM, 0, 0);
 
    if(startDate.getTime() <= endDate.getTime()) {
        // Single range mid-day
        maintenanceWindows.push({start: startDate, end: endDate});
    } else {
        // Date wraps, which means two blackout ranges in a single day
        var morning = new Date(now.getFullYear(), now.getMonth(), now.getDate(), 0, 0, 0, 0),
            midnight = new Date(morning.getTime() + (24 * 60 * 60 * 1000));
 
        maintenanceWindows.push({
            start: morning,
            end: endDate
        });
 
        maintenanceWindows.push({
            start: startDate,
            end: midnight
        });
    }
 
    for(var i in maintenanceWindows) {
        var m = maintenanceWindows[i];
        if(nowTime >= m.start.getTime() && nowTime < m.end.getTime()) {
            return true;
        }
    }
 
    return false;
}

Now, a few of you might have noticed that the above examples are compatible with one another; you can mix-and-match them as you see fit. Just combine the various maintenance object fields into one config object, combine the contents of all the maintenance methods you want to use (just watch not to overwrite the variable declarations), and you can create something that fits just about any situation.

Tweet This Post 

downtime · javascript · maintenance · maintenance windows · Monitoring · RBU · Scripting · scripts

For load tests, yes (we’re working to add support for this to monitoring). You can use BrowserMob API to add any name/value pair to your transactions or steps. Here’s a simple example:

var selenium = browserMob.openBrowser();
 
var tx = browserMob.beginTransaction();
 
var step = browserMob.beginStep("Home Page");
 
selenium.open("http://example.com");
 
// record how long it took for some text to appear
var start = new Date().getTime();
selenium.waitForTextPresent("dynamic text");
var end = new Date().getTime();
 
// get a special cookie out and store it
var cookie = selenium.getCookieByName("myCookie");
tx.put("cookie_value", cookie);
 
// and store the "text" timing, associated with the step
step.put("text_timing", end - start);
 
browserMob.endStep();
 
browserMob.endTransaction();

The important things to notice here are that both beginTransaction() and beginStep() actually return objects that represent the transaction or step, respectively. You can then call put() on those objects to associate arbitrary data, such as custom timings or cookie values, with them.

Then when the transaction is stored in the load test database, you’ll be able to look them up in the name_value_pairs table. For more information on the test database schema, consult the documentation.

Tweet This Post 

API

A couple days ago we announced a brand new pricing system that lowered our pricing across the board. For our existing customers, this pricing is quite a bit different than what they were used to, so we wanted to put together a list Frequently Asked Questions about the new plans.

Is this new pricing more or less expensive?

By any metric, this a price reduction. We’re extremely excited to have found a model that controls costs for our customers while actually making the revenue flow for our company healthier.

Not only did we reduce the overall pricing for monitoring and/or give a better value for the price, we created a new way to buy load testing from us that is significantly less expensive for most projects.

But even if you stick with the old model of paying for load tests, we’ve reduced the price because the first 25 RBUs or 100 VUs are always sold “at cost”. This means that a 100 RBU test that used to cost $300 now costs $15 under the new plan and $228.75 when using the old plan. To understand how you can choose between either plan, please see the Can I still use my old pay-as-you-go plan? question.

How has pricing for monitoring changed?

The old pricing model for monitoring used a complex formula in which the time of the script being used to monitor your site heavily influenced the overall cost. For example, a job that averaged 3 seconds per run might cost half of that of a script that took 7 seconds to run. The problem with this model was that if your site slowed down, your costs would go up, making your monitoring budget unpredictable.

The new pricing model eliminates that unpredictability. Instead, there is a constant number of monthly page views that you’ll used, based entirely on your script, number of locations you check from, and frequency at which you check. No matter how fast or how slow your site is, the price will remain constant.

For example, if you wanted to test a script that hit your home page, visited the login page by clicking the login link, and then logged in, that would be 3 page views. If your monitoring job ran that script every 15 minutes from two locations, you’d use 17,280 page views per month:

17,280 = 3 page views x 60 mins / 15 min freq. 
           x 24 hours x 30 days x 2 locations

Our pricing packages start at 3,000 monthly page views per month (FREE) and go all the way up to 350K monthly page views.

What happens to my current monitoring jobs?

If you were paying for a monthly, recurring plan, then you have already been converted over to the new model and are now paying less than you were before. That’s because we’ve lowered our pricing for all monitoring packages. For example, what was $300/mo is now $249/mo, and what was $175/mo is now $149/mo.

However, if you were not using a monthly credit package to pay for your monitoring, you may need to take some action. Your monitoring jobs have not been disabled, but they are currently running free of charge. Over the next few weeks, we will be contacting each customer who is in this situation and work with them on finding a monitoring package that fits their needs.

In some cases, a customer may want to transfer credits previously purchased and apply them toward a monitoring job. We are more than happy to convert purchases over this way and will work with each customer that is interested in doing this.

How has pricing for load testing changed?

In our previous pricing model, each concurrent RBU cost 10 credits/hour while each concurrent VU cost 1 credit/hour. Customers could buy credits at various rates ranging from 30 cents/credit down to 7.5 cents/credit, depending on the size of the purchase. Purchases of $5,000 or larger were eligible to purchase additional credits of any amount at the original price (usually 7.5 or 10 cents).

Under the new model, there are no more credits. Rather, we have converted them all to “cloud dollars” by multiplying the remaining credits in your account by the price paid for the credits. For example, if you purchased 1000 credits for $300 and had 300 at the time of the switch, you would be left with $90 in cloud dollars.

These cloud dollars can be used to pay for load tests. The price for each load test will be quoted at the bottom of the “schedule load test” page. If the test is larger than the load test package that you currently have, we will quote a higher price that is based on the old pay-as-you-go pricing plan.

How do “cloud dollars” compare to credits?

If you’re used to the old credit system, the easiest way to think of cloud dollars is that they reflect a credit price of just 1.5 cents – a 20X reduction from the 30 cent price most of our customers were using.

This means that a 100 RBU test that used to cost $300 now costs only $15 when you are subscribed to a qualifying load test package. For customers who purchased a larger credit package and therefore bought in to a smaller credit price (ie: 15 or 10 cents/credit), the savings are not as large but are still very significant (ie: 10X or 6.67X, respectively.)

Can I still use my old pay-as-you-go plan?

Absolutely.

Simply keep buying cloud dollars just like if they were the old credit system. When you schedule a test, you’ll be quoted a price to run the test. You may be presented with a note explaining that the per-test fee would be lower if you were using a load test package, but you are under no obligation to sign up for such a package. This is especially ideal for anyone who does infrequent load testing.

The price of test will be based on what your original credit price was. This means that a 100 RBU test might cost $228.75 for one customer at the 30 cent price point and $78.75 for another customer at the 10 cent price point. However, both would only pay $15 if they were subscribed to a load test package.

Can I upgrade to the new pricing?

Yes. We can apply any unused cloud dollars towards load testing or monitoring packages. Just contact us and we can get the process started.

What is the threshold for the free BrowserMob services?

For monitoring, you can do up to 3,000 page views per month for free.

For load testing, you can run up to 25 RBUs and 100 VUs at any time without paying for a load testing package. However, you will have to pay the raw cloud fees for each test. These fees are 15 cents per RBU hour and 1.5 cents per VU hour (ie: 25 RBUs = $3.75 and 100 VUs = $1.50). Each account comes with 10 cloud dollars to get you started, but you can purchase more at any time.

Tweet This Post 

Load Testing · Monitoring · Pricing

At BrowserMob we have two different levels of load testing support: one for Real Browser Users (RBUs) and one for Virtual Users (VUs). Often it’s not immediately clear what the difference is and why you’d use one over the other.

RBU Benefits

The high level difference is this: an 100 RBU test means that there will be 100 browsers in the cloud all loading your site and interacting with it, while a 100 VU test means that there will be 100 “threads” issuing HTTP GET and POST requests to your website, but not actually running a browser. This means that RBUs:

• Check functionality – they don’t just make HTTP requests, they validate if the application is actually working as an end-user would expect it to.
• Provide better error reporting – because there is an actual browser on a desktop running, we can take a screenshot to show you what the failure looked like.
• Are more realistic – today’s browsers load up to 15 objects at a time and have very unique HTTP request patterns that are almost impossible to simulate without a real browser.
• Use a wide range of IP addresses – because RBUs need a lot of resources, we allocate between 1 and 6 browsers per IP.
• Can measure real user wait time – sometimes the UI can’t be used until multiple overlapping AJAX calls are finished. Only a real browser can tell you when the UI is ready for interaction.

RBU Cost

When it comes to load testing website applications, RBUs are superior in almost every way. However, with all that realism and IP address distribution comes a massive hardware requirement: for every RBU in a test we allocate a full CPU core and 1.5GB+ of RAM. That means that a 100 RBU test requires at least 100 CPU cores and 150GB of RAM.

Compared to traditional load testing, which only requires half a CPU core or less and 256MB of RAM, it’s very different. This resource-hungry nature of RBUs means that they cost 10X more than our Virtual User service. That means a 100 RBU test will cost the same as a 1000 VU test. As such, depending on your budget, it may make sense to use a mixture of RBUs and VUs.

Scripting Effort

While RBUs are more realistic but cost a bit more, it also helps to understand the level of effort to write and maintain RBY and VU scripts. Remember that RBU scripts interact at the application level, while VU scripts interact at the protocol level.

Consider this RBU script:

var selenium = browserMob.openBrowser();
 
selenium.open("http://google.com/");
selenium.type("q", "BrowserMob");
selenium.click("btnG");
 
selenium.waitForPageToLoad(30000);
if (!selenium.isTextPresent("BrowserMob")) {
    throw "BrowserMob text not found";
}

Written as a VU script, each and every HTTP request would need to be scripted:

var c = browserMob.openHttpClient();
 
// get the home page and verify we got a 200 OK
c.get("http://www.google.com/", 200);
c.get("http://www.google.com/intl/en_ALL/images/logo.gif");
c.get("http://www.google.com/extern_js/f/.../Y3ut8jJ-OXk.js");
c.get("http://www.google.com/extern_chrome/be5ffa94030d2d34.js");
c.get("http://clients1.google.com/generate_204");
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://www.google.com/csi?v=3&s=webhp&action=&e=...&rt=...");
 
// simulate the AJAX event for the Google auto-complete
// also verify a 200 OK response
c.get("http://clients1.google.com/complete/search?hl=en&client=hp&q=BrowserMob&cp=27", 200);
 
// simulate searching
// also verify a 200 OK response
var resp = c.get("http://www.google.com/search?hl=en&source=hp&q=BrowserMob&btnG=...", 200);
 
// get the other requests caused by the new page load
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://id.google.com/verify/EAAAAECIWFsnXJixiQogbBCZlzA.gif");
c.get("http://www.google.com/csi?v=3&s=web&action=&ei=slYdS_-MGpXKsAPngsX8BA&e=...");
c.get("http://clients1.google.com/generate_204");
 
// finally, validate the browsermob was in the main HTTP request
if (resp.getBody().indexOf("BrowserMob") == -1) {
    throw "BrowserMob text not found";
}

When to Use Virtual Users

As you can imagine, writing and maintaining VU scripts is a lot more work. However, that doesn’t mean VUs are always a bad idea. In fact, they are great for:

• Placing a large amount of low-cost “dumb” load on a website. For example, when trying to saturate simple hits again a home page.
• When testing a web service, such as REST or SOAP, that can’t be directly tested with a web browser.
• When budget constraints require VUs

As a general rule of thumb, if your own hourly cost to write a VU script is more than it would be to quickly run an RBU test, you should probably go with an RBU. However, if it’s an extremely large scale test, VUs may be worth considering for a good portion of the load.

Don’t Forget!

Finally, don’t forget that there is no reason you can’t run two load tests at the same time:

• A smaller RBU test that lets you measure what the real user experience is like
• A larger VU test that hammers at the site in a cost-effective way

This technique allows you to stay within your budget without giving up entirely on the benefits of RBU scripts.

Tweet This Post 

RBU · VU

We get a lot of customer requests about Flash automation, Selenium, and BrowserMob. Because our load testing and website monitoring services uses real browsers, complete with Flash 10, we can do things no one else can do, like run a load test with hundreds of Flash runtime environments driving traffic on your site.

However, while our infrastructure allows for extremely unique Flash testing support, it’s not perfectly streamlined (yet). While we are hard at work on making Flash support for Selenium and BrowserMob significantly better, at this time some work is required by Selenium users before you can get started.

Specifically, the first question that must be answered is: what automation technique do I want to employ? There are two distinct ways to go:

• Native mouse & keyboard integration – good for situations where there is minimal Flash interaction required
• API-level integration – good for situations where there is a lot of Flash interaction required

Native mouse & keyboard integration

Let’s talk briefly about native integration. This is where a mouse movement, mouse click, or keyboard action is simulated at the operating system level. It reproduces the most realistic user simulation. However, it can be very brittle, as it has limited ways to “read” from the screen and validate that functionality worked correctly.

Presently there is no great Selenium-only solution here. There are commercial functional testing products such as eggPlant, but nothing on the load side of things. BrowserMob does have, however, the ability to interact with the operating system using native key events and coordinate-based mouse interaction.

This feature is perfect for those who want to fire off one or two simple interactions, such as clicking on a movie’s “play” button or interacting with a confirmation dialog. If this is all you need for your Flash or Flex integration, please contact our support team and we’ll gladly help you get your script working using this unique API.

API-level integration

For applications that have much more complex Flash/Flex user interfaces or applications that are 100% Flash-based, we don’t recommend native events as they can be brittle and difficult to confirm that the desired functionality worked. Instead, we recommend interacting with your Flash applications using API-level integration.

Important: to do this type of automation you must be able to recompile/modify the underlying Flash object(s). If you cannot do this and require full “black box” style automation, we suggest you consider native automation or you consult with the authors of the original Flash component.

How you proceed from here depends on whether you are using Flash or Flex. The reason is that Flex exposes some automation APIs that you will otherwise have to reproduce if you are using Flash-only.

Flash automation with ExternalInterface

If you are using low-level Flash, your automation options are a little more limited:

• Using ExternalInterface directly – no 3rd parties, but might requires the most programming
• Using FlashSelenium – an open source library to make ExternalInterface callbacks easier to call

We recommend you first start off with ExternalInterface directly, as FlashSelenium is really just a thin wrapper around ExternalInterface. While it’s convenient, we think doing it by yourself once is an important learning experience.

To get started, we recommend reading the ExternalInterface documentation by Adobe. The basic idea is that within your Flash component you execute this code:

ExternalInterface.addCallback("doSomething", doSomething);

This exposes the function doSomething such that it can be executed by JavaScript. Fortunately, because Selenium can execute JavaScript using the getEval, storeEval, assertEval, etc commands, we’re in luck:

var jsExpr = "window.document.getElementById('myFlashObjId').doSomething()";
var something = selenium.getEval(jsExpr);
if (something == 'blah') {
   // throw an error, we weren't expecting 'blah'!
}

Using this technique, you can begin to expose ActionScript functions that can be used to automate the Flash component and validate that the right behavior took place.

Flex automation with Selenium-Flex

If you’re using Flex, automation should be quite a bit easier. Thanks to Selenium-Flex, you don’t have to spend time exposing dozens of functions via ExternalInterface. Instead, just compile your Flex application using the Selenium-Flex library:

1. Copy the sfapi.swc file in to your project (ie: the lib directory)
2. In your IDE/build system, ensure that the library is included (ie: -include-libraries)

That’s it! Now the next time you build your Flex app it’ll automatically have a whole bunch of functions already created. Even better, you don’t have to do all that getEval mumbo-jumbo we did for Flash testing. Instead, thanks to another open source project called FlexUISelenium (which works in concert with Selenium-Flex), the getEval stuff is abstracted away so that your tests look like this:

flex.type("2").at("arg1");
flex.type("3").at("arg2");
flex.click("submit");

If you’re doing functional programming using Java or some other full-blown programming language, you can get started with FlexUISelenium today. If you’re looking to load test or do website monitoring with Selenium, we are in the middle of adding FlexUISelenium support to the BrowserMob API. We are looking for beta participants right now, so contact us if you’re interested.

A note about recording support

Currently, none of the open source solutions referenced here support recording interactions from within Selenium IDE. While not Selenium-compatible (yet), FlexMonkey is a very good open source Flex record and playback functional testing framework. We have plans to integrate Selenium and FlexMonkey in the future, including record support through Selenium IDE.

In the meantime, you will need to write each Selenium command by hand. This means that you’ll need to be aware of the IDs of all the elements you wish to interact with. Once you know those IDs, you can write the code that issues clicks or types on the component.

Tweet This Post 

Flash · Flex · Selenium

We get this question a lot: “Is there any way to change the hosts files on the machines that your browsers run from?” – or something similar.

The reason is always because there is a desire to monitor or test a site that doesn’t yet have the final DNS mappings put in place (ie: dev, staging, etc) and the development process has up until now worked by having team members edit their “hosts file”, a small DNS override that Windows, Mac OS X, and Linux all support.

This hosts file might look like this:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1           localhost
255.255.255.255     broadcasthost
::1                 localhost 
fe80::1%lo0         localhost
 
123.456.789.123     example.com

In here are defaults that the OS configures. This is where “localhost” is actually mapped to your local IP address: 127.0.0.1. And also in this example, the domain example.com is mapped to a fictitious IP address.

While BrowserMob doesn’t allow you to override the hosts file, it does let you accomplish the same goal of remapping a domain/host to a different IP than what is in the public DNS records. All you have to use is the HttpClient’s remapHost API.

This works for Virtual User scripts:

var c = browserMob.openHttpClient();
c.remapHost("example.com", "123.456.789.123");
...

As well as for Real Browser User scripts:

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
c.remapHost("example.com", "staging.example.com");
...

As you can also see in these examples, you can map to another host or an IP address.

Tweet This Post 

No tags

After running the load test customer’s often download the database and run queries to determine their sites performance. Here are some common SQL queries to get you started:

# Analyze the avg, max, and min time for each step in a transaction
SELECT step, COUNT(*), AVG(time_active), MAX(time_active), MIN(time_active) 
FROM step GROUP BY step
 
# General query to look at loading of objects
SELECT path, COUNT(*), AVG(time_active) 
FROM object GROUP BY path 
ORDER BY AVG(time_active) DESC
 
# Get error msg grouped by error, host, status
SELECT err_msg, host, status_code, COUNT(*), AVG(time_active) 
FROM object GROUP BY err_msg, host, status_code

We hope that you find these a useful starting point for your load testing analysis.

Tweet This Post 

No tags

If one is doing monitoring or load testing on a live site it is often beneficial to filter out the test browsers from Google Analytics.

When in the Google Analytics Filter section simply Create New Filter, then give it a name, choose the domain name option, and paste in amazonaws.com. That will ensure that our load testing service and monitoring from Washington DC and Dublin, Ireland will get filtered.

For Dallas, TX monitoring, do the same thing, but use the domain cloud-ips.com.

For San Francisco, CA monitoring, use the domain gogrid.com.

By adding these three filters, your monitoring and load testing with BrowserMob won’t affect your analytics reports.

Tweet This Post 

filter · Google Analytics

People often ask what are the differences between RBUs and VUs. Obviously RBUs are more realistic while VUs are the perfect vehicle for testing APIs, but here is a table giving you a breakdown.

Some things to take note of:

• There is a difference in the difficulty for scripting RBUs vs. VUs. RBU scripts are easier to create with the Selenium IDE tool and have validation built in. Because RBUs mimic human behavior the
  scripts include built-in validation concepts. For example an error will be thrown if you try to click on an element that doesn’t exist because the page being displayed is a “server too busy” page instead of the expected page. With VUs, you have to code in your own validation logic.
• VUs are the preferred way to test SOAP/REST and other “non-browser” HTTP requests.
• Often the right approach involves using a bit of both RBUs and VUs. The VUs can be used to lay down base load on “read-only” pages that don’t involve form interaction or AJAX. The 80/20 rule applies here, you can generate 80% of the read-only traffic with VUs and use RBUs for the remaining 20%.

We hope this give you a better understanding of the differences and relative strengths of RBUs and VUs.

Tweet This Post 

No tags