For load tests, yes (we’re working to add support for this to monitoring). You can use BrowserMob API to add any name/value pair to your transactions or steps. Here’s a simple example:

var selenium = browserMob.openBrowser();
 
var tx = browserMob.beginTransaction();
 
var step = browserMob.beginStep("Home Page");
 
selenium.open("http://example.com");
 
// record how long it took for some text to appear
var start = new Date().getTime();
selenium.waitForTextPresent("dynamic text");
var end = new Date().getTime();
 
// get a special cookie out and store it
var cookie = selenium.getCookieByName("myCookie");
tx.put("cookie_value", cookie);
 
// and store the "text" timing, associated with the step
step.put("text_timing", end - start);
 
browserMob.endStep();
 
browserMob.endTransaction();

The important things to notice here are that both beginTransaction() and beginStep() actually return objects that represent the transaction or step, respectively. You can then call put() on those objects to associate arbitrary data, such as custom timings or cookie values, with them.

Then when the transaction is stored in the load test database, you’ll be able to look them up in the name_value_pairs table. For more information on the test database schema, consult the documentation.

Tweet This Post 

API

A couple days ago we announced a brand new pricing system that lowered our pricing across the board. For our existing customers, this pricing is quite a bit different than what they were used to, so we wanted to put together a list Frequently Asked Questions about the new plans.

Is this new pricing more or less expensive?

By any metric, this a price reduction. We’re extremely excited to have found a model that controls costs for our customers while actually making the revenue flow for our company healthier.

Not only did we reduce the overall pricing for monitoring and/or give a better value for the price, we created a new way to buy load testing from us that is significantly less expensive for most projects.

But even if you stick with the old model of paying for load tests, we’ve reduced the price because the first 25 RBUs or 100 VUs are always sold “at cost”. This means that a 100 RBU test that used to cost $300 now costs $15 under the new plan and $228.75 when using the old plan. To understand how you can choose between either plan, please see the Can I still use my old pay-as-you-go plan? question.

How has pricing for monitoring changed?

The old pricing model for monitoring used a complex formula in which the time of the script being used to monitor your site heavily influenced the overall cost. For example, a job that averaged 3 seconds per run might cost half of that of a script that took 7 seconds to run. The problem with this model was that if your site slowed down, your costs would go up, making your monitoring budget unpredictable.

The new pricing model eliminates that unpredictability. Instead, there is a constant number of monthly page views that you’ll used, based entirely on your script, number of locations you check from, and frequency at which you check. No matter how fast or how slow your site is, the price will remain constant.

For example, if you wanted to test a script that hit your home page, visited the login page by clicking the login link, and then logged in, that would be 3 page views. If your monitoring job ran that script every 15 minutes from two locations, you’d use 17,280 page views per month:

17,280 = 3 page views x 60 mins / 15 min freq. 
           x 24 hours x 30 days x 2 locations

Our pricing packages start at 3,000 monthly page views per month (FREE) and go all the way up to 350K monthly page views.

What happens to my current monitoring jobs?

If you were paying for a monthly, recurring plan, then you have already been converted over to the new model and are now paying less than you were before. That’s because we’ve lowered our pricing for all monitoring packages. For example, what was $300/mo is now $249/mo, and what was $175/mo is now $149/mo.

However, if you were not using a monthly credit package to pay for your monitoring, you may need to take some action. Your monitoring jobs have not been disabled, but they are currently running free of charge. Over the next few weeks, we will be contacting each customer who is in this situation and work with them on finding a monitoring package that fits their needs.

In some cases, a customer may want to transfer credits previously purchased and apply them toward a monitoring job. We are more than happy to convert purchases over this way and will work with each customer that is interested in doing this.

How has pricing for load testing changed?

In our previous pricing model, each concurrent RBU cost 10 credits/hour while each concurrent VU cost 1 credit/hour. Customers could buy credits at various rates ranging from 30 cents/credit down to 7.5 cents/credit, depending on the size of the purchase. Purchases of $5,000 or larger were eligible to purchase additional credits of any amount at the original price (usually 7.5 or 10 cents).

Under the new model, there are no more credits. Rather, we have converted them all to “cloud dollars” by multiplying the remaining credits in your account by the price paid for the credits. For example, if you purchased 1000 credits for $300 and had 300 at the time of the switch, you would be left with $90 in cloud dollars.

These cloud dollars can be used to pay for load tests. The price for each load test will be quoted at the bottom of the “schedule load test” page. If the test is larger than the load test package that you currently have, we will quote a higher price that is based on the old pay-as-you-go pricing plan.

How do “cloud dollars” compare to credits?

If you’re used to the old credit system, the easiest way to think of cloud dollars is that they reflect a credit price of just 1.5 cents – a 20X reduction from the 30 cent price most of our customers were using.

This means that a 100 RBU test that used to cost $300 now costs only $15 when you are subscribed to a qualifying load test package. For customers who purchased a larger credit package and therefore bought in to a smaller credit price (ie: 15 or 10 cents/credit), the savings are not as large but are still very significant (ie: 10X or 6.67X, respectively.)

Can I still use my old pay-as-you-go plan?

Absolutely.

Simply keep buying cloud dollars just like if they were the old credit system. When you schedule a test, you’ll be quoted a price to run the test. You may be presented with a note explaining that the per-test fee would be lower if you were using a load test package, but you are under no obligation to sign up for such a package. This is especially ideal for anyone who does infrequent load testing.

The price of test will be based on what your original credit price was. This means that a 100 RBU test might cost $228.75 for one customer at the 30 cent price point and $78.75 for another customer at the 10 cent price point. However, both would only pay $15 if they were subscribed to a load test package.

Can I upgrade to the new pricing?

Yes. We can apply any unused cloud dollars towards load testing or monitoring packages. Just contact us and we can get the process started.

What is the threshold for the free BrowserMob services?

For monitoring, you can do up to 3,000 page views per month for free.

For load testing, you can run up to 25 RBUs and 100 VUs at any time without paying for a load testing package. However, you will have to pay the raw cloud fees for each test. These fees are 15 cents per RBU hour and 1.5 cents per VU hour (ie: 25 RBUs = $3.75 and 100 VUs = $1.50). Each account comes with 10 cloud dollars to get you started, but you can purchase more at any time.

Tweet This Post 

Load Testing · Monitoring · Pricing

At BrowserMob we have two different levels of load testing support: one for Real Browser Users (RBUs) and one for Virtual Users (VUs). Often it’s not immediately clear what the difference is and why you’d use one over the other.

RBU Benefits

The high level difference is this: an 100 RBU test means that there will be 100 browsers in the cloud all loading your site and interacting with it, while a 100 VU test means that there will be 100 “threads” issuing HTTP GET and POST requests to your website, but not actually running a browser. This means that RBUs:

• Check functionality – they don’t just make HTTP requests, they validate if the application is actually working as an end-user would expect it to.
• Provide better error reporting – because there is an actual browser on a desktop running, we can take a screenshot to show you what the failure looked like.
• Are more realistic – today’s browsers load up to 15 objects at a time and have very unique HTTP request patterns that are almost impossible to simulate without a real browser.
• Use a wide range of IP addresses – because RBUs need a lot of resources, we allocate between 1 and 6 browsers per IP.
• Can measure real user wait time – sometimes the UI can’t be used until multiple overlapping AJAX calls are finished. Only a real browser can tell you when the UI is ready for interaction.

RBU Cost

When it comes to load testing website applications, RBUs are superior in almost every way. However, with all that realism and IP address distribution comes a massive hardware requirement: for every RBU in a test we allocate a full CPU core and 1.5GB+ of RAM. That means that a 100 RBU test requires at least 100 CPU cores and 150GB of RAM.

Compared to traditional load testing, which only requires half a CPU core or less and 256MB of RAM, it’s very different. This resource-hungry nature of RBUs means that they cost 10X more than our Virtual User service. That means a 100 RBU test will cost the same as a 1000 VU test. As such, depending on your budget, it may make sense to use a mixture of RBUs and VUs.

Scripting Effort

While RBUs are more realistic but cost a bit more, it also helps to understand the level of effort to write and maintain RBY and VU scripts. Remember that RBU scripts interact at the application level, while VU scripts interact at the protocol level.

Consider this RBU script:

var selenium = browserMob.openBrowser();
 
selenium.open("http://google.com/");
selenium.type("q", "BrowserMob");
selenium.click("btnG");
 
selenium.waitForPageToLoad(30000);
if (!selenium.isTextPresent("BrowserMob")) {
    throw "BrowserMob text not found";
}

Written as a VU script, each and every HTTP request would need to be scripted:

var c = browserMob.openHttpClient();
 
// get the home page and verify we got a 200 OK
c.get("http://www.google.com/", 200);
c.get("http://www.google.com/intl/en_ALL/images/logo.gif");
c.get("http://www.google.com/extern_js/f/.../Y3ut8jJ-OXk.js");
c.get("http://www.google.com/extern_chrome/be5ffa94030d2d34.js");
c.get("http://clients1.google.com/generate_204");
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://www.google.com/csi?v=3&s=webhp&action=&e=...&rt=...");
 
// simulate the AJAX event for the Google auto-complete
// also verify a 200 OK response
c.get("http://clients1.google.com/complete/search?hl=en&client=hp&q=BrowserMob&cp=27", 200);
 
// simulate searching
// also verify a 200 OK response
var resp = c.get("http://www.google.com/search?hl=en&source=hp&q=BrowserMob&btnG=...", 200);
 
// get the other requests caused by the new page load
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://id.google.com/verify/EAAAAECIWFsnXJixiQogbBCZlzA.gif");
c.get("http://www.google.com/csi?v=3&s=web&action=&ei=slYdS_-MGpXKsAPngsX8BA&e=...");
c.get("http://clients1.google.com/generate_204");
 
// finally, validate the browsermob was in the main HTTP request
if (resp.getBody().indexOf("BrowserMob") == -1) {
    throw "BrowserMob text not found";
}

When to Use Virtual Users

As you can imagine, writing and maintaining VU scripts is a lot more work. However, that doesn’t mean VUs are always a bad idea. In fact, they are great for:

• Placing a large amount of low-cost “dumb” load on a website. For example, when trying to saturate simple hits again a home page.
• When testing a web service, such as REST or SOAP, that can’t be directly tested with a web browser.
• When budget constraints require VUs

As a general rule of thumb, if your own hourly cost to write a VU script is more than it would be to quickly run an RBU test, you should probably go with an RBU. However, if it’s an extremely large scale test, VUs may be worth considering for a good portion of the load.

Don’t Forget!

Finally, don’t forget that there is no reason you can’t run two load tests at the same time:

• A smaller RBU test that lets you measure what the real user experience is like
• A larger VU test that hammers at the site in a cost-effective way

This technique allows you to stay within your budget without giving up entirely on the benefits of RBU scripts.

Tweet This Post 

RBU · VU

We get a lot of customer requests about Flash automation, Selenium, and BrowserMob. Because our load testing and website monitoring services uses real browsers, complete with Flash 10, we can do things no one else can do, like run a load test with hundreds of Flash runtime environments driving traffic on your site.

However, while our infrastructure allows for extremely unique Flash testing support, it’s not perfectly streamlined (yet). While we are hard at work on making Flash support for Selenium and BrowserMob significantly better, at this time some work is required by Selenium users before you can get started.

Specifically, the first question that must be answered is: what automation technique do I want to employ? There are two distinct ways to go:

• Native mouse & keyboard integration – good for situations where there is minimal Flash interaction required
• API-level integration – good for situations where there is a lot of Flash interaction required

Native mouse & keyboard integration

Let’s talk briefly about native integration. This is where a mouse movement, mouse click, or keyboard action is simulated at the operating system level. It reproduces the most realistic user simulation. However, it can be very brittle, as it has limited ways to “read” from the screen and validate that functionality worked correctly.

Presently there is no great Selenium-only solution here. There are commercial functional testing products such as eggPlant, but nothing on the load side of things. BrowserMob does have, however, the ability to interact with the operating system using native key events and coordinate-based mouse interaction.

This feature is perfect for those who want to fire off one or two simple interactions, such as clicking on a movie’s “play” button or interacting with a confirmation dialog. If this is all you need for your Flash or Flex integration, please contact our support team and we’ll gladly help you get your script working using this unique API.

API-level integration

For applications that have much more complex Flash/Flex user interfaces or applications that are 100% Flash-based, we don’t recommend native events as they can be brittle and difficult to confirm that the desired functionality worked. Instead, we recommend interacting with your Flash applications using API-level integration.

Important: to do this type of automation you must be able to recompile/modify the underlying Flash object(s). If you cannot do this and require full “black box” style automation, we suggest you consider native automation or you consult with the authors of the original Flash component.

How you proceed from here depends on whether you are using Flash or Flex. The reason is that Flex exposes some automation APIs that you will otherwise have to reproduce if you are using Flash-only.

Flash automation with ExternalInterface

If you are using low-level Flash, your automation options are a little more limited:

• Using ExternalInterface directly – no 3rd parties, but might requires the most programming
• Using FlashSelenium – an open source library to make ExternalInterface callbacks easier to call

We recommend you first start off with ExternalInterface directly, as FlashSelenium is really just a thin wrapper around ExternalInterface. While it’s convenient, we think doing it by yourself once is an important learning experience.

To get started, we recommend reading the ExternalInterface documentation by Adobe. The basic idea is that within your Flash component you execute this code:

ExternalInterface.addCallback("doSomething", doSomething);

This exposes the function doSomething such that it can be executed by JavaScript. Fortunately, because Selenium can execute JavaScript using the getEval, storeEval, assertEval, etc commands, we’re in luck:

var jsExpr = "window.document.getElementById('myFlashObjId').doSomething()";
var something = selenium.getEval(jsExpr);
if (something == 'blah') {
   // throw an error, we weren't expecting 'blah'!
}

Using this technique, you can begin to expose ActionScript functions that can be used to automate the Flash component and validate that the right behavior took place.

Flex automation with Selenium-Flex

If you’re using Flex, automation should be quite a bit easier. Thanks to Selenium-Flex, you don’t have to spend time exposing dozens of functions via ExternalInterface. Instead, just compile your Flex application using the Selenium-Flex library:

1. Copy the sfapi.swc file in to your project (ie: the lib directory)
2. In your IDE/build system, ensure that the library is included (ie: -include-libraries)

That’s it! Now the next time you build your Flex app it’ll automatically have a whole bunch of functions already created. Even better, you don’t have to do all that getEval mumbo-jumbo we did for Flash testing. Instead, thanks to another open source project called FlexUISelenium (which works in concert with Selenium-Flex), the getEval stuff is abstracted away so that your tests look like this:

flex.type("2").at("arg1");
flex.type("3").at("arg2");
flex.click("submit");

If you’re doing functional programming using Java or some other full-blown programming language, you can get started with FlexUISelenium today. If you’re looking to load test or do website monitoring with Selenium, we are in the middle of adding FlexUISelenium support to the BrowserMob API. We are looking for beta participants right now, so contact us if you’re interested.

A note about recording support

Currently, none of the open source solutions referenced here support recording interactions from within Selenium IDE. While not Selenium-compatible (yet), FlexMonkey is a very good open source Flex record and playback functional testing framework. We have plans to integrate Selenium and FlexMonkey in the future, including record support through Selenium IDE.

In the meantime, you will need to write each Selenium command by hand. This means that you’ll need to be aware of the IDs of all the elements you wish to interact with. Once you know those IDs, you can write the code that issues clicks or types on the component.

Tweet This Post 

Flash · Flex · Selenium

We get this question a lot: “Is there any way to change the hosts files on the machines that your browsers run from?” – or something similar.

The reason is always because there is a desire to monitor or test a site that doesn’t yet have the final DNS mappings put in place (ie: dev, staging, etc) and the development process has up until now worked by having team members edit their “hosts file”, a small DNS override that Windows, Mac OS X, and Linux all support.

This hosts file might look like this:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1           localhost
255.255.255.255     broadcasthost
::1                 localhost 
fe80::1%lo0         localhost
 
123.456.789.123     example.com

In here are defaults that the OS configures. This is where “localhost” is actually mapped to your local IP address: 127.0.0.1. And also in this example, the domain example.com is mapped to a fictitious IP address.

While BrowserMob doesn’t allow you to override the hosts file, it does let you accomplish the same goal of remapping a domain/host to a different IP than what is in the public DNS records. All you have to use is the HttpClient’s remapHost API.

This works for Virtual User scripts:

var c = browserMob.openHttpClient();
c.remapHost("example.com", "123.456.789.123");
...

As well as for Real Browser User scripts:

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
c.remapHost("example.com", "staging.example.com");
...

As you can also see in these examples, you can map to another host or an IP address.

Tweet This Post 

No tags

After running the load test customer’s often download the database and run queries to determine their sites performance. Here are some common SQL queries to get you started:

# Analyze the avg, max, and min time for each step in a transaction
SELECT step, COUNT(*), AVG(time_active), MAX(time_active), MIN(time_active) 
FROM step GROUP BY step
 
# General query to look at loading of objects
SELECT path, COUNT(*), AVG(time_active) 
FROM object GROUP BY path 
ORDER BY AVG(time_active) DESC
 
# Get error msg grouped by error, host, status
SELECT err_msg, host, status_code, COUNT(*), AVG(time_active) 
FROM object GROUP BY err_msg, host, status_code

We hope that you find these a useful starting point for your load testing analysis.

Tweet This Post 

No tags

If one is doing monitoring or load testing on a live site it is often beneficial to filter out the test browsers from Google Analytics.

When in the Google Analytics Filter section simply Create New Filter, then give it a name, choose the domain name option, and paste in amazonaws.com. That will ensure that our load testing service and monitoring from Washington DC and Dublin, Ireland will get filtered.

For Dallas, TX monitoring, do the same thing, but use the domain cloud-ips.com.

For San Francisco, CA monitoring, use the domain gogrid.com.

By adding these three filters, your monitoring and load testing with BrowserMob won’t affect your analytics reports.

Tweet This Post 

filter · Google Analytics

People often ask what are the differences between RBUs and VUs. Obviously RBUs are more realistic while VUs are the perfect vehicle for testing APIs, but here is a table giving you a breakdown.

Some things to take note of:

• There is a difference in the difficulty for scripting RBUs vs. VUs. RBU scripts are easier to create with the Selenium IDE tool and have validation built in. Because RBUs mimic human behavior the
  scripts include built-in validation concepts. For example an error will be thrown if you try to click on an element that doesn’t exist because the page being displayed is a “server too busy” page instead of the expected page. With VUs, you have to code in your own validation logic.
• VUs are the preferred way to test SOAP/REST and other “non-browser” HTTP requests.
• Often the right approach involves using a bit of both RBUs and VUs. The VUs can be used to lay down base load on “read-only” pages that don’t involve form interaction or AJAX. The 80/20 rule applies here, you can generate 80% of the read-only traffic with VUs and use RBUs for the remaining 20%.

We hope this give you a better understanding of the differences and relative strengths of RBUs and VUs.

Tweet This Post 

No tags

We recently wrote a post about how to randomize an IP address and tie it to the domain name for RBU scripts. Here is how it is done for VU scripts.


var ips = ['169.125.57.119', '169.230.103.88', '169.125.17.0', '169.125.17.68', '79.125.17.80', '169.125.38.66', '169.125.43.112'];
var ip = ips[Math.floor(Math.random() * ips.length)];
browserMob.log("Using ip " + ip);
var c = browserMob.openHttpClient()// useful function that rewrites the hosts and auto checks the response code
var get = function(url) {

var rewrittenHost = null;
if (url.indexOf('http://www.aktiesport.nl/') == 0) {
rewrittenHost = 'www.aktiesport.nl';
url = 'http://' + ip + '/' + url.substring('http://www.aktiesport.nl/'.length);
}
// create the request
var req = c.newGet(url);
// Leave validation text blank. This let us avoid specific validation
// but still get BrowserMob to capture the content if there is an error
req.setVerificationText('');
if (rewrittenHost != null) {
req.addRequestHeader("Host", rewrittenHost);
}
// make the request
var resp = req.execute();
// and check the response code
var code = resp.getInfo().getStatusCode();
if (code != 200) {
throw "Expected 200 but got " + code;
}

}
browserMob.beginTransaction()

You can now write the rest of the script as normal simply using mysite.com and ignoring any IP address issues. This is quite useful for running tests when your loadbalancer isn’t in place yet.

Tweet This Post 

No tags

We recently had a customer that wanted to test a website that would be load balanced, but the server wasn’t installed yet. To get around this we randomly chose an IP address and then remapped it to the host. Insert this code at the top of your RBU Script and itwill randomly select one of the IPs each time it is run, but
still act as if it were going through the standard mysite.com host.

var ips = ['169.125.57.119','169.230.103.88','169.125.17.0','169.125.17.68',

'79.125.17.80','169.125.38.66','169.125.43.112'];

Tweet This Post 

No tags