Often when it’s time to run a load test or turn on website monitoring, you don’t necessarily want the transaction hitting all your third party components on the page.

For example, you don’t want your analytics software to record the visits as real visitors, since that would skew your marketing metrics. Likewise, you don’t necessarily want advertisements served up, especially if the ad vendor uses “click-through rates” (CTR) to optimize ad prices and a load test would artificially drive down the CTR.

While there are vendor-specific actions you can take to deal with this problem (ie: we cover Google Analytics here), sometimes the easiest solution is to modify your script so that the request is never made in the first place.

You can do this for both Real Browser User (RBU) and Virtual User (VU) scripts using some new APIs that we recently rolled out. They allow you to whitelist or blacklist certain hosts from having actual HTTP requests sent to them.

For example, suppose you want to exclude all requests to www.google-analytics.com from your website monitoring or website load test. Your script would need to do something like this:

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
c.blacklistRequests("http://www\\.google-analytics\\.com/.*", 
                    200);
 
browserMob.beginTransaction();
browserMob.beginStep("Step 1");
 
// rest of test...

This would ensure that any HTTP request to Google Analytics would be bypassed immediately and a fake 200 response code would be returned instead.

Sometimes it’s easier to whitelist “good” hosts instead of blacklisting specific ones. This is often the case for advertising networks, which tend to host content from dozens of different domains.

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
c.whitelistRequests(["http://www\\.example\\.com/.*", 
                     "http://images\\.example.com\\.com/.*"], 
                    200);
 
browserMob.beginTransaction();
browserMob.beginStep("Step 1");
 
// rest of test...

In this example, we are allowing all requests to www.example.com and images.example.com to go through, but faking any other request with an immediate 200 response code.

These whitelist and blacklist APIs are just one example of the powerful scripting you can do using the BrowserMob API and scripting environment. We encourage you to review the entire API documentation from time to time, as well as contact us if you ever have questions about them.

Tweet This Post 

3rd party widgets · API · Google Analytics

Often when someone wants to test a site that isn’t yet ready for public release, they will use various security measures to keep the general public from looking at it. Sometimes they will use BASIC authentication and other times they will use firewall rules to only allow certain IP addresses access to the site.

However, as we continue to see a growing trend of BrowserMob customers who are deploying websites in the Amazon cloud using EC2, we wanted to highlight that the firewall technique does not work using our service.

The reason is that BrowserMob also runs in the Amazon cloud and as such is able to access the customer website across it’s private network. Because of this, the incoming IP address is not one of the ones we have documented, but a random internal-only IP that we do not know about ahead of time.

Fortunately, there is an easy solution: set up your EC2 network security to allow our Amazon “user ID” direct access through the port your website is hosted on. Simply authorize the User ID 581518868128 in the appropriate EC2 Security Group.

For example, If you are using Elasticfox, go to the Security tab and add a new permission for the appropriate Security Group. It should look like this:

Once done, you should be able to run your load test just fine. We do recommend running a small scale test first to confirm it works before running the larger test. If it’s not working, every request will result in a timeout.

We hope that helps, and we look forward to continuing to work with more companies making the transition to the cloud!

Tweet This Post 

EC2 · Security

Tom Pinckney, co-founder of Hunch, recently prescribed some tips on how to survive being Slashdotted. While Slashdot is no longer the biggest online driver of spikes of traffic, there was a time when it was the king of bringing down sites: a simple mention of a URL on the home page would crush most servers.

These days sites like Yahoo, Digg, and Drudge Report often carry much more weird, but Tom’s lessons apply to any situation where you are going to get a huge spike in traffic (expected or otherwise). There’s a lot of good stuff in Tom’s post, as well as previous one about achieving “consistent performance” that are well worth a read.

Perhaps the best advice Tom gives is:

If using Apache, use worker MPM with MaxClients and ServerLimit as high as you have RAM for. Better to cap max # of threads and reject some users than let the whole machine go down.

This is an important point people often forget: eventually you will fail, so better plan to fail gracefully.

Tweet This Post 

Digg · Slashdot

BASIC authentication is used to provide minimal, low-security protection from anonymous visitors hitting your website. It is frequently used by companies to ensure that their staging or development environments are not accessibly by the general public prior to pushing the changes to production. Typical authentication dialog prompts look like so:

The challenge here is that this dialog box is the kind of dialog that Selenium cannot automate. You cannot issue “type” or “click” commands on it. In fact, if this box comes up, your script is guaranteed to time out because Selenium will continue to wait for the page to load, not realizing a login is required and unable to populate it.

Traditionally Selenium users have worked around this problem by using a URL pattern in which the username and password was encoded:

http://username:password@example.com

Recently, however, this is not working across all browsers. IE no longer supports it and recent versions of Firefox add a confirmation box that breaks the automation with a slightly different popup. And while we’re working on as solution for Selenium 2.0, there aren’t a lot of options for Selenium today.

But when it comes to BrowserMob, we do have a solution available today, which will be soon back-porting to the Selenium 2 codebase. Simply edit your script to look like the following:

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
c.autoBasicAuthorization("example.com", "username", "password");
 
browserMob.beginTransaction();
browserMob.beginStep("Step 1");
 
selenium.open("http://example.com");
// rest of script...

This code tells BrowserMob to automatically put in the required Authorization headers for any HTTP request issued to example.com. And soon this capability will be added to Selenium 2 (after we get the first alpha out in time for the holidays).

Tweet This Post 

Basic Authentication · Scripting · Selenium 2

At BrowserMob we have two different levels of load testing support: one for Real Browser Users (RBUs) and one for Virtual Users (VUs). Often it’s not immediately clear what the difference is and why you’d use one over the other.

RBU Benefits

The high level difference is this: an 100 RBU test means that there will be 100 browsers in the cloud all loading your site and interacting with it, while a 100 VU test means that there will be 100 “threads” issuing HTTP GET and POST requests to your website, but not actually running a browser. This means that RBUs:

• Check functionality – they don’t just make HTTP requests, they validate if the application is actually working as an end-user would expect it to.
• Provide better error reporting – because there is an actual browser on a desktop running, we can take a screenshot to show you what the failure looked like.
• Are more realistic – today’s browsers load up to 15 objects at a time and have very unique HTTP request patterns that are almost impossible to simulate without a real browser.
• Use a wide range of IP addresses – because RBUs need a lot of resources, we allocate between 1 and 6 browsers per IP.
• Can measure real user wait time – sometimes the UI can’t be used until multiple overlapping AJAX calls are finished. Only a real browser can tell you when the UI is ready for interaction.

RBU Cost

When it comes to load testing website applications, RBUs are superior in almost every way. However, with all that realism and IP address distribution comes a massive hardware requirement: for every RBU in a test we allocate a full CPU core and 1.5GB+ of RAM. That means that a 100 RBU test requires at least 100 CPU cores and 150GB of RAM.

Compared to traditional load testing, which only requires half a CPU core or less and 256MB of RAM, it’s very different. This resource-hungry nature of RBUs means that they cost 10X more than our Virtual User service. That means a 100 RBU test will cost the same as a 1000 VU test. As such, depending on your budget, it may make sense to use a mixture of RBUs and VUs.

Scripting Effort

While RBUs are more realistic but cost a bit more, it also helps to understand the level of effort to write and maintain RBY and VU scripts. Remember that RBU scripts interact at the application level, while VU scripts interact at the protocol level.

Consider this RBU script:

var selenium = browserMob.openBrowser();
 
selenium.open("http://google.com/");
selenium.type("q", "BrowserMob");
selenium.click("btnG");
 
selenium.waitForPageToLoad(30000);
if (!selenium.isTextPresent("BrowserMob")) {
    throw "BrowserMob text not found";
}

Written as a VU script, each and every HTTP request would need to be scripted:

var c = browserMob.openHttpClient();
 
// get the home page and verify we got a 200 OK
c.get("http://www.google.com/", 200);
c.get("http://www.google.com/intl/en_ALL/images/logo.gif");
c.get("http://www.google.com/extern_js/f/.../Y3ut8jJ-OXk.js");
c.get("http://www.google.com/extern_chrome/be5ffa94030d2d34.js");
c.get("http://clients1.google.com/generate_204");
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://www.google.com/csi?v=3&s=webhp&action=&e=...&rt=...");
 
// simulate the AJAX event for the Google auto-complete
// also verify a 200 OK response
c.get("http://clients1.google.com/complete/search?hl=en&client=hp&q=BrowserMob&cp=27", 200);
 
// simulate searching
// also verify a 200 OK response
var resp = c.get("http://www.google.com/search?hl=en&source=hp&q=BrowserMob&btnG=...", 200);
 
// get the other requests caused by the new page load
c.get("http://www.google.com/images/nav_logo7.png");
c.get("http://id.google.com/verify/EAAAAECIWFsnXJixiQogbBCZlzA.gif");
c.get("http://www.google.com/csi?v=3&s=web&action=&ei=slYdS_-MGpXKsAPngsX8BA&e=...");
c.get("http://clients1.google.com/generate_204");
 
// finally, validate the browsermob was in the main HTTP request
if (resp.getBody().indexOf("BrowserMob") == -1) {
    throw "BrowserMob text not found";
}

When to Use Virtual Users

As you can imagine, writing and maintaining VU scripts is a lot more work. However, that doesn’t mean VUs are always a bad idea. In fact, they are great for:

• Placing a large amount of low-cost “dumb” load on a website. For example, when trying to saturate simple hits again a home page.
• When testing a web service, such as REST or SOAP, that can’t be directly tested with a web browser.
• When budget constraints require VUs

As a general rule of thumb, if your own hourly cost to write a VU script is more than it would be to quickly run an RBU test, you should probably go with an RBU. However, if it’s an extremely large scale test, VUs may be worth considering for a good portion of the load.

Don’t Forget!

Finally, don’t forget that there is no reason you can’t run two load tests at the same time:

• A smaller RBU test that lets you measure what the real user experience is like
• A larger VU test that hammers at the site in a cost-effective way

This technique allows you to stay within your budget without giving up entirely on the benefits of RBU scripts.

Tweet This Post 

RBU · VU

We get a lot of customer requests about Flash automation, Selenium, and BrowserMob. Because our load testing and website monitoring services uses real browsers, complete with Flash 10, we can do things no one else can do, like run a load test with hundreds of Flash runtime environments driving traffic on your site.

However, while our infrastructure allows for extremely unique Flash testing support, it’s not perfectly streamlined (yet). While we are hard at work on making Flash support for Selenium and BrowserMob significantly better, at this time some work is required by Selenium users before you can get started.

Specifically, the first question that must be answered is: what automation technique do I want to employ? There are two distinct ways to go:

• Native mouse & keyboard integration – good for situations where there is minimal Flash interaction required
• API-level integration – good for situations where there is a lot of Flash interaction required

Native mouse & keyboard integration

Let’s talk briefly about native integration. This is where a mouse movement, mouse click, or keyboard action is simulated at the operating system level. It reproduces the most realistic user simulation. However, it can be very brittle, as it has limited ways to “read” from the screen and validate that functionality worked correctly.

Presently there is no great Selenium-only solution here. There are commercial functional testing products such as eggPlant, but nothing on the load side of things. BrowserMob does have, however, the ability to interact with the operating system using native key events and coordinate-based mouse interaction.

This feature is perfect for those who want to fire off one or two simple interactions, such as clicking on a movie’s “play” button or interacting with a confirmation dialog. If this is all you need for your Flash or Flex integration, please contact our support team and we’ll gladly help you get your script working using this unique API.

API-level integration

For applications that have much more complex Flash/Flex user interfaces or applications that are 100% Flash-based, we don’t recommend native events as they can be brittle and difficult to confirm that the desired functionality worked. Instead, we recommend interacting with your Flash applications using API-level integration.

Important: to do this type of automation you must be able to recompile/modify the underlying Flash object(s). If you cannot do this and require full “black box” style automation, we suggest you consider native automation or you consult with the authors of the original Flash component.

How you proceed from here depends on whether you are using Flash or Flex. The reason is that Flex exposes some automation APIs that you will otherwise have to reproduce if you are using Flash-only.

Flash automation with ExternalInterface

If you are using low-level Flash, your automation options are a little more limited:

• Using ExternalInterface directly – no 3rd parties, but might requires the most programming
• Using FlashSelenium – an open source library to make ExternalInterface callbacks easier to call

We recommend you first start off with ExternalInterface directly, as FlashSelenium is really just a thin wrapper around ExternalInterface. While it’s convenient, we think doing it by yourself once is an important learning experience.

To get started, we recommend reading the ExternalInterface documentation by Adobe. The basic idea is that within your Flash component you execute this code:

ExternalInterface.addCallback("doSomething", doSomething);

This exposes the function doSomething such that it can be executed by JavaScript. Fortunately, because Selenium can execute JavaScript using the getEval, storeEval, assertEval, etc commands, we’re in luck:

var jsExpr = "window.document.getElementById('myFlashObjId').doSomething()";
var something = selenium.getEval(jsExpr);
if (something == 'blah') {
   // throw an error, we weren't expecting 'blah'!
}

Using this technique, you can begin to expose ActionScript functions that can be used to automate the Flash component and validate that the right behavior took place.

Flex automation with Selenium-Flex

If you’re using Flex, automation should be quite a bit easier. Thanks to Selenium-Flex, you don’t have to spend time exposing dozens of functions via ExternalInterface. Instead, just compile your Flex application using the Selenium-Flex library:

1. Copy the sfapi.swc file in to your project (ie: the lib directory)
2. In your IDE/build system, ensure that the library is included (ie: -include-libraries)

That’s it! Now the next time you build your Flex app it’ll automatically have a whole bunch of functions already created. Even better, you don’t have to do all that getEval mumbo-jumbo we did for Flash testing. Instead, thanks to another open source project called FlexUISelenium (which works in concert with Selenium-Flex), the getEval stuff is abstracted away so that your tests look like this:

flex.type("2").at("arg1");
flex.type("3").at("arg2");
flex.click("submit");

If you’re doing functional programming using Java or some other full-blown programming language, you can get started with FlexUISelenium today. If you’re looking to load test or do website monitoring with Selenium, we are in the middle of adding FlexUISelenium support to the BrowserMob API. We are looking for beta participants right now, so contact us if you’re interested.

A note about recording support

Currently, none of the open source solutions referenced here support recording interactions from within Selenium IDE. While not Selenium-compatible (yet), FlexMonkey is a very good open source Flex record and playback functional testing framework. We have plans to integrate Selenium and FlexMonkey in the future, including record support through Selenium IDE.

In the meantime, you will need to write each Selenium command by hand. This means that you’ll need to be aware of the IDs of all the elements you wish to interact with. Once you know those IDs, you can write the code that issues clicks or types on the component.

Tweet This Post 

Flash · Flex · Selenium

Like many in the industry, we were surprised and intrigued by the announcement yesterday that Google would be entering the DNS business. The basic logic was clear: Google has a vested interest in the internet being fast, and so they want to offer a free public utility to help it be faster.

Of course, some were doubtful. OpenDNS, probably the company that has the most to lose by this decision, responded quickly. Some questioned its security, while others pointed out that Google gains a lot more than you might think by serving DNS: it would now know everywhere you were going, regardless of whether you went through Google Search or whether the site had Google Analytics installed.

While we’re not going to get in to the broader debate of whether this move is Good or Evil, we were curious if their service really offered significant performance benefits. So we extracted the DNS code from our BrowserMob website monitoring service (which itself is based on the fantastic xbill Java DNS library), and built a quick-n-dirty tool to measure exactly that.

The results

Here’s how it works: our tool queries the Alexa worldwide top 1000 sites. It does it against Google Public DNS, OpenDNS, and (optionally) the DNS servers of your choosing. Since we’re located in Portland, OR and have a Qwest internet connection, we used Qwest’s DNS servers in our test. The results surprised us:

Starting test...
 
Test 1: Google
85109 ms for 1000 records
Test 2: Google
67586 ms for 1000 records
Test 3: Google
67318 ms for 1000 records
Test 1: OpenDNS
92521 ms for 1000 records
Test 2: OpenDNS
45793 ms for 1000 records
Test 3: OpenDNS
47899 ms for 1000 records
Test 1: Your DNS
62541 ms for 1000 records
Test 2: Your DNS
25561 ms for 1000 records
Test 3: Your DNS
25879 ms for 1000 records

Taking the lowest time and dividing by 1000 tells us the average DNS lookup times from our location in Portland, OR:

• Google – 67.3 ms
• OpenDNS – 45.8 ms
• Qwest DNS – 25.6 ms

What this told us was that despite all the hoopla about performance, our trusty old DNS server we’ve always been using is still the fastest. And when you think about it, it shouldn’t be a surprise: there are fewer hops from the computer to the DNS server because it’s the same ISP.

We were very surprised, however, to see how much faster OpenDNS was compared to Google. While both are slower than your local ISP’s, OpenDNS promises a bunch of features which might help make up for the performance difference. While Google isn’t promising any features right now, they are, as always, pledging to not be evil. Considering that some ISPs have in the past redirect DNS lookups to serve their commercial interests, Google’s pledge is worth noting.

Running your own test

These results were from a Qwest DSL connection in Portland, OR. We’re curious what your results are, so we’ve made the test available for everyone. It requires Java and can be run like so:

java -jar browsermob-dns-perf.jar

If you wish to test your ISP’s DNS, just add to the command line one or more IPs:

java -jar browsermob-dns-perf.jar 123.456.789.012

We hope that you will comment on this blog with the results you get. We’re very curious to see what the worldwide performance results are.

A note about the test

DNS is somewhat hard to test for performance, since there are many moving pieces. The results can depend on whether the server has a cached entry, how far you are from the DNS server, how far the DNS server is from the other servers, etc. This is why we run the test three times for each service an why we think the right approach here is to take the best score.

However, we do understand one could easily argue that the first result is much more critical. We sort of agree, but the problem is that because these are public services you can’t tell how “fresh” the cache is on the servers you are testing against.

In fact, we should note that Google’s FAQ argues that cache misses are big performance issue and the main benefit they bring to the table. Unfortunately, this is extremely difficult, if not impossible, for us to independently test. As such, we decided that the most consistent and useful result would be the faster time for each service, even if that means it was faster due to 100% cache hits.

Tweet This Post 

DNS · Google Public DNS · OpenDNS

We’re excited about Amazon’s new AWS location. So much so we’ve already started development to bring its benefits to our customers by next week. As such, we had to quickly figure out how to adapt our tools to work with it.

In their announcement, Amazon stated that Elasticfox already worked with the new location. While that’s true, if you tried looking for a new Elasticfox release you’d be in for a surprise: there isn’t one. That’s because it doesn’t need an update to work.

Instead, just fire up your existing Elasticfox installation and click on the Regions button in the upper left corner:

Then add new region with the following endpoint details:

• Name = us-west-1
• URL = https://us-west-1.ec2.amazonaws.com

That’s all there is to it. You’re now ready to use the new location with Elasticfox!

Tweet This Post 

Firebug

Late last night Amazon released an early Christmas present: a third location for which to run Amazon Web Services. This is important news for BrowserMob customers for two reasons:

• Monitoring – it will be our fifth location from where checks will run from.
• Load testing – it will be our third location from where traffic can be generated from.

We are committed to moving at the “speed of the cloud” and as such expect to have both of these options available in the next week. We’re excited about Amazon’s commitment to continue to provide geographic diversity to their cloud offering, and we look forward to their expansion to Asia early next year.

Tweet This Post 

No tags

I’m a little late to the party (dynaTrace released their product a couple weeks ago), but I wanted to still highlight their very important tool: dynaTrace AJAX Edition. It’s by far the best browser profiling tool out there – and it’s free!

If you’re familiar with Firebug, then that’ll give you a rough idea of what this product does. However, instead of being a “Swiss Army knife” like Firebug is, dynaTrace did a deep dive on JavaScript profiling.

This means that this tool can help identify hotspots and bottlenecks in your client-side JavaScript. And as applications continue to get more complex, this will continue to be a bigger and bigger area that developers need to keep an eye on. Ultimately, it doesn’t matter if your server is fast if the client application is getting bogged down due to inefficient JavaScript and DOM calls.

Rather than launching in to a full overview of what dynaTrace AJAX Edition does, I recommend the following three articles:

• A Step-by-Step Guide to dynaTrace Ajax Edition – by Andreas Grabner, technology strategist @ dynaTrace
• dynaTrace Ajax Edition: tracing JS performance – by Steve Souders, chief performance engineer @ Google
• Deep Tracing of Internet Explorer – by John Resig, creator of jQuery

If you’re working on performance optimizations for your site, especially if you are concerned about IE (and let’s be honest, who isn’t?), do yourself a favor and add this tool to your toolset. You won’t be disappointed.

Note: I should add that BrowserMob is a dynaTrace partner and we are continuing to explore ways in which BrowserMob, Selenium, and dynaTrace products can work in concert.

Tweet This Post 

No tags