uTest, the world’s largest community of on-demand QA professionals, launched a major update to their customer portal over the weekend. The changes included a bunch of good stuff for both testers and end-users, including:

• A brand new Flash-based UI that loads much faster than before and is much easier to use.
• Backend optimizations to allow the uTest to continue to grow beyond their already impressive 25K+ community of testers
• A brand new API allowing for more advanced integration and partnerships.

We’re most excited about uTest’s new API support. While it requires APIs that are not yet available, we’re really excited about the prospects of being able to utilize the uTest community programatically. We hope that soon we’ll be able to offer a fully integrated service in which BrowserMob customers can simply click a checkbox to have dozens of professional QA engineers participate in a load test, verifying functionality and recording their experience as thousands of automated browsers slam the site.

We’re really excited about the power of combining the cloud and the crowd to facilitate modern website testing. And if you have a need to augment your QA team, you should definitely check out uTest, especially now that the portal looks so good!

Tweet This Post 

Crowd · uTest

Today we’re extremely excited to announce another industry-first in the world of load testing: video capture.

From day one, we always believed that cloud computing ushered in an era where the majority of website load tests could and should be done using real browsers, something no one else had considered before. Doing so made scripting painless, load tests more realistic, and provided better error reporting through the use of screenshots.

Today we’ve raised the bar by also capturing video for every unique failure in a load test. This means that you can now see exactly what led up to the failure and even get a feel for what the user experience was like as your site began to break down.

In addition we also now capture the source HTML (ie: DOM structure) of the web page in the event of a failure, as well as the entire set of request and response headers for the failed transaction. This gives unparalleled information that is simply not available in the traditional load testing approach that relies on headless/virtual users.

We have a lot more great features in store for both our website load testing and website monitoring products, but we’re especially excited about this one. It’s available today so please check it our for yourself and let us know what you think.

Tweet This Post 

Load Testing · Video

Today we are pleased to announce several major improvements to BrowserMob’s monitoring service, as well as a brand new website:

• Drastically improved charts, including object-level performance charts
• Three new locations: New York City, Amsterdam, and Singapore
• Raw HTTP header captures for any failed transaction
• A brand new website, including a major update to our free Instant Test tool

Please check them out, or continue reading for more information.

Improved Charts

The number one feedback we received with our monitoring product was that while it was fun and easy to script real browsers, the charting left a lot to be desired. So we completely overhauled the charts, making them faster and more responsive. They also now break down each monitoring location use rich, vibrant colors, making it easy to see if you have performance issues in certain geographies.

In addition, we introduced object-level charting. Now when you view an individual transaction you’re going to be able to drill down to view individual object performance over time. Simply click on that element and you’ll be presented with a chart that plots response time for that individual HTTP request, broken down by each geography.

Three New Locations

We’ve added support for three new cloud-based locations, all powered by Voxel, a fantastic CDN and cloud computing provider. They are: New York City, Amsterdam, and Singapore. These locations are currently in beta and their long-term status will depend on how well they perform. So please try them out and let us know what you think. This brings our total locations up to seven, as we already support San Francisco, Dallas, Washington DC, and Dublin.

Raw HTTP Captures

In order to give you better root-cause analysis of monitoring failures, we now give you the ability to view the raw HTTP wire traffic for any failed test. This will tell you what HTTP headers were sent for every single GET or POST issued by the browser. Whether you leverage a large CDN or run a one-server deployment, this additional information should be very useful.

Brand New Website

Finally, we extremely happy to have rolled out a new website and complete UI overhaul for our monitoring and load testing services. In addition, we’ve completely rewritten our free Instant Test tool to make it faster, easier to use, and provide more actionable data analysis. Please try it out and let us know what you think!

Tweet This Post 

Monitoring · Performance

Tim Koopmans, one of the partners at Altentee, a partner of ours, recently wrote up some tips on how to emulate think time and pacing of scripts in BrowserMob, specifically for load testing. Tim provides some very handy code samples, so be sure to check them out, especially if you’re concerned about pacing of transactions/sec.

Tweet This Post 

Altentee · Think Time

For load tests, yes (we’re working to add support for this to monitoring). You can use BrowserMob API to add any name/value pair to your transactions or steps. Here’s a simple example:

var selenium = browserMob.openBrowser();
 
var tx = browserMob.beginTransaction();
 
var step = browserMob.beginStep("Home Page");
 
selenium.open("http://example.com");
 
// record how long it took for some text to appear
var start = new Date().getTime();
selenium.waitForTextPresent("dynamic text");
var end = new Date().getTime();
 
// get a special cookie out and store it
var cookie = selenium.getCookieByName("myCookie");
tx.put("cookie_value", cookie);
 
// and store the "text" timing, associated with the step
step.put("text_timing", end - start);
 
browserMob.endStep();
 
browserMob.endTransaction();

The important things to notice here are that both beginTransaction() and beginStep() actually return objects that represent the transaction or step, respectively. You can then call put() on those objects to associate arbitrary data, such as custom timings or cookie values, with them.

Then when the transaction is stored in the load test database, you’ll be able to look them up in the name_value_pairs table. For more information on the test database schema, consult the documentation.

Tweet This Post 

API

One of our customers, SignalFive, recently published a case study about a project they did with Al Gore’s Alliance for Climate Protection.

The project, called the Repower America Wall, involved a massive, Flash-based interface in which thousands of user-created images and videos could be scanned through at high speed.

In the case study, they describe how they calculated the estimated site traffic and decided to use cloud services, such as Amazon and YouTube, to help the site scale. However, before launch they still needed to prove to the customer that the site was ready:

While this stress-testing was valuable, and optimizations were made based on them, none of these techniques truly simulated real-world use. Typically, a testing environment such as Selenium IDE might be used. But since the UI of the wall was built in Flash, things got tricky.

Because of the rich Flash interface, traditional load testing services based on simple HTTP protocol simulation weren’t well-suited. For that reason, they came to BrowserMob. Using our real browser technology, we were able to load the entire UI, allow the background AJAX and AMF calls to take place, and even stream the various videos:

BrowserMob’s team helped us craft some tests which we ran on the dev and production servers. The information was invaluable — based on the test results, we decided to upgrade the production server, and doubled its memory.

Below is some of the load test charts SignalFive generated using our software:

Tweet This Post 

Case Study · Load Testing

I just recently came across a great article titled How tracking scripts affect page loads… can Google Analytics kill my web app?:

This post explains script blocking, and then shows how to safely setup a tracking script or any external script, such as Google Analytics or Quantcast, to not block page loads or other javascript handlers on your site.

It’s a great read and provides a clear explanation on how third party components (everything from advertisements to analytics to content widgets) can slow your site down. JavaScript includes have been commonly used in websites for coming on a decade, but there are still many people who don’t understand the destruction a rogue script can cause to user experience.

One such example of that kind of destruction recently happened to customer of ours who ran an advertisement during the Super Bowl. In the weeks and months leading to the event, they had done massive amounts of tests and verified that their data center could easily handle 2.5 gigabits per second or more.

They were more than prepared and, overall, the event went off without a hitch. However, there was one minor problem that did plague their launch: a third party that served a component on their site couldn’t keep up with the load.

It turns out that one page (unfortunately, an important one that allowed visitors to search for their products) included an inline JavaScript file hosted externally. This included file was added at the last minute and didn’t get subject to the intensive testing that the rest of the site had gone through.

When the external site went down, visitors’ browsers would timeout after 1-2 minutes of trying to fetch that content before finally firing the important onload event in the browser. Nothing hosted by the company had a problem, but yet it still affected the user experience.

To make matters worse, the customer did have certain JavaScript that fired on the onload event. This JavaScript added rich functionality, such as making a calendar pop up when clicking on a date field. Because of the 1-2 minute delay, this functionality wasn’t available immediately, causing the experience to deteriorate.

In addition to all the guidance given in the aforementioned article, there are two things they can do to avoid this next time:

• Remove the third party reference entirely. In this case the reference was to some JavaScript that in turn produced a “Follow me on Twitter” image. That image could have easily been hosted in their own data center, which had been tested thoroughly.
• Take advantage of the “on content ready” psuedo event. Most modern JavaScript frameworks support a non-standard event that will fire before onload. Most of the time, this is much more desirable as it means that the JavaScript will fire closer to when the user sees content in their browser – not just when all images and JavaScript has been downloaded.

The “on content ready” psuedo-event doesn’t have an official name, but a lot of people also refer to it as “on DOM ready”. YUI, Dojo, Prototype, jQuery, and many others support it, so be on the lookout.

Tweet This Post 

3rd party widgets · Performance

Our use of a real web browsers makes it really easy to do website load testing and monitoring. By default a page is considered to have timed out if the page “onload event” can’t fire within 30 seconds. However, sometimes the page appears to be fully loaded to a typical user and shouldn’t be counted as a timeout error in your tests.

We call these errors “soft timeouts”, and they require a little bit of code to ignore them:

var selenium = browserMob.openBrowser();
 
selenium.setTimeout(2500);
 
function ignoreIfSafeTimeout(e, step, minObjects, acceptableThreshold) {
    if (e.message.indexOf("Timed out after ") == -1) {
        throw e; // this isn't a timeout, so throw it
    }
 
    var objects = step.getObjects();
    var objectCount = objects.size();
 
    if (objectCount < minObjects) {
        throw "Expected at least " + minObjects + 
              " objects but only saw " + objectCount;
    }
 
    var errors = 0;
    for (var i = 0; i &lt; objectCount; i++) {
        var object = objects.get(i);
        var code = object.getStatusCode();
        if (code < 200 || code >= 400) {
            errors++;
        }
    }
 
    var errorRatio = errors / objectCount;
    if (errorRatio > acceptableThreshold) {
        throw "Expected no more than " + acceptableThreshold + 
              " error ratio but saw " + errorRatio;
    }
}
 
browserMob.beginTransaction();
 
var step1 = browserMob.beginStep("Home Page");
try {
    selenium.open("http://marketwatch.com");
} catch (e) {
    ignoreIfSafeTimeout(e, step1, 130, 0.1);
}
 
browserMob.endStep();
 
browserMob.endTransaction();

What this code does is catch the automatic timeout exception thrown when visiting a website (in this case http://marketwatch.com) and decide whether it ignore it or not enough of the page content had loaded. There are a few things to note:

• The timeout for the page load is set to 2.5 seconds. This is set artificially low to simulate a page timeout, which is bound to happen on a heavy page like MarketWatch where there are 100+ objects.
• A custom function, ignoreIfSafeTimeout , is defined that takes several arguments:
  • e – the exception that gets thrown when opening the page
  • step – the step that holds a list of all objects requested
  • minObjects – the minimum number of objects that should be associated with the step
  • acceptableThreshold – an acceptable ratio of errors-to-successes in the objects downloaded
• When browserMob.beginStep() is called, we save a reference to it (”step1″) that gets used to call ignoreIfSafeTimeout if an exception is thrown.
• The selenium.open() command is wrapped with a try/catch block.

The most important thing in here is the call to ignoreIfSafeTimeout() in the catch block. When called, we pass in the exception that we just caught along with the step object. The last two arguments are the important ones.

The third argument, 130, means that if there is a timeout and less than 130 objects have been requested, an error will be thrown. The fourth argument, 0.1, means that if more than 10% of the requests are error codes, then an error will be thrown.

Feel free to use this code as a starting point for reducing the number of “soft timeouts” in your load tests and monitoring jobs.

Tweet This Post 

Timeouts

A couple days ago we announced a brand new pricing system that lowered our pricing across the board. For our existing customers, this pricing is quite a bit different than what they were used to, so we wanted to put together a list Frequently Asked Questions about the new plans.

Is this new pricing more or less expensive?

By any metric, this a price reduction. We’re extremely excited to have found a model that controls costs for our customers while actually making the revenue flow for our company healthier.

Not only did we reduce the overall pricing for monitoring and/or give a better value for the price, we created a new way to buy load testing from us that is significantly less expensive for most projects.

But even if you stick with the old model of paying for load tests, we’ve reduced the price because the first 25 RBUs or 100 VUs are always sold “at cost”. This means that a 100 RBU test that used to cost $300 now costs $15 under the new plan and $228.75 when using the old plan. To understand how you can choose between either plan, please see the Can I still use my old pay-as-you-go plan? question.

How has pricing for monitoring changed?

The old pricing model for monitoring used a complex formula in which the time of the script being used to monitor your site heavily influenced the overall cost. For example, a job that averaged 3 seconds per run might cost half of that of a script that took 7 seconds to run. The problem with this model was that if your site slowed down, your costs would go up, making your monitoring budget unpredictable.

The new pricing model eliminates that unpredictability. Instead, there is a constant number of monthly page views that you’ll used, based entirely on your script, number of locations you check from, and frequency at which you check. No matter how fast or how slow your site is, the price will remain constant.

For example, if you wanted to test a script that hit your home page, visited the login page by clicking the login link, and then logged in, that would be 3 page views. If your monitoring job ran that script every 15 minutes from two locations, you’d use 17,280 page views per month:

17,280 = 3 page views x 60 mins / 15 min freq. 
           x 24 hours x 30 days x 2 locations

Our pricing packages start at 3,000 monthly page views per month (FREE) and go all the way up to 350K monthly page views.

What happens to my current monitoring jobs?

If you were paying for a monthly, recurring plan, then you have already been converted over to the new model and are now paying less than you were before. That’s because we’ve lowered our pricing for all monitoring packages. For example, what was $300/mo is now $249/mo, and what was $175/mo is now $149/mo.

However, if you were not using a monthly credit package to pay for your monitoring, you may need to take some action. Your monitoring jobs have not been disabled, but they are currently running free of charge. Over the next few weeks, we will be contacting each customer who is in this situation and work with them on finding a monitoring package that fits their needs.

In some cases, a customer may want to transfer credits previously purchased and apply them toward a monitoring job. We are more than happy to convert purchases over this way and will work with each customer that is interested in doing this.

How has pricing for load testing changed?

In our previous pricing model, each concurrent RBU cost 10 credits/hour while each concurrent VU cost 1 credit/hour. Customers could buy credits at various rates ranging from 30 cents/credit down to 7.5 cents/credit, depending on the size of the purchase. Purchases of $5,000 or larger were eligible to purchase additional credits of any amount at the original price (usually 7.5 or 10 cents).

Under the new model, there are no more credits. Rather, we have converted them all to “cloud dollars” by multiplying the remaining credits in your account by the price paid for the credits. For example, if you purchased 1000 credits for $300 and had 300 at the time of the switch, you would be left with $90 in cloud dollars.

These cloud dollars can be used to pay for load tests. The price for each load test will be quoted at the bottom of the “schedule load test” page. If the test is larger than the load test package that you currently have, we will quote a higher price that is based on the old pay-as-you-go pricing plan.

How do “cloud dollars” compare to credits?

If you’re used to the old credit system, the easiest way to think of cloud dollars is that they reflect a credit price of just 1.5 cents – a 20X reduction from the 30 cent price most of our customers were using.

This means that a 100 RBU test that used to cost $300 now costs only $15 when you are subscribed to a qualifying load test package. For customers who purchased a larger credit package and therefore bought in to a smaller credit price (ie: 15 or 10 cents/credit), the savings are not as large but are still very significant (ie: 10X or 6.67X, respectively.)

Can I still use my old pay-as-you-go plan?

Absolutely.

Simply keep buying cloud dollars just like if they were the old credit system. When you schedule a test, you’ll be quoted a price to run the test. You may be presented with a note explaining that the per-test fee would be lower if you were using a load test package, but you are under no obligation to sign up for such a package. This is especially ideal for anyone who does infrequent load testing.

The price of test will be based on what your original credit price was. This means that a 100 RBU test might cost $228.75 for one customer at the 30 cent price point and $78.75 for another customer at the 10 cent price point. However, both would only pay $15 if they were subscribed to a load test package.

Can I upgrade to the new pricing?

Yes. We can apply any unused cloud dollars towards load testing or monitoring packages. Just contact us and we can get the process started.

What is the threshold for the free BrowserMob services?

For monitoring, you can do up to 3,000 page views per month for free.

For load testing, you can run up to 25 RBUs and 100 VUs at any time without paying for a load testing package. However, you will have to pay the raw cloud fees for each test. These fees are 15 cents per RBU hour and 1.5 cents per VU hour (ie: 25 RBUs = $3.75 and 100 VUs = $1.50). Each account comes with 10 cloud dollars to get you started, but you can purchase more at any time.

Tweet This Post 

Load Testing · Monitoring · Pricing

Today we launched a brand new pricing structure for both our website load testing and website monitoring service. When coming up with new pricing for both our products, we set out do to four things:

1. Make it simpler and easier for anyone to quickly understand.
2. Make our advanced products (ie: our real browser services) more affordable.
3. Give a better value by reducing our pricing across the board.
4. Provide a free offering that never expires.

Load Testing Pricing

Our load testing service now has two types of fees:

• A base fee to use the software.
• A “cloud fee” for each test that gets run.

The base fee is where BrowserMob makes money as a company. We offer a weekly, monthly, or annual price – with significant discounts for annual purchases. We also offer three levels of service: Basic, Advanced, and Professional. Each package includes enough “cloud dollars” (see below) to complete most testing projects.

The cloud fee represents our raw costs to run in the cloud. This means after you pay the base fee, you are free to run as many tests as you’d like with no extra markup. For example, $15 in “cloud dollars” gets you a 100 browser test or a 1,000 virtual user test.

The most exciting thing about this model is that if your tests require no more than 25 browsers or 100 virtual users, there is no base fee. This means that after you use up the complementary 10 cloud dollars with each signup, you can keep using the service by simply covering the raw fees incurred with each test.

We’re also excited to offer any of our Basic, Advanced, or Professional packages to non-profits and open source organizations free of charge.

You can review all the load testing packages here.

Monitoring Pricing

Our monitoring pricing is now based on a very predictable number of “monthly page views”. For example, if you want to check a page every 15 minutes from one location, you would require 2,880 page views:

2,880 = 1 page view x 60 min / 15 min x 24 hrs x 30 days

The best part about this new model is that if you need 3,000 or fewer monthly page views, the service is 100% free. For more frequent monitoring, our monthly plans start at $49. You can review the monitoring pricing here.

Existing Customers

We will be posting a follow-up Q&A for all of our existing customers shortly. It will explain the transition process and what options are available. At a high level, we expect 99% of our customers to immediately get a better value. However, if any existing customer prefers to stick with the old model, we will gladly honor it.

Update: Please see our FAQ entry that discusses the changes for existing customers.

Tweet This Post 

Load Testing · Monitoring · Non-profit · Opensource · Pricing