Today we launched a brand new pricing structure for both our website load testing and website monitoring service. When coming up with new pricing for both our products, we set out do to four things:

1. Make it simpler and easier for anyone to quickly understand.
2. Make our advanced products (ie: our real browser services) more affordable.
3. Give a better value by reducing our pricing across the board.
4. Provide a free offering that never expires.

Load Testing Pricing

Our load testing service now has two types of fees:

• A base fee to use the software.
• A “cloud fee” for each test that gets run.

The base fee is where BrowserMob makes money as a company. We offer a weekly, monthly, or annual price – with significant discounts for annual purchases. We also offer three levels of service: Basic, Advanced, and Professional. Each package includes enough “cloud dollars” (see below) to complete most testing projects.

The cloud fee represents our raw costs to run in the cloud. This means after you pay the base fee, you are free to run as many tests as you’d like with no extra markup. For example, $15 in “cloud dollars” gets you a 100 browser test or a 1,000 virtual user test.

The most exciting thing about this model is that if your tests require no more than 25 browsers or 100 virtual users, there is no base fee. This means that after you use up the complementary 10 cloud dollars with each signup, you can keep using the service by simply covering the raw fees incurred with each test.

We’re also excited to offer any of our Basic, Advanced, or Professional packages to non-profits and open source organizations free of charge.

You can review all the load testing packages here.

Monitoring Pricing

Our monitoring pricing is now based on a very predictable number of “monthly page views”. For example, if you want to check a page every 15 minutes from one location, you would require 2,880 page views:

2,880 = 1 page view x 60 min / 15 min x 24 hrs x 30 days

The best part about this new model is that if you need 3,000 or fewer monthly page views, the service is 100% free. For more frequent monitoring, our monthly plans start at $49. You can review the monitoring pricing here.

Existing Customers

We will be posting a follow-up Q&A for all of our existing customers shortly. It will explain the transition process and what options are available. At a high level, we expect 99% of our customers to immediately get a better value. However, if any existing customer prefers to stick with the old model, we will gladly honor it.

Update: Please see our FAQ entry that discusses the changes for existing customers.

Tweet This Post 

Load Testing · Monitoring · Non-profit · Opensource · Pricing

Over the last few months Google has continued to focus on performance as a key message to developers. From their “Let’s make the web faster” initiative on Google Code, to their public DNS service, to the constant focus on performance by their Chief Performance guru, Steve Souders, it’s clear that Google wants people to have a faster experience on the web.

This makes sense: the faster the web, the more people will use it and be more likely to search and click on ads. In fact, in an interview with Web Pro News in November last year, Matt Cutts (head of Google’s webspam team) casually mentioned that in 2010 Google may give search ranking preference to sites that load faster. Om Malik went on to discuss whether this was a good idea or not.

More recently, Google opened up a new “Labs” feature in their Google Webmaster Tools that actually shows you the performance of your site as measured by the Googlebot. This move is further evidence that Google is indeed going to shift to making performance part of it’s ranking routine if it hasn’t already. See a screenshot of what you can expect to see:

In addition to the above chart, there is also some basic analysis and recommendations on how to improve performance, similar to what Page Speed and YSlow recommend. While it is a good start and provides the clearest view of how Google perceives your site’s performance, the data is still rather basic.

As such, when measuring and optimizing page load times we’d still recommend you work with a more in-depth tool such as Page Speed and YSlow. And, of course, for website monitoring and website load testing, we’d have to recommend BrowserMob

Tweet This Post 

Firebug · Google · Google Public DNS · Page Speed · Performance · YSlow

Brad Feld, an early stage investor and entrepreneur who runs the popular blog Feld Thoughts, recently dug up an old video showing how things have come almost completely full circle in the world of computing.

The video is embedded below for your convenience. It’s a bit long (~30 minutes), but if you’re interested in today’s cloud computing, it’s pretty entertaining to watch. It definitely gives you a good perspective on the evolution of computing:

Tweet This Post 

Cloud Computing

Often when it’s time to run a load test or turn on website monitoring, you don’t necessarily want the transaction hitting all your third party components on the page.

For example, you don’t want your analytics software to record the visits as real visitors, since that would skew your marketing metrics. Likewise, you don’t necessarily want advertisements served up, especially if the ad vendor uses “click-through rates” (CTR) to optimize ad prices and a load test would artificially drive down the CTR.

While there are vendor-specific actions you can take to deal with this problem (ie: we cover Google Analytics here), sometimes the easiest solution is to modify your script so that the request is never made in the first place.

You can do this for both Real Browser User (RBU) and Virtual User (VU) scripts using some new APIs that we recently rolled out. They allow you to whitelist or blacklist certain hosts from having actual HTTP requests sent to them.

For example, suppose you want to exclude all requests to www.google-analytics.com from your website monitoring or website load test. Your script would need to do something like this:

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
c.blacklistRequests("http://www\\.google-analytics\\.com/.*", 
                    200);
 
browserMob.beginTransaction();
browserMob.beginStep("Step 1");
 
// rest of test...

This would ensure that any HTTP request to Google Analytics would be bypassed immediately and a fake 200 response code would be returned instead.

Sometimes it’s easier to whitelist “good” hosts instead of blacklisting specific ones. This is often the case for advertising networks, which tend to host content from dozens of different domains.

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
c.whitelistRequests(["http://www\\.example\\.com/.*", 
                     "http://images\\.example.com\\.com/.*"], 
                    200);
 
browserMob.beginTransaction();
browserMob.beginStep("Step 1");
 
// rest of test...

In this example, we are allowing all requests to www.example.com and images.example.com to go through, but faking any other request with an immediate 200 response code.

These whitelist and blacklist APIs are just one example of the powerful scripting you can do using the BrowserMob API and scripting environment. We encourage you to review the entire API documentation from time to time, as well as contact us if you ever have questions about them.

Tweet This Post 

3rd party widgets · API · Google Analytics

Often when someone wants to test a site that isn’t yet ready for public release, they will use various security measures to keep the general public from looking at it. Sometimes they will use BASIC authentication and other times they will use firewall rules to only allow certain IP addresses access to the site.

However, as we continue to see a growing trend of BrowserMob customers who are deploying websites in the Amazon cloud using EC2, we wanted to highlight that the firewall technique does not work using our service.

The reason is that BrowserMob also runs in the Amazon cloud and as such is able to access the customer website across it’s private network. Because of this, the incoming IP address is not one of the ones we have documented, but a random internal-only IP that we do not know about ahead of time.

Fortunately, there is an easy solution: set up your EC2 network security to allow our Amazon “user ID” direct access through the port your website is hosted on. Simply authorize the User ID 581518868128 in the appropriate EC2 Security Group.

For example, If you are using Elasticfox, go to the Security tab and add a new permission for the appropriate Security Group. It should look like this:

Once done, you should be able to run your load test just fine. We do recommend running a small scale test first to confirm it works before running the larger test. If it’s not working, every request will result in a timeout.

We hope that helps, and we look forward to continuing to work with more companies making the transition to the cloud!

Tweet This Post 

EC2 · Security

Tom Pinckney, co-founder of Hunch, recently prescribed some tips on how to survive being Slashdotted. While Slashdot is no longer the biggest online driver of spikes of traffic, there was a time when it was the king of bringing down sites: a simple mention of a URL on the home page would crush most servers.

These days sites like Yahoo, Digg, and Drudge Report often carry much more weird, but Tom’s lessons apply to any situation where you are going to get a huge spike in traffic (expected or otherwise). There’s a lot of good stuff in Tom’s post, as well as previous one about achieving “consistent performance” that are well worth a read.

Perhaps the best advice Tom gives is:

If using Apache, use worker MPM with MaxClients and ServerLimit as high as you have RAM for. Better to cap max # of threads and reject some users than let the whole machine go down.

This is an important point people often forget: eventually you will fail, so better plan to fail gracefully.

Tweet This Post 

Digg · Slashdot

BASIC authentication is used to provide minimal, low-security protection from anonymous visitors hitting your website. It is frequently used by companies to ensure that their staging or development environments are not accessibly by the general public prior to pushing the changes to production. Typical authentication dialog prompts look like so:

The challenge here is that this dialog box is the kind of dialog that Selenium cannot automate. You cannot issue “type” or “click” commands on it. In fact, if this box comes up, your script is guaranteed to time out because Selenium will continue to wait for the page to load, not realizing a login is required and unable to populate it.

Traditionally Selenium users have worked around this problem by using a URL pattern in which the username and password was encoded:

http://username:password@example.com

Recently, however, this is not working across all browsers. IE no longer supports it and recent versions of Firefox add a confirmation box that breaks the automation with a slightly different popup. And while we’re working on as solution for Selenium 2.0, there aren’t a lot of options for Selenium today.

But when it comes to BrowserMob, we do have a solution available today, which will be soon back-porting to the Selenium 2 codebase. Simply edit your script to look like the following:

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
c.autoBasicAuthorization("example.com", "username", "password");
 
browserMob.beginTransaction();
browserMob.beginStep("Step 1");
 
selenium.open("http://example.com");
// rest of script...

This code tells BrowserMob to automatically put in the required Authorization headers for any HTTP request issued to example.com. And soon this capability will be added to Selenium 2 (after we get the first alpha out in time for the holidays).

Tweet This Post 

Basic Authentication · Scripting · Selenium 2

Like many in the industry, we were surprised and intrigued by the announcement yesterday that Google would be entering the DNS business. The basic logic was clear: Google has a vested interest in the internet being fast, and so they want to offer a free public utility to help it be faster.

Of course, some were doubtful. OpenDNS, probably the company that has the most to lose by this decision, responded quickly. Some questioned its security, while others pointed out that Google gains a lot more than you might think by serving DNS: it would now know everywhere you were going, regardless of whether you went through Google Search or whether the site had Google Analytics installed.

While we’re not going to get in to the broader debate of whether this move is Good or Evil, we were curious if their service really offered significant performance benefits. So we extracted the DNS code from our BrowserMob website monitoring service (which itself is based on the fantastic xbill Java DNS library), and built a quick-n-dirty tool to measure exactly that.

The results

Here’s how it works: our tool queries the Alexa worldwide top 1000 sites. It does it against Google Public DNS, OpenDNS, and (optionally) the DNS servers of your choosing. Since we’re located in Portland, OR and have a Qwest internet connection, we used Qwest’s DNS servers in our test. The results surprised us:

Starting test...
 
Test 1: Google
85109 ms for 1000 records
Test 2: Google
67586 ms for 1000 records
Test 3: Google
67318 ms for 1000 records
Test 1: OpenDNS
92521 ms for 1000 records
Test 2: OpenDNS
45793 ms for 1000 records
Test 3: OpenDNS
47899 ms for 1000 records
Test 1: Your DNS
62541 ms for 1000 records
Test 2: Your DNS
25561 ms for 1000 records
Test 3: Your DNS
25879 ms for 1000 records

Taking the lowest time and dividing by 1000 tells us the average DNS lookup times from our location in Portland, OR:

• Google – 67.3 ms
• OpenDNS – 45.8 ms
• Qwest DNS – 25.6 ms

What this told us was that despite all the hoopla about performance, our trusty old DNS server we’ve always been using is still the fastest. And when you think about it, it shouldn’t be a surprise: there are fewer hops from the computer to the DNS server because it’s the same ISP.

We were very surprised, however, to see how much faster OpenDNS was compared to Google. While both are slower than your local ISP’s, OpenDNS promises a bunch of features which might help make up for the performance difference. While Google isn’t promising any features right now, they are, as always, pledging to not be evil. Considering that some ISPs have in the past redirect DNS lookups to serve their commercial interests, Google’s pledge is worth noting.

Running your own test

These results were from a Qwest DSL connection in Portland, OR. We’re curious what your results are, so we’ve made the test available for everyone. It requires Java and can be run like so:

java -jar browsermob-dns-perf.jar

If you wish to test your ISP’s DNS, just add to the command line one or more IPs:

java -jar browsermob-dns-perf.jar 123.456.789.012

We hope that you will comment on this blog with the results you get. We’re very curious to see what the worldwide performance results are.

A note about the test

DNS is somewhat hard to test for performance, since there are many moving pieces. The results can depend on whether the server has a cached entry, how far you are from the DNS server, how far the DNS server is from the other servers, etc. This is why we run the test three times for each service an why we think the right approach here is to take the best score.

However, we do understand one could easily argue that the first result is much more critical. We sort of agree, but the problem is that because these are public services you can’t tell how “fresh” the cache is on the servers you are testing against.

In fact, we should note that Google’s FAQ argues that cache misses are big performance issue and the main benefit they bring to the table. Unfortunately, this is extremely difficult, if not impossible, for us to independently test. As such, we decided that the most consistent and useful result would be the faster time for each service, even if that means it was faster due to 100% cache hits.

Tweet This Post 

DNS · Google Public DNS · OpenDNS

We’re excited about Amazon’s new AWS location. So much so we’ve already started development to bring its benefits to our customers by next week. As such, we had to quickly figure out how to adapt our tools to work with it.

In their announcement, Amazon stated that Elasticfox already worked with the new location. While that’s true, if you tried looking for a new Elasticfox release you’d be in for a surprise: there isn’t one. That’s because it doesn’t need an update to work.

Instead, just fire up your existing Elasticfox installation and click on the Regions button in the upper left corner:

Then add new region with the following endpoint details:

• Name = us-west-1
• URL = https://us-west-1.ec2.amazonaws.com

That’s all there is to it. You’re now ready to use the new location with Elasticfox!

Tweet This Post 

Firebug

People often ask what are the differences between RBUs and VUs. Obviously RBUs are more realistic while VUs are the perfect vehicle for testing APIs, but here is a table giving you a breakdown.

Some things to take note of:

• There is a difference in the difficulty for scripting RBUs vs. VUs. RBU scripts are easier to create with the Selenium IDE tool and have validation built in. Because RBUs mimic human behavior the
  scripts include built-in validation concepts. For example an error will be thrown if you try to click on an element that doesn’t exist because the page being displayed is a “server too busy” page instead of the expected page. With VUs, you have to code in your own validation logic.
• VUs are the preferred way to test SOAP/REST and other “non-browser” HTTP requests.
• Often the right approach involves using a bit of both RBUs and VUs. The VUs can be used to lay down base load on “read-only” pages that don’t involve form interaction or AJAX. The 80/20 rule applies here, you can generate 80% of the read-only traffic with VUs and use RBUs for the remaining 20%.

We hope this give you a better understanding of the differences and relative strengths of RBUs and VUs.

Tweet This Post 

No tags