This morning I will be presenting at with Jinesh Varia, an Amazon Web Services evangelist, at FutureTest in New York City. The conference is centered around the idea of exploring the future directions to take testing, QA, and software development. Jinesh Varia wrote up a nice summary of the slides we presented.

In our talk, we will demonstrate how cloud computing is primed to drastically change the way both automated and manual testing is done. Using service like Amazon EC2 and Amazon Mechanical Turk, it has never been easier to tap in to the power of the cloud. The key point of our presentation is that these new services allow for massive parallelization.

BrowserMob’s load testing service is an example of massive parallelization used for testing from the client’s perspective. We’re doing something no one has ever done before: spinning up thousands of web browsers and sending them against your website, allowing you to simulate the most realistic load test possible.

This concept just isn’t possible for most companies. Consider this: a 2,000 real browser user (RBU) test from BrowserMob consumes more than 2.5TB of memory and over 2,500 CPU cores. Assuming each CPU core and 1GB of memory costs $500, you’d have to spend $1.25M just for the hardware alone. It’s clearly not scalable to own that hardware, but our cost effective load test pricing shows that it is definitely affordable to rent it.

But as much as we love BrowserMob, the main point of our presentation today is that services like ours is just the tip of the iceberg. For example, Amazon Mechanical Turk, which is a human-powered intelligence engine, could allow you to enroll thousands of testers, paying them for each test case, to temporarily increase your manual QA effort from two QA engineers to hundreds or thousands of additional eyes.

In the case of Amazon EC2, suppose you had a nightly integration test, which runs Selenium scripts against your application. If you were to parallelize both the client side (ex: BrowserMob) and the server side (ex: Rails, J2EE, .NET, etc), you could execute each test case individually and independently from the rest. Assuming you had 1,000 tests, each taking one minute to run, you could reduce your integration test time from over 16 hours to minutes.

Whether it’s using parallel computing or parallel human resources, the cloud now offers the tools necessary to revolutionize testing and, in turn, software engineering in general. Look for new innovations in the coming months and weeks from all parts of the software stack – from vendors such as BrowserMob to open source projects such as Ruby on Rails. Do you have ideas for how the cloud can accelerate or improve the world of software testing and software development? If so, please leave a comment and join the discussion!

Tweet This Post 

No tags

When running a load test, there is usually a ton of data that gets generated. You get charts for response times and error rates. You usually also end up with large amounts of web server logs, which may have interesting data embedded in them. If you’re profiling your servers for their CPU, memory, and disk utilization, you’ll likely have additional charts to look at. All of these are necessary to piece together a complete picture of the results of your load test.

The Throughput Chart

But if I could only pick one metric that I could get out of a load test, it’d almost certainly be the data throughput charting that BrowserMob and most other load testing tools provide. This chart shows you how many bytes per minute were transfered during the load test. The following example shows almost 4GB/minute, which is almost 550 mbps in throughput (most sites never peak above 50 mbps):

The reason I like the throughput chart more than any other is because I can usually tell multiple things about the test from this one chart. For example, in this test, I can make the following conclusions without looking at any other data:

• Load was likely ramped up for the first half of the test.
• If load was applied at a constant rate during the second half of the test, response times likely stayed flat during the whole test.
• If load continued to ramp up during the whole test, response times likely increased in the second half of the test, causing throughput to flatten out.
• Near the middle of the test, something drastic happened to decrease throughput. It could be that load levels were reduced temporarily, but it could also be because a page or object in the transaction path suddenly took longer to load, reducing total throughput temporarily.

That’s just some of the reasons why I love the throughput chart in BrowserMob. What’s your favorite metric or chart when it comes to load testing?

Tweet This Post 

No tags

Contegix, an amazing internet hosting provider, recently ran some tests to determine the performance differences between various configurations of Apache and Tomcat. In the process of doing their testing, they used Pylot, an open source Python-based load testing tool, and BrowserMob. You can read their entire findings here.

What’s interesting is the use of both internal (Pylot) and external load testing (BrowserMob). This is something we strongly encourage our customers to do. In talking with Mark Rogers at Contegix, it was clear he saw a big difference between the type of traffic generated by Pylot and BrowserMob.

This shouldn’t be too surprising, since internal traffic runs on a super-fast local network, so the timings for opening and closing sockets are measured in nanoseconds rather than milliseconds. Content transfers so much faster and “cleaner” over a local network that the two styles of tests can look very different.

Internal load testing tools are great and simple, cost-effective ways to quickly validate that individual code/algorithms are executing in a high performance manner, but they don’t do a good job at telling you how your site will be experienced by real users from the real internet. That’s why external load testing is so important if you plan for anyone outside of your firewall to visit your site and have a good experience.

So next time you need to run some performance testing, don’t think of it as an internal vs. external question. Instead, use both for different purposes, like Mark and Contegix did, and you’ll always end up better informed than if you had only used one technique.

Tweet This Post 

No tags

A couple weeks ago we were approached by Tim Davis from MGX Lab to help with a load test for one of their clients who was going to be running an ad in the Super Bowl. They were working closely with their ISP and had been doing some internal testing up to that point, but they were looking for a much larger external test that could validate that the client website could withstand the huge volume of traffic sent it’s way during the big game.

That’s where BrowserMob’s web load testing system comes in. Here’s what Tim had to say about the experience:

The staff at BrowserMob were very knowledgeable, helpful, and truly interested in the actual performance of our web application. They worked with us to identify bottlenecks and even helped optimize and improve the load times and performance of some pages, which is above and beyond “stress testing”. We are very pleased to work with them and looking forward to working with them for our future high profile websites.

In the process of testing we generated over 100GB of data, executed more than 100K simulated visits, downloaded more than 4M objects, and created a peak data throughput of over 550mbps. The result? Their site withstood the volume from BrowserMob and survived the Super Bowl onslaught a week later. Not all websites featured in the Super Bowl fared as well.

Whether it’s the Super Bowl, Super Tuesday, or Cyber Monday, BrowserMob has the capacity and expertise to make sure your site is ready. Contact us today to see how we can help get your prepared.

Tweet This Post 

No tags

Traditional load testing has always involved simulating user traffic by replaying a series of HTTP requests. This traffic is designed to appear just like a the traffic a real user would cause and is considered a virtual user.

However, achieving realistic traffic can sometimes be very tricky, especially with new technologies like AJAX and things like .NET’s view state. The upside of this approach is that it requires a fraction of the hardware that would be required to run real browsers. For this reason, load testing tools have opted to simulate HTTP traffic for years.

Benefits of Real Browser Users over Virtual Users

That all changed when BrowserMob launched. With our service, you could leverage the power of the cloud to emulate real user behavior with real web browsers. Although this approach does indeed require much more hardware, we are able to make it lower cost than traditional load testing through the cost savings of cloud computing and open source.

Real browsers have many benefits:

• They generate load based on how the site is at runtime, rather than how the site was when the script was created.
• They automatically handle AJAX, .NET view state, JSF, and other technologies that are difficult to script in traditional load testing.
• They provide real screenshots of what a user would see in the event of an error.
• They are very easy to script, since you only have to worry about user interaction with the browser and not proper HTTP traffic simulation.

For as little as $1 per concurrent browser per hour, you could use real web browsers in your load test. But as great as real browsers are, sometimes they are overkill and there are lower-cost alternatives.

Using the Right Tool for the Job

But what about the times when you want to test something simple, such stressing your web server with a bunch of HTTP requests for the home page? Or suppose 90% of your users navigate the site but never fill out any forms or interact with complex AJAX components? Do you really need to use the overhead of a full browser in these cases?

The answer is: of course not. That’s why we have introduced a secondary service that is even lower cost. If you can script the HTTP traffic ahead of time, you can save 90% on your load test using our virtual user technology. BrowserMob even provides tools to allow you to convert a real browser user (RBU) script to a virtual user (VU) script, making it easy to pick the right tool for the job.

A common approach our customers use is to schedule two load tests at the same time: one smaller one using real web browsers and one larger one using virtual users. For example, suppose you wanted to test an e-Commerce website. While 10% of your traffic involves users logging in, adding items to shopping carts, and checking out purchases, 90% of your traffic might be doing simple searches and browsing your online catalog.

By using a mixture of approaches you can save money, reach the traffic levels you need to test, and still get the high fidelity/real browser playback for the use cases for the revenue-generating use cases.

Comparing the Differences

When deciding whether to use real browsers or virtual users, consider the following differences in features and pricing:

Consider the different requirements for your test by asking yourself the following questions:

• What kind of bandwidth do you hope to generate? How many IP addresses do you want to use for your test?
• Can you verify if a transaction is successful by simply looking for a text match in an HTTP request, or do you require more complex validation, such as checking whether a particular element on the page is visible?
• Is it enough to know the HTTP response code when an error happens, or do you wish to see a screenshot of the error?
• Can you sacrifice a bit of fidelity in order to reduce the price of the test?

When in doubt, we recommend using real browsers initially and contacting our support team for some additional advice.

Tweet This Post 

No tags

As of today, BrowserMob now supports both concurrent browsers and simulated virtual users. The term virtual user was adopted by early load test providers as a way to describe the technique of simulating the HTTP traffic that a real user would cause when visiting a website. While BrowserMob’s core technology sidesteps the concept of virtual users and actually drives real browsers, we felt it was important to also provide a second option for our customers.

As a service that prides itself on delivering load tests that closely emulate reality, it may at first seem odd for us to embrace the traditional concept of virtual users (VUs) that other load testing tools use. But we think it makes perfect sense when viewed in the larger context of our goal: to offer on-demand, low-cost load testing to everyone, including companies previously unable to afford it.

Reducing Costs Even More

We still think that using real browsers is the best way to generate load, as it gives the most realistic playback of load, responding to JavaScript, AJAX, and Flash in realtime. And we’re proud of the fact that we can offer real browsers at a fraction of the price of our competitors offer virtual users.

In fact, the only way to reduce our pricing further was to drop the overhead a real browser introduces. And that’s just what we did. For as little as 10 cents per VU/hour, you can schedule a test that uses VUs and generates massive amounts of load at an extremely low price. For example, 2,000 VUs can now run for an hour, generating over a 750mbps in data throughput, with a cost of only $200.

Visit our pricing page to learn more about this new offering and see how little it costs compared to the alternatives out there. Alternatively, if you have any questions, such as when to use virtual users and when to use real browsers, don’t hesitate to contact us.

So Which is Better?

This is a complex topic and there is no good answer. In a few days we’ll post a follow-up that discusses when to use real browsers vs. virtual users. In general, it comes down to a trade-off between price and fidelity, but usually at least some use cases can be simulated with virtual users and still achieve your requirements for load. Look out for our follow-up discussion for more on this.

Tweet This Post 

No tags

For any moderate-to-large size website, a load balancer is almost always used to spread web traffic across multiple web servers. When it comes to load testing an environment with load balancers, it’s important to understand how your network infrastructure and software behavior is configured to handle the incoming load. It’s also equally important to understand how your load testing tool generates it’s load. Only by knowing both of these things is it possible to run an accurate load test that yields useful results.

Types of Load Balancers

Load balancers often have three types of logic for distributing incoming HTTP requests. They are:

• Round-robin: incoming traffic is spread evenly across all the web servers, resulting in a single user’s browser session to likely hit some or all of the web servers. If a server goes down, incoming traffic is simply routed to the the other servers.
• Sticky sessions (cookie-based): incoming traffic is “stuck” to a single web server based on a cookie value that is added by either the web server or the load balancer. If a server goes down, the sessions that were tied to it are distributed to other servers.
• Sticky sessions (IP-based): incoming traffic is “stuck” to a single web server based on the IP address of the requesting client. If a server goes down, the sessions that were tried to it are distributed to other servers.

Depending on the load balancer logic used and how stateful your web application is, you may also need to configure your web server (IIS, Apache, Tomcat, etc) to share backend session state among the servers using a clustered memory technology. There are also other types of load balancing logic (ie: Cisco’s Aeropoint technology), but they are usually derivatives of one of the three outlined here.

Common Mistakes with Load Testing

One of the most common mistakes made during a load test is not understanding the mechanics in which traffic is generated from your load testing tool. Let’s use the following deployment architecture as an example:

In this example, 300 virtual users are being generated from one physical machine/IP address. Because the load balancer is configured to use sticky IP addresses, all the load is being sent to web server A, leaving web servers B and C entirely unused.

This scenario is very common during load tests. In fact, it’s one of the first things to check when your load testing tool reports increased response times and/or error rates even when manually browsing the site shows no such degradation (also known as the “it works for me” observation).

The reason it works well for individual browsers but is slow for the load testing tool is that most load balancers are smart to route new IP addresses to web servers B and C. They do this because they are seeing that web server A is being inundated with requests and is much busier.

Spreading the Traffic Around

There are multiple ways to address this problem. You could change the load balancer logic, bypass the load balancer and directly test just one web server, or temporarily remove enough web servers from the cluster to ensure that the ones remaining receive an even distribution of load.

Or you could simply spread the traffic being generated from your load test across more physical machines. Consider the following diagram:

In this example, 300 virtual users are now being generated from 15 different IP addresses. Now the traffic is spread evenly among the three web servers. By configuring your load test to spread the traffic among multiple sources, you not only solve the “it works for me” problem, but you also make your load test that much more realistic to the real world scenario of one IP address = one real user.

This is why it’s important to know how your traffic is being generated. If you’re using a self-hosted tool (Apache JMeter, OpenSTA, Mercury Load Runner, Borland SilkPerformer, etc), try acquire enough machines to spread the load appropriately. If you’re using a hosted service (Keynote, AlertSite, WebMetrics, etc), ask them about the number of IP addresses and machines being used in your load test. Doing this will help prepare you for any surprises when it comes time to test.

Getting Even More Distribution

BrowserMob is also a hosted load testing service, so it’s important to know what our machine distribution is as well. We offer two types of load tests:

• Ultra low-cost load tests using HTTP browser simulation.
• Low-cost load tests using real web browsers to drive traffic.

In both cases, our service uses a very low IP-to-VU ratio – much better than the rest of the industry. In the case of our simulated browsers, we use no more than 20 VUs per IP address. This means that if you run a 300 VU test you’ll be getting traffic from 15 different IP addresses (just as you saw in the last diagram). For our real web browsers, we do even better and use no more than six VUs per IP address. That is, for a 300 VU test you would see load coming from 50 different IP addresses!

It may not be practical or cost-effective to simulate an exact 1:1 ratio of IPs to VUs, as IP addresses are a limited quantity and can be costly to acquire in large volumes for this type of load testing. However, the lower you can keep that ratio the more realistic your test will likely be.

Tweet This Post 

No tags