As of today, BrowserMob now supports both concurrent browsers and simulated virtual users. The term virtual user was adopted by early load test providers as a way to describe the technique of simulating the HTTP traffic that a real user would cause when visiting a website. While BrowserMob’s core technology sidesteps the concept of virtual users and actually drives real browsers, we felt it was important to also provide a second option for our customers.

As a service that prides itself on delivering load tests that closely emulate reality, it may at first seem odd for us to embrace the traditional concept of virtual users (VUs) that other load testing tools use. But we think it makes perfect sense when viewed in the larger context of our goal: to offer on-demand, low-cost load testing to everyone, including companies previously unable to afford it.

Reducing Costs Even More

We still think that using real browsers is the best way to generate load, as it gives the most realistic playback of load, responding to JavaScript, AJAX, and Flash in realtime. And we’re proud of the fact that we can offer real browsers at a fraction of the price of our competitors offer virtual users.

In fact, the only way to reduce our pricing further was to drop the overhead a real browser introduces. And that’s just what we did. For as little as 10 cents per VU/hour, you can schedule a test that uses VUs and generates massive amounts of load at an extremely low price. For example, 2,000 VUs can now run for an hour, generating over a 750mbps in data throughput, with a cost of only $200.

Visit our pricing page to learn more about this new offering and see how little it costs compared to the alternatives out there. Alternatively, if you have any questions, such as when to use virtual users and when to use real browsers, don’t hesitate to contact us.

So Which is Better?

This is a complex topic and there is no good answer. In a few days we’ll post a follow-up that discusses when to use real browsers vs. virtual users. In general, it comes down to a trade-off between price and fidelity, but usually at least some use cases can be simulated with virtual users and still achieve your requirements for load. Look out for our follow-up discussion for more on this.

Tweet This Post 

No tags

For any moderate-to-large size website, a load balancer is almost always used to spread web traffic across multiple web servers. When it comes to load testing an environment with load balancers, it’s important to understand how your network infrastructure and software behavior is configured to handle the incoming load. It’s also equally important to understand how your load testing tool generates it’s load. Only by knowing both of these things is it possible to run an accurate load test that yields useful results.

Types of Load Balancers

Load balancers often have three types of logic for distributing incoming HTTP requests. They are:

• Round-robin: incoming traffic is spread evenly across all the web servers, resulting in a single user’s browser session to likely hit some or all of the web servers. If a server goes down, incoming traffic is simply routed to the the other servers.
• Sticky sessions (cookie-based): incoming traffic is “stuck” to a single web server based on a cookie value that is added by either the web server or the load balancer. If a server goes down, the sessions that were tied to it are distributed to other servers.
• Sticky sessions (IP-based): incoming traffic is “stuck” to a single web server based on the IP address of the requesting client. If a server goes down, the sessions that were tried to it are distributed to other servers.

Depending on the load balancer logic used and how stateful your web application is, you may also need to configure your web server (IIS, Apache, Tomcat, etc) to share backend session state among the servers using a clustered memory technology. There are also other types of load balancing logic (ie: Cisco’s Aeropoint technology), but they are usually derivatives of one of the three outlined here.

Common Mistakes with Load Testing

One of the most common mistakes made during a load test is not understanding the mechanics in which traffic is generated from your load testing tool. Let’s use the following deployment architecture as an example:

In this example, 300 virtual users are being generated from one physical machine/IP address. Because the load balancer is configured to use sticky IP addresses, all the load is being sent to web server A, leaving web servers B and C entirely unused.

This scenario is very common during load tests. In fact, it’s one of the first things to check when your load testing tool reports increased response times and/or error rates even when manually browsing the site shows no such degradation (also known as the “it works for me” observation).

The reason it works well for individual browsers but is slow for the load testing tool is that most load balancers are smart to route new IP addresses to web servers B and C. They do this because they are seeing that web server A is being inundated with requests and is much busier.

Spreading the Traffic Around

There are multiple ways to address this problem. You could change the load balancer logic, bypass the load balancer and directly test just one web server, or temporarily remove enough web servers from the cluster to ensure that the ones remaining receive an even distribution of load.

Or you could simply spread the traffic being generated from your load test across more physical machines. Consider the following diagram:

In this example, 300 virtual users are now being generated from 15 different IP addresses. Now the traffic is spread evenly among the three web servers. By configuring your load test to spread the traffic among multiple sources, you not only solve the “it works for me” problem, but you also make your load test that much more realistic to the real world scenario of one IP address = one real user.

This is why it’s important to know how your traffic is being generated. If you’re using a self-hosted tool (Apache JMeter, OpenSTA, Mercury Load Runner, Borland SilkPerformer, etc), try acquire enough machines to spread the load appropriately. If you’re using a hosted service (Keynote, AlertSite, WebMetrics, etc), ask them about the number of IP addresses and machines being used in your load test. Doing this will help prepare you for any surprises when it comes time to test.

Getting Even More Distribution

BrowserMob is also a hosted load testing service, so it’s important to know what our machine distribution is as well. We offer two types of load tests:

• Ultra low-cost load tests using HTTP browser simulation.
• Low-cost load tests using real web browsers to drive traffic.

In both cases, our service uses a very low IP-to-VU ratio – much better than the rest of the industry. In the case of our simulated browsers, we use no more than 20 VUs per IP address. This means that if you run a 300 VU test you’ll be getting traffic from 15 different IP addresses (just as you saw in the last diagram). For our real web browsers, we do even better and use no more than six VUs per IP address. That is, for a 300 VU test you would see load coming from 50 different IP addresses!

It may not be practical or cost-effective to simulate an exact 1:1 ratio of IPs to VUs, as IP addresses are a limited quantity and can be costly to acquire in large volumes for this type of load testing. However, the lower you can keep that ratio the more realistic your test will likely be.

Tweet This Post 

No tags