A common question we get here at BrowserMob and elsewhere in the open source and QA communities we’re involved in is: “what’s the difference between stress testing, load testing, and performance testing?”

The three types of tests are often used interchangeably, though they do actually have somewhat different meanings. Fortunately, a friend of ours, Grig Gheorghiu, wrote about this very topic a few years ago in two blog posts:

• Performance vs. load vs. stress testing
• More on performance vs. load testing

Both of his posts are extremely good reads and should do a great job of clarifying the differences between the three. In summary, Grig argues (and we tend to agree with Grig) that:

• Performance testing is a process that focuses on testing individual components of the web app, such as databases, algorithms, network infrastructure, and cache layers.
• Load testing is a process of determining how an application under specific volumes of load, usually a range of the upper and lower limits expected by the business.
• Stress testing is a process of identifying when and how systems fail (and recover) under extreme levels of load.

BrowserMob can help with all three types of tests, though it’s best used for load testing and stress testing. Because it uses real browsers, BrowserMob is uniquely suited to assist with these two types of tests, since it is able to simulate the type of load that you would see from your real users.

In the case of performance testing, BrowserMob can assist, especially if you run low-level amounts of load (ie: 5-10 concurrent users) while also collecting performance metrics at various levels of your application. But there are other tools that can help too. Grig’s article provides a list of many tools he finds useful, but we’d also like to add one to his list: Firebug.

We’ll post a full blog entry on Firebug soon, but for know now that it is a “Swiss Army Knife” for web developers. It helps with CSS, HTML, and JavaScript development. But in addition to that, it also provides a simple ability to see the network timings of every object downloaded in a web page. When used in conjuction with clearing your cache (or clicking shift-reload), it can serve as a great way to see the HTTP performance of your application when there is no load on the system.

UPDATE: For another good list of definitions, check out Corey Goldberg’s Glossery of Terms for his Python-oriented testing tools.

Tweet This Post 

No tags

Selenium is a popular open source, cross browser functional testing tool. You can learn more about it here. If you’re looking for functional testing solutions that use real web browsers, we strongly encourage you to take a look at Selenium.

We have a close relationship with Selenium: BrowserMob’s founder, Patrick Lightbody, was the original creator of the most popular version of Selenium, Selenium Remote Control. So it’s not surprising that BrowserMob utilizes Selenium technologies for both the record and playback of web traffic.

Because Selenium is a functional testing tool, it isn’t really meant to be used as a load product. There are two reasons for that:

1. Selenium doesn’t capture and report the type of data that load testers want.
2. Selenium uses real browsers, meaning you need a ton of hardware to simulate a load test.

BrowserMob solves both of those issues and has successfully turned a popular functional testing tool in to an excellent load testing tool. This has an added benefit that if you are already using Selenium for functional testing, you likely already have scripts needed to do load testing.

Tweet This Post 

No tags

There is no hard or fast rule on how long your tests should be, since it all depends on what you’re trying to do. However, we can provide some general guidance as well as some tips to optimize your value with BrowserMob.

First, it helps to know a bit about how the application works, since that may very well determine how long you should run your test for.

For example, suppose that your application took a request at the end of a user transaction and put it in to an in-memory queue for processing later. This would mean that if your load test executes more transactions than the queue can process, memory will get consumed, eventually resulting in a failure. Therefore, it might be a good idea to run your test long enough to determine when that failure occurs.

On the other hand, if your application doesn’t have any characteristic that would require running a test for hours on end, a shorter test of 10-15 minutes of sustained load might be all you need. Talk with the developers and architects of the software to get a feeling for what type of testing is important.

You should also know early on what type of test you want to run. There are different types of tests that you might be interested in, such as a multi-day “burn in” test, a short “burst” test that only lasts a few minutes, or an hour long test designed to simulate real load during a peak hour in the day. Different types of tests have different lengths, so we encourage you to discuss this with your team.

Finally, it also helps to optimize your costs with BrowserMob. Currently, BrowserMob charges on a per-hour basis, rounding up to the nearest hour. This means that a 100 user test for 5 minutes costs the same as a 100 user test for 60 minutes. Therefore, we recommend that even if you need only a small 5 or 10 minute test, you schedule a longer one and take advantage of the “pause” feature.

By scheduling a longer test and/or configuring multiple phases at different load levels, you can save money. And if you need to stop the load activity for a few minutes while you configure services or reset the database, you can pause the active test first, do your work, and then resume the test activity.

If you have questions about how long your test should be or how many users you might need, please don’t hesitate to contact us.

Tweet This Post 

No tags

There is no clear cut answer to this, but we’re happy to provide guidance.

Your business liaison (usually a product manager, project manager, or analyst) may know the answer right away, since there may be specific business goals associated with the project.

The system architect may also have a good idea of what the capacity of the overall system is, so that might be another place to start asking.

You should also read our FAQ on hits/second vs. concurrent users, as that may help you better think about how many virtual users you need in a load test.

When in doubt, a good rule of thumb (but certainly far from accurate) is that a single web server can handle roughly 250 to 500 concurrent users. So if the environment you’re testing has two web servers behind a load balancer, it’s reasonable to expect it can handle between 500 and 1000 concurrent users from BrowserMob.

Warning! Even if you have reasonably good confidence that the system can handle hundreds or thousands of users, we strongly encourage you to start small, ramping up and scheduling several tests over time. This will almost certainly catch any simple performance errors early on, before you’ve spent a lot of time and energy on larger tests.

Tweet This Post 

No tags

Like most aspects of load testing, there is no easy or simple answer to this question. The answer often depends on what your goals are, how the application logic works, and what parts of the application you’re trying to test.

But in general, it’s important to clearly define various terms, which will help communication amongst your team and help you get to a clear goal before starting testing.

Some common words or phrases that are worth clarifying are:

• hits – A hit is most commonly known as an HTTP request to your website. It could be for a web page, an image, a style sheet, some JavaScript, a video file, etc. All hits are not equal: some may just need a simple image to be served, while others might cause complex application logic and database access.
• visits – A visit is usually referred to as the end-to-end interaction between a single user and your website.
• sessions – Sessions are similar to visits, but often have a more server-centric meaning. Most modern web application environments today have a concept of a server-side session, which is responsible for storing things like what is in the user’s shopping cart. Sessions often stay active for some time after the user’s visit is over, meaning additional memory is consumed until that session expires.
• virtual users – See this FAQ entry for more information on virtual users.
• concurrent users – Concurrent users can take on multiple meaning, but it most often represents how many virtual users are actively simulating a visit to your site.
• page views – A request to a web page by a browser. A page view always involves one hit (for the page itself) plus dozens of subsequent hits for images, JavaScript, CSS, etc that the page depends on.
• requests – An overloaded term that often means many things, such as a hit or a page view. We suggest using the term “hits” for raw HTTP requests and “requests” as a synonym for “page views”, or just avoiding the term “requests” entirely to avoid confusion.
• transactions – A single execution of the recorded script that the virtual user is supposed to run. A single virtual user will run as many transactions as it can (as long as the VU is active).

With these definitions out of the way, now you can ask yourself: what am I most concerned about testing?

If your application is mostly stateless, meaning it likely doesn’t consume much or any session state in memory, you are probably more interested in discovering what your peak hits/second, page views/second, or transactions/minute is.

If your application is not stateless or otherwise has significant background processing and load that is due to new sessions, you’re probably more interested in concurrent users.

Often you care about both, so thinking about these different terms and setting expected goals for all of them is important. Find out how many hits you expect for a particular visit or transaction. Determine how many total sessions you expect in a peak time period, but also how many concurrent users you expect at the same time. Then rank the metrics from most important to least important in terms of which you’d like to simulate closely.

With this information, you’ll be better prepared to schedule the appropriate load tests. Or, if you’re still unsure, feel free to share the information you have already gathered with our support team. We’d be happy to help advise you on how to proceed or get you in touch with one of our partners who can provide dedicated, hands-on support.

Tweet This Post 

No tags

BrowserMob can support hundreds and thousands of concurrent browsers hitting your website. Because we use real browsers (vs simulating traffic only) this consumes far more resources than any other load testing solution out there. As such our top-end capacity is not necessarily as high as other offerings, though we feel that we can provide enough load for most needs.

See our pricing page for details of our current capacity. If you’re not sure if we can support the capacity you need, try scheduling a test on your own. If we can support it without any human interaction (which is often the case), the test will be scheduled without any warning.

If we can’t support it or we require some minor manual preparation, which is sometimes needed for very large tests, you’ll be notified and asked to contact our support department.

In most cases, we can help you achieve the load you desire, but in some cases our technology simply won’t yet be able to support that many users. If that is the case, you’re more than welcome to use our service for smaller scale tests (ex: again subsets of the overall server farm) and for the larger tests we’d be happy to advise and/or refer you to other tools or solutions that can help you.

Tweet This Post 

No tags

People sometimes wonder how we can charge so much less, especially considering that we’re running real browsers for every virtual user, which costs significantly more than any other load testing technique.

The answer is that we have built an innovative technology that utilizes open source and cloud computing, which allows us to keep our costs very low, despite our higher overhead per virtual user.

What this means in terms of physical location of our browsers is that it’s determined by our cloud computing provider, Amazon. Those locations are:

• Three data centers are located in the Washington, DC (US east coast).
• Two data centers are located in Dubline, Ireland (western Europe).

You can select these different geographic regions when scheduling a load test, as shown in this screenshot:

Tweet This Post 

No tags

BrowserMob is an external load testing service, meaning traffic originates from outside your firewall. We believe this is the most realistic way to test your web applications, as it properly shows how your site will be perceived by actual end users using real web browsers from external locations, such as their homes or offices. The topic of whether you should test on a production or production-like staging environment is discussed here, so please read that as well.

There are three common approaches if your application is only available internally:

1. Deploy your application temporarily to a public or semi-public location. You can keep the URL secret, but it needs to be accessibly by the BrowserMob network.
2. Open up your corporate firewall to allow the BrowserMob network to access your internally deployed application. See here for a list of the IP addresses that BrowserMob traffic comes from.
3. Use an open source load testing tool, such as JMeter or OpenSTA, to simulate basic traffic internally. Then, when it’s time to do a full scale test, try option #1 or #2 when using BrowserMob.

If you’d like to try option #1 (deploy your app externally) but don’t have any hardware readily available to do it, we strongly encourage you to look at Amazon EC2. It’s one of the networks that BrowserMob runs on and can allow you to rent machines for as little as 10 cents an hour. For a couple of dollars, you can build a temporary staging environment that is still private but allows BrowserMob to test it’s performance.

Tweet This Post 

No tags

Some of our customers want to know the IP addresses of our browsers so that they can allow them inside their firewall for the purpose of testing internally-hosted applications. Because we can support up to thousands of browsers, we can’t possibly list every individual IP address. However, we can share with you the range of IP addresses.

For our Washington DC location, they are:

• 216.182.224.0/20 (216.182.224.0 – 216.182.239.255)
• 72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
• 67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
• 75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
• 174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
• 204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
• 184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
• 184.72.128.0/17 (184.72.128.0 – 184.72.255.255)

For our San Francisco, CA (Bay Area) location, they are:

• 204.236.128.0/18 (216.236.128.0 – 216.236.191.255)
• 184.72.0.0/18 (184.72.0.0 – 184.72.63.255)

For Dublin, Ireland location, they are:

• 79.125.0.0/17 (79.125.0.0 – 79.125.127.255)

For Singapore location, they are:

• 180.92.186.194
• 175.41.128.0/18 (175.41.128.0 – 175.41.191.255)

For New York City location, they are:

• 72.251.214.162

For Dallas, TX location, they are:

• 173.203.221.231

For Amsterdam location, they are:

• 72.26.217.194

Tweet This Post 

No tags

The concept of a “virtual user” is commonly used in load testing, but the truth is that the definition means different things to different people. Here we will explain what it means to BrowserMob and how you can use them in different ways to test capacity, load, or stress of your application.

In general, a virtual user is exactly that: a virtualize representation of a user, specifically designed to simulate the same behaviors and interactions with your website that a real user would. For example, if you knew that at peak you had 100 real users on your site last year, you could simulate that same behavior with 100 virtual users running one or more scripts that invoke similar navigation and interaction that your real users do.

For example, suppose during peak time your site got 4 visitors over a specific an hour, as demonstrated here:

While you received four users in an hour, you only received three concurrent users at most (V1+V2+V4 and V2+V3+V4). So this would mean the max concurrent users you have seen is three, and that should probably be the benchmark and/or baseline in which you do your load testing around, depending on your goals.

In BrowserMob, a virtual user is a dedicated web browser that is ready to play back a pre-recorded script one or more times endlessly. It’s very important to understand that a virtual user == a web browser and that the browser will repeat the script endlessly until the test ends or the number active virtual users are turned down. This means that if you scheduled a load test that ramped from zero to three users over an hour long period, the load profile might look like the following:

In other words, with three virtual users, you might see the overall visits be closer to five or six. This means that when planning out your load test, it’s important to think about total visits/sessions as well as concurrency. Usually concurrent is what people are most interested in, but sometimes sessions are very important too. By remembering that a “virtual user” in BrowserMob replays the script over and over again, you can properly plan out the expected concurrency and total sessions.

This often means that if you aren’t worried about total sessions but instead just the raw hits/traffic that they cause, you can usually use far less virtual users than you might initially have thought you needed. In this example, we could likely use two browsers/virtual users and achieve the same number of visits we saw in the real world.

This topic is a very broad topic, and often related questions come up, such as the difference between load testing and stress testing which is more important: hits or concurrent users. Please see our FAQ and our blog for a full list of related questions.

Tweet This Post 

No tags