BrowserMob can support hundreds and thousands of concurrent browsers hitting your website. Because we use real browsers (vs simulating traffic only) this consumes far more resources than any other load testing solution out there. As such our top-end capacity is not necessarily as high as other offerings, though we feel that we can provide enough load for most needs.

See our pricing page for details of our current capacity. If you’re not sure if we can support the capacity you need, try scheduling a test on your own. If we can support it without any human interaction (which is often the case), the test will be scheduled without any warning.

If we can’t support it or we require some minor manual preparation, which is sometimes needed for very large tests, you’ll be notified and asked to contact our support department.

In most cases, we can help you achieve the load you desire, but in some cases our technology simply won’t yet be able to support that many users. If that is the case, you’re more than welcome to use our service for smaller scale tests (ex: again subsets of the overall server farm) and for the larger tests we’d be happy to advise and/or refer you to other tools or solutions that can help you.

Tweet This Post 

No tags

People sometimes wonder how we can charge so much less, especially considering that we’re running real browsers for every virtual user, which costs significantly more than any other load testing technique.

The answer is that we have built an innovative technology that utilizes open source and cloud computing, which allows us to keep our costs very low, despite our higher overhead per virtual user.

What this means in terms of physical location of our browsers is that it’s determined by our cloud computing provider, Amazon. Those locations are:

• Three data centers are located in the Washington, DC (US east coast).
• Two data centers are located in Dubline, Ireland (western Europe).

You can select these different geographic regions when scheduling a load test, as shown in this screenshot:

Tweet This Post 

No tags

BrowserMob is an external load testing service, meaning traffic originates from outside your firewall. We believe this is the most realistic way to test your web applications, as it properly shows how your site will be perceived by actual end users using real web browsers from external locations, such as their homes or offices. The topic of whether you should test on a production or production-like staging environment is discussed here, so please read that as well.

There are three common approaches if your application is only available internally:

1. Deploy your application temporarily to a public or semi-public location. You can keep the URL secret, but it needs to be accessibly by the BrowserMob network.
2. Open up your corporate firewall to allow the BrowserMob network to access your internally deployed application. See here for a list of the IP addresses that BrowserMob traffic comes from.
3. Use an open source load testing tool, such as JMeter or OpenSTA, to simulate basic traffic internally. Then, when it’s time to do a full scale test, try option #1 or #2 when using BrowserMob.

If you’d like to try option #1 (deploy your app externally) but don’t have any hardware readily available to do it, we strongly encourage you to look at Amazon EC2. It’s one of the networks that BrowserMob runs on and can allow you to rent machines for as little as 10 cents an hour. For a couple of dollars, you can build a temporary staging environment that is still private but allows BrowserMob to test it’s performance.

Tweet This Post 

No tags

Some of our customers want to know the IP addresses of our browsers so that they can allow them inside their firewall for the purpose of testing internally-hosted applications. Because we can support up to thousands of browsers, we can’t possibly list every individual IP address. However, we can share with you the range of IP addresses.

For our Washington DC location, they are:

• 216.182.224.0/20 (216.182.224.0 – 216.182.239.255)
• 72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
• 67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
• 75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
• 174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
• 204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
• 184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
• 184.72.128.0/17 (184.72.128.0 – 184.72.255.255)

For our San Francisco, CA (Bay Area) location, they are:

• 204.236.128.0/18 (216.236.128.0 – 216.236.191.255)
• 184.72.0.0/18 (184.72.0.0 – 184.72.63.255)

For Dublin, Ireland location, they are:

• 79.125.0.0/17 (79.125.0.0 – 79.125.127.255)

For Singapore location, they are:

• 180.92.186.194
• 175.41.128.0/18 (175.41.128.0 – 175.41.191.255)

For New York City location, they are:

• 72.251.214.162

For Dallas, TX location, they are:

• 173.203.221.231

For Amsterdam location, they are:

• 72.26.217.194

Tweet This Post 

No tags

The concept of a “virtual user” is commonly used in load testing, but the truth is that the definition means different things to different people. Here we will explain what it means to BrowserMob and how you can use them in different ways to test capacity, load, or stress of your application.

In general, a virtual user is exactly that: a virtualize representation of a user, specifically designed to simulate the same behaviors and interactions with your website that a real user would. For example, if you knew that at peak you had 100 real users on your site last year, you could simulate that same behavior with 100 virtual users running one or more scripts that invoke similar navigation and interaction that your real users do.

For example, suppose during peak time your site got 4 visitors over a specific an hour, as demonstrated here:

While you received four users in an hour, you only received three concurrent users at most (V1+V2+V4 and V2+V3+V4). So this would mean the max concurrent users you have seen is three, and that should probably be the benchmark and/or baseline in which you do your load testing around, depending on your goals.

In BrowserMob, a virtual user is a dedicated web browser that is ready to play back a pre-recorded script one or more times endlessly. It’s very important to understand that a virtual user == a web browser and that the browser will repeat the script endlessly until the test ends or the number active virtual users are turned down. This means that if you scheduled a load test that ramped from zero to three users over an hour long period, the load profile might look like the following:

In other words, with three virtual users, you might see the overall visits be closer to five or six. This means that when planning out your load test, it’s important to think about total visits/sessions as well as concurrency. Usually concurrent is what people are most interested in, but sometimes sessions are very important too. By remembering that a “virtual user” in BrowserMob replays the script over and over again, you can properly plan out the expected concurrency and total sessions.

This often means that if you aren’t worried about total sessions but instead just the raw hits/traffic that they cause, you can usually use far less virtual users than you might initially have thought you needed. In this example, we could likely use two browsers/virtual users and achieve the same number of visits we saw in the real world.

This topic is a very broad topic, and often related questions come up, such as the difference between load testing and stress testing which is more important: hits or concurrent users. Please see our FAQ and our blog for a full list of related questions.

Tweet This Post 

No tags

Because BrowserMob is an external load testing service – meaning the traffic doesn’t originate from your internal network or from the data center where your application is hosted – you might be wondering which can cause potential slowdowns: the code or the network between BrowserMob and your servers?

This is a great question, but before we dive in to it, let’s remember what your users are likely to answer: “who cares!”

We don’t really mean to give such a flip answer, but we’re trying to highlight the important fact that at the end of the day the user doesn’t care if his ISP, your ISP, your database, or your code is the root cause of a slowdown. All he cares is that his visit to your site met his expectations.

Of course, that doesn’t mean you shouldn’t care where slowdowns are happening, since knowing the root cause is likely the only way to fix the problem. So how can you tell where the problem exists?

Currently, BrowserMob only can tell you what the performance looked like from the browser/end user’s perspective. We can tell you how fast or slow page load times were, if AJAX calls started taking longer as you added more concurrent sessions, or if images were loading slower or faster compared to stylesheets. But we can’t directly tell you why or where the slowdown occurred.

There are a few things you can do directly with BrowserMob:

• Schedule smaller tests. Don’t just run one large test. It may be that a small software glitch or system configuration is causing your site to slow down, so running several progressively larger tests will help you identify issues sooner and will likely save you money in the long run.
• Use the ramp feature. By ramping from 0 to 100 users, you can see what performance looks like with very few users and compare it to many users.
• Schedule multiple phases with ramping and constant levels of load. By putting some constant load in as well, you can get a “baseline” at different load levels and compare it to other baselines.

There are also a few things you can do outside of BrowserMob:

• Run two baseline tests Selenium IDE and a stopwatch: one from the local network where the server is located and one from a remote location (such as your home or remote office). This will give you a general idea of what the “network overhead” is.
• Work with your developers and database administrators. If possible, get them involved early and often during a load test. They can usually add some logging or profiling code that can pinpoint the exact problem.
• Know your network infrastructure. Knowing what throughput the environment can handle under test will help you know if your test has saturated the physical limits of the network. You can compare this data to the BrowserMob throughput chart.
• If you have any systems monitoring solutions, use them during load tests. Tools like Hyperic are great for telling you when the CPU or RAM usage goes critical on various machines, such as your web server and database server.

We should also note that we’re hard at work on some new features that will make your job of monitoring internal and external metrics much easier. Please stay tuned!

Tweet This Post 

No tags

Real browsers absolutely matter. That’s why we created BrowserMob! There are two major reasons why real browsers matter:

1. It simplifies the script creation process by letting you avoid all the complexities and hacks you have to do with traditional load testing tools.
2. It ensures that you’ll see 100% of the traffic and load against your site that a real user would cause.

These are fairly complex topics once you scratch beneath the surface. If you’re interested in knowing the detailed reasoning why real browsers help in both of these cases, continue reading.

Simplifies Script Creation

In today’s modern web applications, AJAX is just about everywhere. And we’re not necessarily talking about super rich applications like Google Maps or Yahoo Mail, but even simple sites like google.com now use advanced AJAX techniques. See Google’s auto-complete for a real-world example:

In this case, when typing values in to the search box, the web browser executes JavaScript logic that in turn makes AJAX calls to Google’s search engine, asking for search suggestions to display. It does this on every keystroke that the user types in.

When recording a script with a traditional load testing tool, one of two things may happen here:

• The recorder will see the AJAX traffic and capture it for playback in the load test
• The record will not see the AJAX traffic and will only capture the request made when the user clicks the “submit” button

Obviously these AJAX requests are causing real load, so we want to make sure they get played back in a load test. So let’s assume that the script you recorded does indeed capture the AJAX traffic. What would it look like? Something like this:


http://clients1.google.com/complete/search?hl=en&gl=us&q=s
http://clients1.google.com/complete/search?hl=en&gl=us&q=se
http://clients1.google.com/complete/search?hl=en&gl=us&q=sel
...
http://clients1.google.com/complete/search?hl=en&gl=us&q=seleniu
http://clients1.google.com/complete/search?hl=en&gl=us&q=selenium


As you can see, the keys typed by the user are included in each subsequent search term. Let’s ignore the requirement of validating the results that come back from the AJAX requests for the moment (they are usually in JSON or XML format and difficult to validate using most tools). Instead, let’s just add a twist to the load test requirement for doing searches: the load test must search from 100 different search terms.

Parameterization very common requirement, since it ensures that the load is realistic and doesn’t get cached in any unnatural way. This means that now in addition to searching for the term “selenium”, we’re also searching for “AJAX”, and “browser”, among others.

However, this means your script can’t just blindly submit requests to those previous URLs either, since those were tied to the “selenium” term. Instead, they must search for the sequential characters of the respective search term, such as:


http://clients1.google.com/complete/search?hl=en&gl=us&q=b
http://clients1.google.com/complete/search?hl=en&gl=us&q=br
http://clients1.google.com/complete/search?hl=en&gl=us&q=brow
http://clients1.google.com/complete/search?hl=en&gl=us&q=brows
http://clients1.google.com/complete/search?hl=en&gl=us&q=browse
http://clients1.google.com/complete/search?hl=en&gl=us&q=browser  


Unfortunately, this is where even the best traditional load testing tools fall down. They don’t provide any help here, so it’s up to you to figure out how to, if it’s even possible, write complex scripting logic that breaks down the randomly selected search term by characters and then subsequently issue AJAX requests for each character in the term. Oh, and also note that the URLs start with “clients1″, which indicates that perhaps Google has many servers to handle these requests and the server is embedded in some JavaScript or HTML from the original page, so you may have to parse that out too!

At this point, you’re basically rewriting the same logic that the application developer wrote originally. If you’re a QA engineer, this may be difficult since you don’t know all the internal details coded in to the application. If you’re the developer, it’s still annoying because it’s tedious and likely in a language other than the original JavaScript that you wrote your code in.

So how does BrowserMob help?

Because BrowserMob uses real browsers to both record and playback load, that means you don’t have to worry about trying to simulate the logic in a web browser. Instead, all you have to do is record the human interaction with the browser, such as typing in a randomly selected search term. BrowserMob will then pass those instructions on to the hundreds or thousands of browsers participating in the load test, and those browsers will in turn “do the right thing” and issue the proper AJAX requests.

And if the underlying logic, such as the request URL pattern for those AJAX requests, changes? With traditional load testing it’s up to you to detect and fix the problem. With BrowserMob, your script won’t need to change one bit – the new AJAX logic will be run by the browser in real time.

Ensures Realistic Traffic

We’ve seen how BrowserMob helps with recording, but what about playback? As we just learned, using real browsers simplifies the process of recording and shrinks the behavior coded in to the script itself. This means we’re letting the real browser – the same type of program your end users will use – make the decisions about what requests to make.

For example, when visiting http://ebay.com you might see the following page:

But reload the page and now you might see this:

Notice a difference? the “Today on eBay” section has completely different images displayed. That’s because eBay’s home page chooses what to display based on complex and multi-variant logic determined at runtime. It’s quite likely that it’s going to be impossible for a load tester to know which images will be displayed on any given request.

It’s true that some load testing tools will try to parse the pages in real time and figure out which images should be displayed, but that’s hardly comforting once you’ve already learned they can’t deal with even the most simply AJAX components, as we just saw.

Instead, the only way to guarantee that every single object (image, JavaScript, AJAX request, advertisement from an ad partner, etc) gets requested is to use a real web browser. And no one does that except for BrowserMob.

Tweet This Post 

No tags

We offer free and/or discounted load tests for qualifying non-profits, open source software projects, and other noble purposes.

We commercial customers, our pricing policy provides two price options: a pay-as-you-go model and an up-front annual subscription model. The subscription model provides a heavily discounted price per virtual user per hour.

If you believe you are going to run more than 2000 virtual user hours in a year (ie: ten 200 user tests for an hour), the discounts you will receive with the subscription package is well worth the investment.

To learn more, please visit our pricing information.

Tweet This Post 

No tags

Our policy is simple:

• Any unused credits can always be refunded, no questions asked.
• Any used credits will be refunded with no questions asked if the problem was due to BrowserMob-related issues (service outage, etc)
• Any credit refund from a result of improper usage, neglect, or failure to follow recommendations on our blog and/or documentation will be subject to a case-by-case basis.

The basic premise behind our policy is simple: we don’t want to tie you in or hold you hostage to our service at any time. If there is no cost to us already put in to a load test or an error occurred that was our fault, we’re happy to provide a refund immediately.

If there was a mistake on your end, such as running a test without having tested the script on a smaller test before hand, we’ll make a strong effort to make the situation right with either a partial or full refund.

In short: rest assured that we have your happiness and satisfaction as our top priority. If you have questions, please contact us.

Tweet This Post 

No tags

Today while browsing the SQA Forums, I found a question very relevant to BrowserMob: should I test on production?

There is no right/wrong answer to whether you should run a test on a production environment. In an ideal world, you’d have a staging environment that is exactly like your production one. But that is rarely actually the case.

Definitely do your initial performance testing internally first. Keep the tests simple and focussed, pinpointing specific parts of the app (certain parts of the UI, the database, etc). Sometimes these tests can be as simple as attaching a profiler (ie: YourKit for Java) to a developer’s desktop and just navigating through the app by hand a few times or playing back a Selenium IDE test 10 times in a row.

Often times you’ll find 80% of your performance problems with really simple, low-volume tests, since poor code usually stands out like a sore thumb even under low volumes.

Then it’s time to ensure the whole environment will work together. This is where a full scale load test comes in. If you have a test/stage environment that you can use, definitely start there.

If things look good, then the next question is: is my production environment different enough from my test/stage environment to warrant additional testing there?

It’s impossible to say for sure in your case, but in my experience, my test environments wouldn’t have a load balancer or redundant databases and web servers. In those cases, I considered that a large enough difference that I wanted to do a real world, production environment test. Otherwise, my load balancers could be broken and I wouldn’t necessarily know it until it’s too late. The same could be for my database clustering configuration or any number of other settings.

If you do decide to test on production, then you’ll need to analyze the risks:

• If users are allowed to access the site, they could experience major slowdown and/or errors. Is this acceptable?
• If users are allowed to access the site, it may be difficult for you to pinpoint the root cause of performance issues, especially if there are a lot of users doing different things.
• If users are not allowed to access the site, is the business prepared to allow the site to be unavailable for some time.
• If the load test generates a lot of data on the production database, it could cause slowdown that will continue even after the test is complete.

These are just some of the risks, but I hope it gives you an idea. In general, I tend to lean towards doing some sort of “real world” test. Afterall, that’s why we created BrowserMob in the first place!

Whether that test is your actual production environment or a close approximation of that environment, the key thing is that you analyze the risks and understand what you’re testing for. You’re not just testing the software now, but you’re testing the whole environment (load balancers, web servers, routers, etc).

Tweet This Post 

No tags