Due to requests from many of our users, we’ve rolled out two new features to BrowserMob:

1. The ability to specify “think time” for virtual users with the setSpeed and pause commands.
2. The option to reference the unique “virtual user” ID, which makes testing sites that don’t allow concurrent user logins much easier to test.

Here’s a brief summary of each feature. You can consult our documentation or our support team if you have any additional questions.

New Think Time Support

BrowserMob has always had support for the setSpeed and pause commands that are bundled in to Selenium, but until now those artificial delays encoded in the script would be interpreted as simply slow loading pages. Of course, this is not ideal for our users who wish to simulate real user behavior, complete with think time, but still want to know what the raw page load times were.

Now these commands will still work as expected, but instead the detailed data captured will differentiate between active and paused time. That is, you can determine how much time was spent actively interacting with the page (clicks, downloads, page load times, etc) and how much was spent waiting. This data is available at both the page level and the transaction level.

If you’re not familiar with the setSpeed and pause commands, check out the Selenium reference command documentation embedded inside of Selenium IDE. In short, setSpeed allows you to control the default pause between every single action (open, click, type, dragAndDrop, etc). The pause command is similar, but instead is a one-time pause rather than for every command.

The times specified for both is in milliseconds, so “setSpeed 5000” would instruct BrowserMob to wait five seconds in between each command. Similarly, “pause 3000” would tell BrowserMob to make all virtual users pause for three seconds when running tests.

Unique Virtual User Identification

Parameterizing/randomizing data in a load test is very common. BrowserMob supports this with some simple JavaScript code, as demonstrated in our recent video blog post. But sometimes random data won’t work. For example, some applications only allow one user to be logged in at a time. If we randomly selected usernames for each run of a test, there is a chance two virtual users could log in at the same time with the same username, causing an artificial test failure.

The solution to this problem is to support locking parameterized data to each virtual user. Then if you have 100 VUs in a test, then you’d have an array of 100 usernames, each one assigned a virtual user. BrowserMob now supports this technique with the new #{browserNum} variable.

For example, this command randomly types in one of 10 usernames in to a login field:

storeEval [‘bob’, ..., ‘mary’][Math.floor(Math.random() * 10)] randomUser
type username_field ${randomUser}

But if we wanted to ensure that each VU got it’s own username, we’d do:

storeEval [‘bob’, ..., ‘mary’][#{browserNum}] randomUser
type username_field ${randomUser}

The #{browserNum} variable will always be the same for each virtual user. It is also guaranteed to be unique across all active virtual users. With the above code, we’ve now ensured that no username will be used in overlapping test scripts.

Summary

We hope you enjoy these new features. Please remember that we’re always working to make BrowserMob the most realistic and easiest load testing service available. If you have feature ideas, please don’t hesitate to let us know.

Tweet This Post 

No tags

Firebug is a popular tool among web developers that helps with JavaScript debugging, CSS and HTML design, and web performance optimization. It’s truly a Swiss Army Knife for web developers. But it’s also a great tool for anyone doing web testing. Unfortunately, not all testers know about Firebug yet, so this is our attempt to share.

Whether you’re using BrowserMob for load testing, Selenium for functional testing, or any other open source or commercial traditional load testing tool, Firebug can be a huge asset. Three things we at BrowserMob find love about Firebug are:

1. The ability to view and verify network traffic, including AJAX requests, timings, byte sizes, hosts, etc.
2. Inspecting elements so that you can get a “feel” for how the web page is constructed. This also helps for building out XPath expressions when using Selenium for functional testing or for load testing with BrowserMob.
3. Evaluating JavaScript expressions to trigger internal events as well as to test expressions you might run in your Selenium test.

We’ve put together a movie that shows how you can use these parts of Firebug to enhance your QA efforts. This will be the first of many video segments we plan to do, so check back often!

Tweet This Post 

No tags

Scott Barber answered a question on SearchSoftwareQuality.com about how to conduct performance testing without using any tools. He goes on to state that being denied tools is likely a bad idea, but regardless of whether you have great tools (such as BrowserMob) or you have none, there are a lot of areas to focus on in addition to what a load testing tool does:

Now, after saying all of that, I must admit I have found that the vast majority of value that is gained by quality performance testing comes outside of the load-generation tool. Some of my favorite techniques (assuming you are testing websites):

Scott’s first two tips are probably the most important, both of which are discussed in the December 2007 issue of the AST Update magazine (PDF). Definitely check it out, as it discusses how to use Firebug and other tools to drastically cut down the end-user’s response time from the browser. Best of all, you can verify these improvements with BrowserMob!

Tweet This Post 

No tags

Steve Souders recently wrote up an excellent article on how to build high performance web sites. His advice is to focus on what the end-user experiences, which means focusing on everything from the HTML construction to browser caching to network infrastructure and delivery. He provides 14 “rules” to follow and it’s a great eye opener!

Steve is also the author of YSlow, a plugin for the popular Firebug tool. We strongly encourage all our customers to use Firebug and YSlow when doing any load and performance testing on their site. They are great tools to go along with Selenium IDE and BrowserMob.

PS: We’re going to be publishing a blog post on Firebug soon, so we’ll cover some of the topics that Steve mentions and how they relate to load testing with BrowserMob.

Tweet This Post 

No tags

Sanjay Dange wrote up an excellent summary of why QA professionals don’t have to hide from open source. You can find the article on his blog and reproduced at Test Republic. I made the following comment to his article and I think it’s worthwhile to share here as well, as it explains why we set out to create BrowserMob:

As a contributor to Selenium, an open source functional testing tool, I have also seen a lot more willingness to try out open source technologies.

I see the biggest challenge for adoption by QA engineers is the lack of documentation in open source. Normally, developers can pick up open source by just reading the source code, so documentation suffers. This has allowed libraries and frameworks to flourish.

But for tools, documentation is more critical. Unfortunately, this is usually the last thing to get attention in open source.

I see a chance for success with hybrid approaches: commercial companies that offer lower cost services and products that leverage open source. The commercial companies can provide support/documentation/training, but can keep their prices low and offer products that don’t have vendor lock in.

This is what I’m doing with my company, BrowserMob, which provides on-demand, low-cost load testing that uses real web browsers. We leverage Selenium and the power of cloud computing to offer a better product at lower prices. Without open source, our company couldn’t exist!

Tweet This Post 

No tags

A common question we get here at BrowserMob and elsewhere in the open source and QA communities we’re involved in is: “what’s the difference between stress testing, load testing, and performance testing?”

The three types of tests are often used interchangeably, though they do actually have somewhat different meanings. Fortunately, a friend of ours, Grig Gheorghiu, wrote about this very topic a few years ago in two blog posts:

• Performance vs. load vs. stress testing
• More on performance vs. load testing

Both of his posts are extremely good reads and should do a great job of clarifying the differences between the three. In summary, Grig argues (and we tend to agree with Grig) that:

• Performance testing is a process that focuses on testing individual components of the web app, such as databases, algorithms, network infrastructure, and cache layers.
• Load testing is a process of determining how an application under specific volumes of load, usually a range of the upper and lower limits expected by the business.
• Stress testing is a process of identifying when and how systems fail (and recover) under extreme levels of load.

BrowserMob can help with all three types of tests, though it’s best used for load testing and stress testing. Because it uses real browsers, BrowserMob is uniquely suited to assist with these two types of tests, since it is able to simulate the type of load that you would see from your real users.

In the case of performance testing, BrowserMob can assist, especially if you run low-level amounts of load (ie: 5-10 concurrent users) while also collecting performance metrics at various levels of your application. But there are other tools that can help too. Grig’s article provides a list of many tools he finds useful, but we’d also like to add one to his list: Firebug.

We’ll post a full blog entry on Firebug soon, but for know now that it is a “Swiss Army Knife” for web developers. It helps with CSS, HTML, and JavaScript development. But in addition to that, it also provides a simple ability to see the network timings of every object downloaded in a web page. When used in conjuction with clearing your cache (or clicking shift-reload), it can serve as a great way to see the HTTP performance of your application when there is no load on the system.

UPDATE: For another good list of definitions, check out Corey Goldberg’s Glossery of Terms for his Python-oriented testing tools.

Tweet This Post 

No tags

Selenium is a popular open source, cross browser functional testing tool. You can learn more about it here. If you’re looking for functional testing solutions that use real web browsers, we strongly encourage you to take a look at Selenium.

We have a close relationship with Selenium: BrowserMob’s founder, Patrick Lightbody, was the original creator of the most popular version of Selenium, Selenium Remote Control. So it’s not surprising that BrowserMob utilizes Selenium technologies for both the record and playback of web traffic.

Because Selenium is a functional testing tool, it isn’t really meant to be used as a load product. There are two reasons for that:

1. Selenium doesn’t capture and report the type of data that load testers want.
2. Selenium uses real browsers, meaning you need a ton of hardware to simulate a load test.

BrowserMob solves both of those issues and has successfully turned a popular functional testing tool in to an excellent load testing tool. This has an added benefit that if you are already using Selenium for functional testing, you likely already have scripts needed to do load testing.

Tweet This Post 

No tags

There is no hard or fast rule on how long your tests should be, since it all depends on what you’re trying to do. However, we can provide some general guidance as well as some tips to optimize your value with BrowserMob.

First, it helps to know a bit about how the application works, since that may very well determine how long you should run your test for.

For example, suppose that your application took a request at the end of a user transaction and put it in to an in-memory queue for processing later. This would mean that if your load test executes more transactions than the queue can process, memory will get consumed, eventually resulting in a failure. Therefore, it might be a good idea to run your test long enough to determine when that failure occurs.

On the other hand, if your application doesn’t have any characteristic that would require running a test for hours on end, a shorter test of 10-15 minutes of sustained load might be all you need. Talk with the developers and architects of the software to get a feeling for what type of testing is important.

You should also know early on what type of test you want to run. There are different types of tests that you might be interested in, such as a multi-day “burn in” test, a short “burst” test that only lasts a few minutes, or an hour long test designed to simulate real load during a peak hour in the day. Different types of tests have different lengths, so we encourage you to discuss this with your team.

Finally, it also helps to optimize your costs with BrowserMob. Currently, BrowserMob charges on a per-hour basis, rounding up to the nearest hour. This means that a 100 user test for 5 minutes costs the same as a 100 user test for 60 minutes. Therefore, we recommend that even if you need only a small 5 or 10 minute test, you schedule a longer one and take advantage of the “pause” feature.

By scheduling a longer test and/or configuring multiple phases at different load levels, you can save money. And if you need to stop the load activity for a few minutes while you configure services or reset the database, you can pause the active test first, do your work, and then resume the test activity.

If you have questions about how long your test should be or how many users you might need, please don’t hesitate to contact us.

Tweet This Post 

No tags

There is no clear cut answer to this, but we’re happy to provide guidance.

Your business liaison (usually a product manager, project manager, or analyst) may know the answer right away, since there may be specific business goals associated with the project.

The system architect may also have a good idea of what the capacity of the overall system is, so that might be another place to start asking.

You should also read our FAQ on hits/second vs. concurrent users, as that may help you better think about how many virtual users you need in a load test.

When in doubt, a good rule of thumb (but certainly far from accurate) is that a single web server can handle roughly 250 to 500 concurrent users. So if the environment you’re testing has two web servers behind a load balancer, it’s reasonable to expect it can handle between 500 and 1000 concurrent users from BrowserMob.

Warning! Even if you have reasonably good confidence that the system can handle hundreds or thousands of users, we strongly encourage you to start small, ramping up and scheduling several tests over time. This will almost certainly catch any simple performance errors early on, before you’ve spent a lot of time and energy on larger tests.

Tweet This Post 

No tags

Like most aspects of load testing, there is no easy or simple answer to this question. The answer often depends on what your goals are, how the application logic works, and what parts of the application you’re trying to test.

But in general, it’s important to clearly define various terms, which will help communication amongst your team and help you get to a clear goal before starting testing.

Some common words or phrases that are worth clarifying are:

• hits – A hit is most commonly known as an HTTP request to your website. It could be for a web page, an image, a style sheet, some JavaScript, a video file, etc. All hits are not equal: some may just need a simple image to be served, while others might cause complex application logic and database access.
• visits – A visit is usually referred to as the end-to-end interaction between a single user and your website.
• sessions – Sessions are similar to visits, but often have a more server-centric meaning. Most modern web application environments today have a concept of a server-side session, which is responsible for storing things like what is in the user’s shopping cart. Sessions often stay active for some time after the user’s visit is over, meaning additional memory is consumed until that session expires.
• virtual users – See this FAQ entry for more information on virtual users.
• concurrent users – Concurrent users can take on multiple meaning, but it most often represents how many virtual users are actively simulating a visit to your site.
• page views – A request to a web page by a browser. A page view always involves one hit (for the page itself) plus dozens of subsequent hits for images, JavaScript, CSS, etc that the page depends on.
• requests – An overloaded term that often means many things, such as a hit or a page view. We suggest using the term “hits” for raw HTTP requests and “requests” as a synonym for “page views”, or just avoiding the term “requests” entirely to avoid confusion.
• transactions – A single execution of the recorded script that the virtual user is supposed to run. A single virtual user will run as many transactions as it can (as long as the VU is active).

With these definitions out of the way, now you can ask yourself: what am I most concerned about testing?

If your application is mostly stateless, meaning it likely doesn’t consume much or any session state in memory, you are probably more interested in discovering what your peak hits/second, page views/second, or transactions/minute is.

If your application is not stateless or otherwise has significant background processing and load that is due to new sessions, you’re probably more interested in concurrent users.

Often you care about both, so thinking about these different terms and setting expected goals for all of them is important. Find out how many hits you expect for a particular visit or transaction. Determine how many total sessions you expect in a peak time period, but also how many concurrent users you expect at the same time. Then rank the metrics from most important to least important in terms of which you’d like to simulate closely.

With this information, you’ll be better prepared to schedule the appropriate load tests. Or, if you’re still unsure, feel free to share the information you have already gathered with our support team. We’d be happy to help advise you on how to proceed or get you in touch with one of our partners who can provide dedicated, hands-on support.

Tweet This Post 

No tags