Thanks to Dave Thompson for sharing his experience with using BrowserMob to test the performance of a Flex app. To read the full article, check out it here.

From 1:01 am to 3:01 am PDT on Sept 18, 2011 the data center that houses our secondary queue server became inaccessible from all monitoring locations. During the outage we were not running any monitoring jobs. At this time everything is running normally and our network ops team is working to determine the root cause.

Normally an outage of one of our queueing clusters would not be an issue, as we have both automatic and manual failover policies in place. But this time we were in the middle of migrating the physical server of our primary queue server and were unable to switch over. There are a few spots we have already identified where we can improve our infrastructure to prevent any further outages like this.

For 42 minutes tonight, 21:02 EDT to 21:44 EDT, Browsermob was unavailable from multiple locations.  We worked quickly to resolve the issue as soon as it was detected and are happy to announce that service has been restored.  If you have any questions for us, please let us know at support@browsermob.com.

From 9:25pm PDT through 10:02pm PDT tonight, monitoring was unavailable. An investigation is being performed right now to determine the root cause of this isolated issue.  We are very sorry for any inconvenience you may have experienced, and will learn from this incident to prevent this in the future.  If you have any questions, please contact us at support@browsermob.com.

Hello Readers!! Welcome back to Part 3 of our Load Testing Best Practices series, which focuses on writing good load testing scripts.

As I’m sure you would agree to, load testing is an imperative part of every development effort. One of the key components to your load testing efforts is a good load testing script that simulates real user behavior on your web site in the most realistic and accurate way possible.

Before you take a deep dive into the scripting, there are few things you need to plan ahead of time.

• The first one obviously is the scenario outline. Many clients seek the help of their analytics to identify their most visited pages.
• Secondly, identify a reasonable range of pause times between steps.
• As the next step, identify if your script will involve randomization or parameterization. If so, are you planning on using a datafile or an array of random values in the script?

If you have the answers for all of the above then you are half way through already as what’s remaining is packaging this into a script which is the objective of this post. This post is a step by step tutorial on how to write a good load testing script using the Selenium API and the BrowserMob platform. However, some of the tips and information can be used to help with script creation on other platforms as well. We will be focusing on how to organize your scripts, how and why to perform content validation, how to build robust and reusable scripts with the help of optimized selenium locators amongst other tips and guidelines on the best practices. I’m confident that at the end of this article you will be able to create your own load testing scripts in no time.

If you haven’t done so already, you are encouraged to install the following to assist in building the load testing scripts.

• Selenium IDE
• Firebug
• Xpath Finder

What makes a good load testing script?

1. Detailed Comments and Descriptions

Although the presence or absence of these is in no way going to affect the performance of your load test, it is always a good practice to include comments wherever possible as they make a good reference for later use, irrespective of the scripting language under use. It is easy to add comments while scripting with the IDE. Right click on the IDE and choose ‘Insert New Comment’.



Tip: All comments added in the IDE are automatically converted to “Step Descriptions” when the script is uploaded to BrowserMob. i.e the comment in the picture above would be converted to a BrowserMob step description like below:

browserMob.beginStep("Open CNN Home Page");

Giving each step/page a brief description rather than just “Step 1”, “Step 2” etc also helps find out easily how each step/page is performing during the load testing.



2. Content validation combined with proper use of ‘waitForPageToLoad’ and ‘waitForNetworkTrafficToStop’

“Content validation” is an essential part of every load testing script. Content validation is the only means to confirm the right page got downloaded or the right action was performed during a load test and this can be accomplished by searching for an appropriate piece of content in every step that confirms the validity of the page being downloaded. Selenium has various useful commands that can be used for content validation such as verifyTextPresent, VerifyElementPresent, assertTextPresent, assertElementPresent, waitForTextPresent, waitForElementPresent etc.

Tip: While recording actions with IDE, it is easy to add content validation at the appropriate places. Highlight the appropriate piece of content in each page load, right click and choose the appropriate content validation command as in the screenshot below:



Consider the snippets below:

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1a]

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForTextPresent("Latest news");
browserMob.endStep();

[Example: 1b]

Which one of the above is the right one?

The first one will fail immediately because the page needs to load completely or at least partially until the content “Latest news” becomes visible. The second one may not fail but is recommended only if you intend to measure the “Perceived Load Time” rather than the complete “Page Load Time”. What is the difference between the two? In the second snippet above (script 1b), the script is going to wait only until the content “Latest news” becomes visible before moving on to the next step. To a user that intends to load the CNN Home page and click on the first article under “Latest news”, this perceived load time is all they are going to care about. Whereas, the complete “Page Load Time” is measured only if the script waits for the page to load completely before moving on to the next step and this can be accomplished by using selenium API’s waitForPageToLoad and/or BrowserMob API’s waitForNetworkTrafficToStop.

Modifying the second snippet from above:

var timeout = 60000;
selenium.setTimeout(timeout);
 
browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1c]

When is BrowserMob API’s waitForNetworkTrafficToStop helpful?

Consider the scenario wherein the user loads the site united.com and tries to find the airport code using the site’s AutoSuggest feature.



Let’s consider the snippet below that tries to simulate the above scenario:

browserMob.beginStep("Step 1: United Airlines Home");
selenium.open("http://www.united.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent(“Travel information”);
browserMob.endStep();
 
browserMob.beginStep("Step 2: Choose Flights using AutoSuggest");
selenium.click("id=shop_from0_temp");
selenium.typeKeys("id=shop_from0_temp", "SAN ");
selenium.mouseOver("css=div#autoSuggestDiv > div");
selenium.clickAt("css=div#autoSuggestDiv > div", "");
………
browserMob.endStep();

[Example: 1d]

Running a validation for the script above threw the error “Element css=div#autoSuggestDiv > div not found”. What was the reason? Look at the waterfall chart for united.com Home page below:








The ‘waitForPageToLoad’ at Step 1 captured the traffic only until the document was complete. But as the waterfall chart above shows, several images, JavaScript files and iframes loaded asynchronously after the document was complete, especially the JavaScript file ‘airportSuggestion.js’ which was required for the autosuggestion at Step 2. Since the script simulated entering of the airport code ‘SAN’ even before the airportSuggestion.js was loaded, the autosuggest did not work as expected, resulting in a script failure.

Here is the modified script that works as expected:

browserMob.beginStep("Step 1: United Airlines Home ");
selenium.open("http://www.united.com/");
browserMob.waitForNetworkTrafficToStop(2000,timeout);
selenium.verifyTextPresent("Travel information");
browserMob.endStep();
 
browserMob.beginStep("Step 2: Choose Flights using AutoSuggest ");
selenium.click("id=shop_from0_temp");
selenium.typeKeys("id=shop_from0_temp", "SAN ");
// wait for the network traffic to stop and the autosuggest to populate
browserMob.waitForNetworkTrafficToStop(2000,timeout);
selenium.mouseOver("css=div#autoSuggestDiv > div");
selenium.clickAt("css=div#autoSuggestDiv > div", "");
browserMob.endStep();

[Example: 1e]

I’d like to highlight one more important aspect of the ‘waitForPageToLoad/waitFor[Text/Element]Present’ and ‘waitForPageToLoad/verify[Text/Element]Present’ combination. While both work fine it is recommended to use the combination waitForPageToLoad/verify[Text/Element]Present for load testing. Why?

Consider the snippets below:

var timeout = 60000;
selenium.setTimeout(timeout);
 
browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1f]

var timeout = 60000;
selenium.setTimeout(timeout);
 
browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.waitForTextPresent("Latest news");
browserMob.endStep();

[Example: 1g]

Both would exhibit the same behavior when the page loads successfully. However, in the instance of an error, say an ‘Internal Server’ error or a ‘Connection Reset’ error, the script would declare an error immediately after the error page loads with script 1f while it’ll wait for 60 more seconds to declare the same error with script 1g as the ‘waitForTextPresent(“Latest news”) is going to wait for 60 more seconds, which is the timeout we’ve set in the script, to declare the same content error, irrespective of the page’s status code. Besides,  script 1f will help simulate more transactions in the same amount of time than script 1g, especially in the event of errors.

3. Setting Proper Timeouts

The default selenium timeout is 30 seconds (30000 milliseconds) and the default TimeUnit in use by BrowserMob is milliseconds. It must be remembered that most of the ‘waitFor’ commands like the ‘waitForElementpresent’, ‘waitForVisible’, ‘waitFortextPresent’ etc wait for however long this default selenium timeout is set to. Timeouts can be set the following way in the script:

var timeout = 60000;
selenium.setTimeout(timeout);

It is recommended to set the timeout value before the actual transaction begins in the script. Studies show that the ideal page load time is 5 seconds. Any amount of load time greater than this is considered unacceptable for real users. However, 5 seconds is too small of a timeout threshold for load testing as load times are expected to rise with increase in user load. Hence, 60 seconds to 120 seconds would be considered ideal. The upper limit on the timeout threshold set by BrowserMob is 120 seconds.

Occasionally, you may have the necessity to set a different timeout value for different page loads which can be accomplished by the solution proposed in this article. The article on waitForCondition may also make an interesting read to learn more about advanced handling of timeouts.

4. Optimization of Locators

Selenium provides several ways of locating elements on pages. It may be worthwhile to read through this before proceeding further: http://release.seleniumhq.org/selenium-remote-control/0.9.2/doc/dotnet/Selenium.html

The most commonly used locator is the xpath. FireFox plugins like xpath finder, firebug may be used to identify the correct xpath for the appropriate elements. Below are a couple of useful articles on this topic:

http://blog.browsermob.com/2008/12/using-firebug-for-load-testing/

http://blog.browsermob.com/2009/04/test-your-selenium-xpath-easily-with-firebug/



Optimization of the locators used is extremely important for script robustness and re-usability. In the screenshot below, the IDE has captured the xpath ‘link=Arrested GI had gunpowder, ammo’ when clicking on the first article under “Latest News”. Obviously, the script will work only until this article is available on the Home Page. The xpath shown by firefbug for the same is ‘/html/body/div[5]/div/div[4]/div[3]/div/div[3]/div/ul[2]/li/a’ . Again, this might work temporarily but the moment the site is updated and the DOM changes, the script could break because of this complex xpath as the hierarchy of the elements could have been altered. ‘//ul[@class='cnn_bulletbin']/li[1]/a’ would be a much more optimized and a less brittle xpath for this example. Learn more about locators and pattern matching here.

5. Pause Times

Inserting pause times in between steps is essential in order to simulate real user actions more closely. A range of pause times is recommended over a fixed time as different users browse through a web site at different paces. Both BrowserMob and Selenium API’s pause method can be used to insert think/pause times in between steps.

6. Randomization

In the context of load testing, I want to emphasize the extreme importance of having large datasets available for testing. Many important bugs simply do not surface unless you deal with very large entities such thousands of users, thousands of mail server mailboxes etc. For scenarios involving product checkout, browsing through product pages etc, consider randomizing the product being purchased or browsed through, from a list of several products in order to simulate real user behavior more realistically as not all users are going to be purchasing the same product at the same time. Randomization and parameterization can be achieved in a few ways which are explained in depth here and here.

7. Use of quickStop() and stopExpected()

The BrowserMob API has two methods that come in handy when you want to pause a test that has exceptionally lengthy scripts running, such as loops that repeat certain actions several times over the course of the script execution.

By default, when a test is paused, BrowserMob will wait for the actively running scripts to finish executing and complete the current transaction. Long running scripts with loops or scripts testing a streaming video will not be interrupted until they complete. So in order to abort such transactions immediately on pausing, it is important to include the method browserMob.quickStop() at the top of the script.

However, forcefully aborting the scripts could result in an increased error rate, skewing the results. This can be avoided by using quickStop() in conjunction with stopExpected(). This instructs BrowserMob that the script is expecting to be stopped at any time and if that happens, the sudden stop should not be considered an error.

Here is a complete script, incorporating all of the above:

browserMob.quickStop();
browserMob.stopExpected();
 
var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
//  Set the timeout threshold
//  Set a different timeout threshold for local validation as it has a limit on the
// transaction length, capped at 2 minutes.
if (browserMob.isValidation()){
 	var timeout = 30000;
}
else
{
 	var timeout = 120000;
}
 
selenium.setTimeout(timeout);
 
//*************** Randomize think time *********************//
function randomXToY(minVal,maxVal)
{
 	var randVal = minVal+(Math.random()*(maxVal-minVal));
 	return Math.floor(randVal);
}
 
function sleep(x,y)
{
 	if(!browserMob.isValidation()){
  		browserMob.pause(randomXToY(x,y)*1000);
 }
 	else{
  		browserMob.pause(randomXToY(1,2)*1000);
  }
}
 
//********** Randomly pick an articleId, SearchString combination *************//
function randOption()
{
	var list = "1,PERMALINK;2,Related Links;3,Related Articles;4,Most Popular";
	var myArray = list.split(";");
	//pick a random combination from the array every time
	var position = Math.floor(Math.random()*myArray.length);
	var myArray1 = myArray[position];
	return myArray1;
}
 
var selected = randOption().split(",");
var id = selected[0];
var string = selected[1];
 
// Logging the variables being set provides a good means of troubleshooting
browserMob.log(id);
browserMob.log(string);
 
// Begin Transaction
transaction = browserMob.beginTransaction();
 
browserMob.beginStep("Open CNN Home");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();
 
sleep(2,5);
 
browserMob.beginStep("Click on a random article");
// Randomly clicking on an article with every script run
selenium.click("//ul[@class='cnn_bulletbin']/li["+id+"]/a");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent(string);
browserMob.endStep();
 
browserMob.endTransaction();

As web sites get complex, the scenarios for the load testing tend to get complex and so does the scripting and other load testing requirements such as configuration and set up. Our team of trained load testing experts is waiting to assist you with all your complex load testing requirements. Contact us right away and get all your load testing needs fulfilled!!

Hope you enjoyed reading this article as much as I enjoyed writing it. Coming up next in this Load Testing Best Practices blog series is “Test Execution” so be sure to stay tuned.

Happy Testing!!

More articlesIn the Driver’s Seat Series:
Part One: Planning
Part Two: Configuration

As ConnectEDU continues to expand, it must have reliable web apps. Anything less will hamper performance and profitability. With BrowserMob Performance Monitoring solution, the company is ready to sustain their growth—not just theirs but that of the students whose dreams they help achieve. Learn More.

Are you a start-up looking to get your rock star business out of a basement and into an office complete with technology solutions fit for an Enterprise? This year Contegix is hosting its national Start-Up, Boot-Up^TM competition, where one winner will be given more than $75,000 in “Enterprize” technology tools and office space.

Along with other solution providers, BrowserMob will award the winning start-up with a free load testing and monitoring plan, perfect for a website that is sure to get a ton of new traffic.

Interested? See below for the details.

Who: Contegix and BrowserMob

What: Start-Up, Boot-Up^TM (SUBU) competition

Where: Submit under the ‘Enter Now’ tab on the SUBU website

When: Fill out the entry form on the SUBU website by Sunday, April 10

The winner will be announced on Tuesday, May 3. Good luck! For more information check out the Start-Up, Boot-Up^TM website.

As you may have seen, Neustar’s BrowserMob Load Testing and FriendRunner announced last week that the two companies have joined forces to provide developers with a cost-effective solution to test Facebook applications before viral deployment to the community.  As such, we would like to hear from you regarding best practices and challenges you have overcome in developing applications for Facebook. Included below are details for the BrowserMob-sponsored contest, where you can simply send in a 200-word entry in order to win a $100 Amazon gift card!

Who: BrowserMob and FriendRunner

What: Application developers highlighting best practices with developing Facebook applications for community use – WINNERS receive $100 Amazon Gift Card!

Where: Submit within the “request more information section”

When: Deadline for submissions is 5pm ET on Tuesday, April 5th

How: Provide a 200 word overview of best practices for developing Facebook applications and describe obstacles you have overcome within the development phase. Winners will be announced on Friday, April 8th.

Good Luck!

As you may have seen, we announced some exciting news today: Neustar’s BrowserMob Load Testing and FriendRunner have joined forces. With these platforms combined, developers will be better able to test Facebook applications and ensure they are ready for viral deployment before launching them publicly to the Facebook community.

So why did we form this relationship? Well, Facebook’s unique architecture makes it more challenging to perform traditional load testing. However, with this solution, application developers can run load tests easily and efficiently, all while staying within budget.

BrowserMob’s loyal customers know that it’s Load Testing helps to identify web performance bottlenecks, get a handle on how much traffic your site can handle, and make sure that your applications are ready for launch. FriendRunner is a drop-in replacement for the Facebook servers, providing a cloud-based platform developed by Sericon Technology for load testing Facebook applications outside the Facebook system.

Some of the key  features includes:

• Selenium open-source toolkit
• Export MySQL
• Fast set-up — start testing in minutes, plus run a free test immediately
• Supports testing of flash applications
• And More….

Combined, these two solutions provide a robust tool for application and web developers alike.  Don’t wait — learn more about this solution at http://bit.ly/igkOUe and get started testing applications today!

Happy Holidays from the BrowserMob crew! I hope everyone has a chance to spend some quality time with friends and family this week, and a few moments away from the keyboard.

We’re taking an opportunity while everyone is distracted to perform a few performance upgrades on the site. The maintenance window is scheduled for a 12 hour period on Sunday December 26, from noon to midnight PST. We don’t expect more than a few minutes of actual interruption to any individual service and the website will remain accessible during the maintenance window. Monitoring will continue as expected the entire time, with a small possibility of delayed alerts. We will leave the maintenance notification up until all upgrades have been completed. Here’s a rundown of what we are changing and how it might effect you.

Upgrades to the website: browsermob.com will go offline for about 5 minutes at the beginning of the maintenance window so that we can release the latest code. Once rebooted, all of the normal functionality will be available except that you won’t be able to schedule a load test until maintenance is complete. Our priority is to make sure our customer’s needs are met, and if you have a requirement to run a test during this time we will be sure to accommodate, just give us a call (1 503 828-9003) or send us an email

Upgrades to load testing: We’re in the middle of improving the load testing infrastructure with the aim of increasing efficiency and ultimately providing our customers with the best features and pricing possible. The load testing updates this time around will go largely unnoticed, as it is mostly back-end changes, but tests will be more reliable and it will set us up for some bigger changes to come.

Upgrades to monitoring: Each account’s monitoring database will be upgraded to increase performance. While the upgraded is being performed, monitoring charts will be unavailable and new transactions will be delayed. The entire process should take less than five minutes for each account, before and after which you’ll be able to access your monitoring charts as usual. Just to be clear, your monitoring will not be interrupted, it will continue to run as scheduled throughout the entire monitoring window. There is a slight chance that if a site you are monitoring goes down at the same time we’re upgrading your particular account, you will receive an email notification a few minutes later than expected. After which you should notice a marked improvement in the speed of your monitoring charts and reports.

Please feel free to contact us with any questions or concerns during this period. Our aim was to pick a window that would impact as few users as possible, but we are definitely here to make sure everyone is taken care of. Enjoy the holidays and have a wonderful end to 2010!