Our Professional Services team has been extremely busy over the past several months helping Customers prepare for what proved to be a very busy online shopping season. I’d like to share with you a couple of unique challenges that we came across and how we were able to use the BrowserMob platform and some special customizations in order to successfully test what normally would be impossible using just Selenium or a traditional http-based testing approach.

File Uploads

We worked with a large online greeting card company to completely automate the creation of a card, complete with multiple file uploads. Typically, we see file uploads implemented using a standard web form which are easy to automate with Selenium and the BrowserMob filePath directive. However, in this case, the upload used a Flash component to control the upload which then presented a native OS dialog to the user to select the file as in the screenshot below.

Unfortunately, these types of dialogs are not accessible by Selenium and thus require native mouse and keyboard integration as mentioned in this this earlier blog post. As the interaction was relatively simple (just select the file to be uploaded), we were able to automate this using BrowserMob’s special native keyboard and mouse utilities.

As a result, we were able to completely automate the creation of a card from start to finish and the Customer was subsequently able to integrate this script into their regular testing process. Please keep in mind that this type of interaction is quite brittle as it relies on specific x,y coordinates to be clicked on and can cause the script to break over time as objects are re-positioned on the site. In this case, as the interaction was relatively simple, it worked out well. For more complex interactions using Flex, we strongly recommend checking out Gorilla Logic’s solutions as mentioned in this article.

Streaming Media

Another Customer came to us with a requirement to simulate a streaming media event for hundreds of users they were hosting using a Windows Media infrastructure. We were able to customize the boot-up process of our standard load testing agents and install the Mplayer plugin for Firefox which allowed us to stream the content across hundreds of browser and multiple locations. The Customer was then able to diagnose performance issues and bottlenecks in advance of the actual live event, saving them a lot of time and effort in the process.

During this engagement, we also used a special statistical sampling technique to verify that the content was playing as expected. In the script, we purposely threw an error, approximately 5% of the time which then took a screenshot and video of the transaction which we were able to review in real-time along with the Customer’s team. The relevant script snippet to accomplish this is included below:

function randomid(minVal,maxVal)
{
  var randVal = minVal+(Math.random()*(1+maxVal-minVal));
  return Math.floor(randVal);
}

if (!browserMob.isValidation()) {
    	var id = randomid(1,1000000000);
    	//randomly take a screenshot 5% of the time
    	var shot = randomid(1,20);
    	if (shot == 5) {
		throw("Catch body broken: IOException " + "user id " + userid + "-" + id + " " + new Date().toString());
        }

What type of special testing requirements do you have that are complex and challenging? Contact us and let us know, as we can help you come up with a solution!

Consider scheduling your load tests for up to 60 minutes. You can always stop the test earlier.  By scheduling it for 60 minutes, you retain the option to run the test longer or even pause/resume the test, effectively getting a second test for free. The cost of a shorter test, i.e. 5, 10, or 15 minutes, is the same as a 60 minute test, so there’s really nothing to loose.

We are happy to announce that we recently added support for Firefox 7 to our monitoring and load testing products! You can now explicitly specify the Firefox version your script will run against.  Currently we support both Firefox 3.6 and Firefox 7.

For example, the following script will open the Firefox 7 browser and navigate to a url:

var selenium = browserMob.openBrowser("FF7");
browserMob.beginTransaction();
browserMob.beginStep("Home Page");
selenium.open("http://seleniumhq.org/");
browserMob.endStep();
browserMob.endTransaction();

The same is possible when using the Selenium 2.0 API (WebDriver):

var webDriver = browserMob.openBrowserWebDriver("FF7");
browserMob.beginTransaction();
browserMob.beginStep("Home Page");
webDriver.get("http://seleniumhq.org/");
browserMob.endStep();
browserMob.endTransaction();

Firefox 3.6

Existing monitors and load tests will continue to use Firefox 3 by default.  New monitors will default to using Firefox 7, so you if you want to create a new monitor and have it run against Firefox 3.6, specify “FF3″ as the browser type:

var selenium = browserMob.openBrowser("FF3");

Future Versions

In the future, we will add support for more Firefox versions. If you want to keep up with the latest version of Firefox installed on our systems, you will need to  specify “FF”:

var selenium = browserMob.openBrowser("FF");

When we upgrade to a later version of Firefox, the script will automatically use the new version.

Local Validator

Local validator will use the Firefox version you have installed by default. You can override the locations it looks in for the different browsers in a property file. The file is located in your home directory on Unix and under your user folder on Windows.

$HOME/.browserMob/config.properties

For example, if you have Firefox 7 installed in the default path and Firefox 3 under a different directory,  you can use the following config.properties to point local validator to the Firefox executables:

FF3 = C:/Program Files/Mozilla Firefox 3/firefox.exe
FF7 = C:/Program Files/Mozilla Firefox/firefox.exe

Hello Readers!! It gives me immense pleasure in meeting you with yet another article on WebDriver. WebDriver has been creating a lot of buzz among the Selenium community lately and BrowserMob, an active member of the Selenium community, wasted no time in announcing its support for the WebDriver API.

I hope you had the chance to read the great introduction article on the Webdriver API by fellow blogger Ben.

In this article I would like to touch on the basics of scripting with the WebDriver and the BrowserMob API.
I’m sure many of you, who are new to this API are asking yourselves these questions just like I did.

• What is WebDriver?
• Why is it needed?

WebDriver is a framework for automated testing of web applications. WebDriver is a part of Selenium 2.0, which is a major departure from the 1.0 model. Selenium 1.0 is a client-server architecture which uses the RC libraries to program tests that communicate with the server, and the server relays those commands to a browser.

The primary downside of the 1.0 model is that it is written in JavaScript. Browsers impose a strict security model on any JavaScript that they execute in order to protect a user from malicious scripts. Additionally, the API for RC has grown over time that users find it hard to keep up with this ever growing dictionary of methods. The WebDriver helps overcome both these hurdles with an object oriented API that controls the browser itself rather than running as a JavaScript application within the browser.

WebDriver has five major Interfaces – ImeHandler, Navigation, Options, TargetLocator , Timeouts, the methods of which fall into following three categories.

• Control of the browser itself

  Eg: get(), navigate(), close()etc

• Selection of WebElements

  Eg: findElement(), findElements() etc

• Debugging aids

  Eg: getCurrentUrl(), getPageSource(), getTitle() etc

Now that we have a clear understanding of what WebDriver is and what its benefits are, let’s take a deep dive into the various useful WebDriver methods exposed via BrowserMob through some example scripts. Let’s look at this simple script below, to begin with.

var driver = browserMob.openBrowserWebDriver();
 
var tx = browserMob.beginTransaction();
 
var step = browserMob.beginStep("Home Page");
 
driver.get("http://seleniumhq.org/docs/06_test_design_considerations.html");
 
// polls DOM every 500 ms for ten seconds until the text 'Test Design Considerations' is found
driver.manage().timeouts().implicitlyWait(10,  TimeUnit.SECONDS);
 
// Content validation - find element by xpath
driver.findElement(By.xpath("//div[@id='test-design-considerations']/h1[contains(text(), 'Test Design Considerations')]"));
 
browserMob.endStep();
 
browserMob.endTransaction();

Deciphering the script…

• var driver = browserMob.openBrowserWebDriver();
  The openBrowserWebDriver()method of WebDriver starts a WebDriver browser session.
• driver.get(“http://seleniumhq.org/docs/06_test_design_considerations.html”);
  The ‘get’ method which is equivalent to selenium.open() is used to navigate to any web page.
• driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
  driver.findElement(By.xpath(“//div[@id='test-design-considerations']/h1[contains(text(), 'Test Design Considerations')]“));

  WebDriver has a blocking API. i.e. WebDriver will wait until the page has fully loaded (that is, the “onload” event has fired) before returning control to your test or script. However, under some conditions it is possible for a ‘get’ call to return before the page has finished loading. The classic example is JavaScript starting to run after the page has loaded (triggered by onload). Browsers (e.g. Firefox) will notify WebDriver when the basic HTML content has been loaded, which is when WebDriver returns. It’s difficult to know when JavaScript has finished executing, since JS code may schedule functions to be called in the future, depend on server response, etc. WebDriver cannot wait for all conditions to be met before the test proceeds because it does not know them. The solution to ensure whether the element involved in the next interaction is present and ready is to use the ‘Wait’ class to wait for a specific element to appear. This class simply calls findElement over and over, until the element is found (or a timeout has expired). Since this is the behavior desired by default for many users, a mechanism for implicitly-waiting for elements to appear has been implemented. This is accessible through the Timeouts() interface of the WebDriver class (WebDriver.manage().timeouts() ). In the example script above, the driver.manage.timeouts().implicitlyWait call polls the DOM every 500 ms for 10 seconds until the text ‘Test Design Considerations’ is found.

Tips:

• Optionally you can reset the Implicit Timeout to zero using the code below, as once set, it remains effective until the life of the WebDriver instance. The default setting is zero, i.e zero wait time.

  driver.manage().timeouts().implicitlyWait(0,  TimeUnit.SECONDS);

• It is possible to set a timeout for the entire step as such using the setStepTimeout method of the browserMob API.

  // Setting a step timeout of 10 seconds
  browserMob.setStepTimeout(10000);
   
  browserMob.beginStep();
  …….
  browserMob.endStep();

• Alternatively, browserMob’s waitFor() method could be used in place of the implicitlyWait() for content validation. Modifying our script with waitFor()…

  var step = browserMob.beginStep("Home Page");
  driver.get("http://seleniumhq.org/docs/06_test_design_considerations.html");
  browserMob.waitFor(function() {
        return check = driver.findElement(By.xpath("//div[@id='test-design-considerations']/h1")).getText().toLowerCase().startsWith("test design considerations");
      }, 10000);
  if (!check)
  	throw "Content validation failed";
  browserMob.endStep();

Great!! We just wrote our first WebDriver script. We now know how to interact with page elements via the findElement method of WebDriver’s WebElement Interface using XPath. But does WebDriver provide other alternatives for interaction? Absolutely. Below are few alternative ways. As implied through the script, using the ‘findElement’ is a great way to validate page content before proceeding to the next step in the script.

By.cssSelector:
driver.findElement(By.cssSelector(“div#test-design-considerations > h1″));

By.linkText:
driver.findElement(By.linkText(“Data Driven Testing”));

By.partialLinkText:
driver.findElement(By.partialLinkText(“Data Driven”));

By.id:
driver.findElement(By.id(“side”));

By.className:
driver.findElement(By.className(“expandable”));

By.name:
driver.findElement(By.name(“q”));

By.tagName:
driver.findElement(By.tagName(“input”));

driver.findElements:
In situations where you have to iterate through multiple elements with the same id/class/tag name etc or have to pick one randomly, the ‘findElements’ method comes in handy. The example script below uses this method to pick a random suggestion from an AutoSuggest list.

browserMob.beginStep("AutoSuggest");
 
driver.get("http://www.simon.com/default.aspx");
 
var elem = driver.findElement(By.cssSelector("input[id*='KeywordTextBox']"));
 
// clear the search box first
elem.clear();
 
// type Carlsbad in the search box
elem.sendKeys("carlsbad");
 
// wait for autosuggest to populate
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
 
var autoSuggest = driver.findElements(By.xpath("//div[@class='ac_results']/ul/li"));
 
// 'autoSuggest' from above is a Java List. Either use size() to get its length 
// or convert this to an array - autoSuggest.toArray().length
 
browserMob.log(autoSuggest.size());
 
// click on a random element – picking the last element in this case
var selected = driver.findElement(By.xpath("//div[@class='ac_results']/ul/li["+autoSuggest.size()+"]")).click();
 
// throw an error if content validation returns false
if (!driver.getTitle().contains("Search Results"))
	throw 'Content error: "Search Results" not found';
 
browserMob.endStep();

As you can see, I’ve also managed to introduce a few other useful methods through this script. The comments above each one of them describe their purpose.

Tips:

• sendKeys can also be used to simulate pressing of the keyboard keys by using the “Keys” class. It is possible to call sendKeys on any element. The code below triggers the “ENTER’ key press to force a form submission. The list of Keys supported can be found here.

  driver.findElement(By.id('finalExpenses')).sendKeys(Keys.RETURN);

• With advanced locators, it is possible to traverse down the DOM with a single line of code as in below:

  driver.findElements(By.tagName("ul")).get(1).findElements(By.tagName("li")).get(2).click();

  As the index starts with zero, the line of code above would click on the third “li” of the second “ul” on the page.

A few more common page element interactions in detail…

How to handle frames?

Swinging between frames is easy with the switchTo() method of the TargetLocator interface of the WebDriver class.

• Switch to the frame from main window
• By index

  driver.switchTo().frame(0);

• By frame name

  driver.switchTo().frame("frameName");

• Switch back to the parent window

  driver.switchTo().defaultContent();

The following example illustrates a simple swinging between two frames on a web page. Let’s assume the page has two frames, frame1 and frame2, out of which frame1 has a link and frame2 has a form.

driver.switchTo().frame("frame1");
 
driver.findElement(By.linkText("SPAZ")).click();
 
driver.switchTo().defaultContent();
 
driver.switchTo().frame("frame2");
 
driver.findElement(By.id("field_0")).sendKeys("anu");
 
driver.findElement(By.className("form_button")).submit();
 
driver.switchTo().defaultContent();

Try running this snippet commenting out the “driver.switchTo().defaultContent();” before switching to frame2 and you would end up with an error that frame2 cannot be located. This explains the need for switching back to the parent page from inside a frame first, before navigating to another frame on the same page.

How to handle JavaScript Alert and Confirmation boxes?

Confirmation boxes are handled the same way as the Alerts.

• Switch focus to the alert/confirmation box

  var alert = driver.switchTo().alert();

• Dismiss – equivalent to hitting “Cancel” button on the Confirmation box or ignoring an Alert by closing it.

  alert.dismiss();

• Alternatively accept – equivalent to hitting “OK” on the Alert or the Confirmation box

  alert.accept();

• Handle Confirmation boxes the JavaScript way

  driver.executeScript("window.confirm = function(msg){return true;};");

How to handle Pop-up Windows?

browserMob.beginStep("Handle Popup Windows");
 
driver.get("http://www.javascript-coder.com/files/window-popup/javascript-window-open-example1.html");
 
// Get the handle to the current window
var currentHandle = driver.getWindowHandle(); 
 
driver.manage().timeouts().implicitlyWait(10,  TimeUnit.SECONDS);
 
// Open Pop-up
driver.findElement(By.partialLinkText("Open the JavaScript Window")).click();
 
// Switch focus to Pop-up using its name
driver.switchTo().window('mywindow');
 
// Alternate way - if the Pop-up window doesn't have a name, which is very common - 
// get the handle to the last opened window
var handles = driver.getWindowHandles().toArray();
var id = handles.length - 1
driver.switchTo().window(handles[id]);
 
//close popup
driver.executeScript("self.close ();");
 
//switch focus to parent window
driver.switchTo().window(currentHandle);
 
browserMob.endStep();

How to interact with SELECT drop-downs?

var lastItem = driver.findElement(By.xpath("//select[@class='dropDown']/option[last()]"));
 
lastItem.click();

How to simulate advanced interaction such as Drag and Drop?

The Actions subclass of the Selenium Interactions class makes emulation of these complex user gestures easier.

• Drag an element to drop on to another element

  driver.get("http://devfiles.myopera.com/articles/735/example.html");
  var from = driver.findElement(By.xpath("//ul[@id='Fav']/li[3]"));
  var to = driver.findElement(By.xpath("//ul[@id='Tol']/li"));
  Actions(driver).dragAndDrop(from,to).build().perform();

• Drag an element to an offset

  driver.get("http://jqueryui.com/demos/draggable/");
  var draggable = browser.findElement(By.id("draggable")); 
  Actions(driver).dragAndDropBy(draggable, 200, 10).build().perform();

I hope this article served as a comprehensive tutorial, covering most of the basics of WebDriver, if not all. See you soon with more useful WebDriver tips and tricks. Happy testing until then!!!

Select any time coordinate in your test plan and simply add a comment in the annotation field.  Your annotations are tracked and numbered with a flag, so to review, simply reference the number on the flag in the list of annotations made for that test plan.  Don’t worry about making mistakes, you can delete an annotation at any time.

Do you know where your website traffic is coming from? How about the percentage of traffic coming from various regions?

A new report from, comScore, a market research firm, found that nearly 24.1% of total number of users visiting Amazon’s retail website in the month of June, came from the Asia Pacific region. Similarly, 24.9% of the total traffic on Apple’s website are from this same region.

As companies continue to expand their brands globally, it becomes increasingly important to keep track of where traffic is coming from, as load times and performance may vary depending on the region.

So how can you make sure you are running load tests that encompass traffic from around the world? Select a load testing tool that will allow you to test various traffic scenarios from around the world. For instance,  BrowserMob offers Tokyo as a location from which you can generate traffic for your load test, as well as five other locations around the world.

For more information on load testing best practices, read the series: http://blog.browsermob.com/tag/load-testing-best-practices/

Hello Testers – First, I have to admit that I am not a Flex/Flash developer and cannot take credit for the following content. This was produced by my colleague and fellow BrowserMob blogger, Anu. It is documented internally but I believe the BrowserMob community will find value so I scrubbed it and published it.

Overview

Traditionally, testing Flex/Flash applications was difficult because the logic/behavior is encapsulated from the browser. With the introduction of Flex 3.x,  came the release of the Adobe Automation Framework; and Gorilla Logic developed a lightweight testing application called FlexMonkey. In conjunction, a Selenium plugin was created – FlexMonkium. Now Flex recording and playback are seamlessly interleaved with native Selenium recording and playback so you can easily automate the testing of hybrid web applications that mix HTML and Javascript with Flex.

Prerequisites

• In order to enable selenium interaction with Flex, you will have to compile your Flex objects with the FlexMonkey libraries. If you do not have access to your  swf files or don’t have the ability to compile swf files with FlexMonkey, then this solution may not be  suitable.
  • More on the compilation here: http://www.gorillalogic.com/books/flexmonkey-docs/flexmonkey-quick-start

• FlexMonkium needs to be installed.
  • FlexMonkium is a bridge between FlexMonkey and Selenium that enables creating and running FlexMonkey test cases through the Selenium IDE.
  • Installation Instructions:
    • Install the Selenium-IDE.
    • Install the FlexMonkiumConsole by running the FlexMonkium installer (requires registration). Please note the location of your executable because you will need to tell the Firefox plugin where to find it. When the installation is complete,  the FlexMonkiumConsole window will appear (a small application window with three buttons); you can close it for now.
    • Download and unzip the FlexMonkium zip file (requires registration). It contains:flexmonkium.xpi – The FlexMonkium Firefox extension installer; user-extensions.js – The FlexMonkium extensions for Selenium-RC.
      • Install the .xpi file by opening Firefox and selecting File–>Open from the File menu.
• If installed properly, a small black icon should appear on restarting firefox and opening the selenium IDE as indicated in the picture below.

• If the console does not appear, select Options > FlexMonkium… from the Selenium menu, click on the Browse… button and navigate to where you installed the FlexMonkiumConsole executable.
• Click on the little black FlexMonkey icon in the IDE. It should open up a small console. If the .swf files were compiled properly with the FlexMonkey libraries, a little green ball should appear in the console, as indicated in the picture below



• It shows up in yellow, otherwise

It is very important to add “verifyFlexMonkey” commands wherever applicable to ensure that the Flex loaded completely and is ready to move on to the next step.

FlexMonkey methods that are supported currently by BrowserMob

• input(String id, String value);

• changeFocus(String id);

• open(String id);

• change(String id, String value);

• click(String id);

• verifyProperty(String id, String propertyType, String value);

• selectText(String id, int StartPositionOfString int EndPositionOfString);

• select(String id,String Value);

• getProperty(String id, String propertyType);

IDE Script to BrowserMob script conversion

Currently automatic conversion of the IDE script to BrowserMob script isn’t available. Manual conversion is required.

• First script the site with selenium IDE.
• Save the script and upload to BrowserMob.
• The BrowserMob validation will fail because BrowserMob doesn’t do the automatic conversion of the FlexMonkey markup into BrowserMob compatible code. Copy/paste your  script to a favorite text editor and manually convert the script to a BrowserMob compatible version.
• Whenever the script goes to a new page, a new handle needs to be created to the Flex object like below:

browserMob.beginStep("Step 1: Home");
selenium.open("https://www.domain.com");
// create handle to the flex object - This waits for however long
// the selenium timeout is set, to create the handle
// in other words, this line of code waits for the however long
// the selenium timeout is set, for the swf to load on the page.
// SWF is the id of the swf
var fm = selenium.flexMonkey("SWF");
....

• Manually convert the rest of the FlexMonkey markup as illustrated below:
  • verifyProperty(String id, String propertyType, String value);

  <Verify value=”trendChart” prop=”automationName”><Attribute name=”zoom” expectedValue=”1″ type=”property”/></Verify>

  // SWF is the id of the swf
  var timeout = 120000;
  //creating a handle for the flex object
  fm = selenium.flexMonkey("SWF");
  browserMob.waitFor(function () {
   try {
    fm.verifyProperty("trendChart", "zoom", "1");
    success = true;
    return success;
   }
   catch (e) {
    exception = e;
    success = false;
    return success;
    }
   }, timeout);
  if(!success){
    browserMob.log(success);
    transaction.setSuccess(false);
    throw exception+" after " + timeout + " ms";
  }

  • change(String id, String value); (used when changing tabs)

  <UIEvent command=”Change” value=”bar” prop=”automationName”><arg value=”Current Bill”/></UIEvent>

  fm = selenium.flexMonkey("SWF");
  fm.change("bar","Current Bill");

  • input(String id, String value); (form inputs)

  <UIEvent command=”Input” value=”password” prop=”automationName”><arg value=”abc”/></UIEvent>

  fm = selenium.flexMonkey("SWF");
  fm.input("password", pw);

  • selectText(String id, int StartPositionOfString int EndPositionOfString); (used to select form inputs before inputting any text)

  <UIEvent command=”SelectText” value=”password” prop=”automationName”><arg value=”0″/><arg value=”6″/></UIEvent>

  fm = selenium.flexMonkey("SWF");
  fm.selectText("password", 0, 6);

  • click(String id);

  <UIEvent command=”Click” value=”Logout” prop=”automationName”/>

  fm = selenium.flexMonkey("SWF");
  fm.click("Logout");

Hopefully this provides a starting point for external monitoring and/or testing of Flex applications. It should be noted that Webmetrics also offers the capability for monitoring Silverlight and Flash with the Visual Monitoring service.

Hello Readers!! Welcome back to Part 3 of our Load Testing Best Practices series, which focuses on writing good load testing scripts.

As I’m sure you would agree to, load testing is an imperative part of every development effort. One of the key components to your load testing efforts is a good load testing script that simulates real user behavior on your web site in the most realistic and accurate way possible.

Before you take a deep dive into the scripting, there are few things you need to plan ahead of time.

• The first one obviously is the scenario outline. Many clients seek the help of their analytics to identify their most visited pages.
• Secondly, identify a reasonable range of pause times between steps.
• As the next step, identify if your script will involve randomization or parameterization. If so, are you planning on using a datafile or an array of random values in the script?

If you have the answers for all of the above then you are half way through already as what’s remaining is packaging this into a script which is the objective of this post. This post is a step by step tutorial on how to write a good load testing script using the Selenium API and the BrowserMob platform. However, some of the tips and information can be used to help with script creation on other platforms as well. We will be focusing on how to organize your scripts, how and why to perform content validation, how to build robust and reusable scripts with the help of optimized selenium locators amongst other tips and guidelines on the best practices. I’m confident that at the end of this article you will be able to create your own load testing scripts in no time.

If you haven’t done so already, you are encouraged to install the following to assist in building the load testing scripts.

• Selenium IDE
• Firebug
• Xpath Finder

What makes a good load testing script?

1. Detailed Comments and Descriptions

Although the presence or absence of these is in no way going to affect the performance of your load test, it is always a good practice to include comments wherever possible as they make a good reference for later use, irrespective of the scripting language under use. It is easy to add comments while scripting with the IDE. Right click on the IDE and choose ‘Insert New Comment’.



Tip: All comments added in the IDE are automatically converted to “Step Descriptions” when the script is uploaded to BrowserMob. i.e the comment in the picture above would be converted to a BrowserMob step description like below:

browserMob.beginStep("Open CNN Home Page");

Giving each step/page a brief description rather than just “Step 1”, “Step 2” etc also helps find out easily how each step/page is performing during the load testing.



2. Content validation combined with proper use of ‘waitForPageToLoad’ and ‘waitForNetworkTrafficToStop’

“Content validation” is an essential part of every load testing script. Content validation is the only means to confirm the right page got downloaded or the right action was performed during a load test and this can be accomplished by searching for an appropriate piece of content in every step that confirms the validity of the page being downloaded. Selenium has various useful commands that can be used for content validation such as verifyTextPresent, VerifyElementPresent, assertTextPresent, assertElementPresent, waitForTextPresent, waitForElementPresent etc.

Tip: While recording actions with IDE, it is easy to add content validation at the appropriate places. Highlight the appropriate piece of content in each page load, right click and choose the appropriate content validation command as in the screenshot below:



Consider the snippets below:

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1a]

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForTextPresent("Latest news");
browserMob.endStep();

[Example: 1b]

Which one of the above is the right one?

The first one will fail immediately because the page needs to load completely or at least partially until the content “Latest news” becomes visible. The second one may not fail but is recommended only if you intend to measure the “Perceived Load Time” rather than the complete “Page Load Time”. What is the difference between the two? In the second snippet above (script 1b), the script is going to wait only until the content “Latest news” becomes visible before moving on to the next step. To a user that intends to load the CNN Home page and click on the first article under “Latest news”, this perceived load time is all they are going to care about. Whereas, the complete “Page Load Time” is measured only if the script waits for the page to load completely before moving on to the next step and this can be accomplished by using selenium API’s waitForPageToLoad and/or BrowserMob API’s waitForNetworkTrafficToStop.

Modifying the second snippet from above:

var timeout = 60000;
selenium.setTimeout(timeout);
 
browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1c]

When is BrowserMob API’s waitForNetworkTrafficToStop helpful?

Consider the scenario wherein the user loads the site united.com and tries to find the airport code using the site’s AutoSuggest feature.



Let’s consider the snippet below that tries to simulate the above scenario:

browserMob.beginStep("Step 1: United Airlines Home");
selenium.open("http://www.united.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent(“Travel information”);
browserMob.endStep();
 
browserMob.beginStep("Step 2: Choose Flights using AutoSuggest");
selenium.click("id=shop_from0_temp");
selenium.typeKeys("id=shop_from0_temp", "SAN ");
selenium.mouseOver("css=div#autoSuggestDiv > div");
selenium.clickAt("css=div#autoSuggestDiv > div", "");
………
browserMob.endStep();

[Example: 1d]

Running a validation for the script above threw the error “Element css=div#autoSuggestDiv > div not found”. What was the reason? Look at the waterfall chart for united.com Home page below:








The ‘waitForPageToLoad’ at Step 1 captured the traffic only until the document was complete. But as the waterfall chart above shows, several images, JavaScript files and iframes loaded asynchronously after the document was complete, especially the JavaScript file ‘airportSuggestion.js’ which was required for the autosuggestion at Step 2. Since the script simulated entering of the airport code ‘SAN’ even before the airportSuggestion.js was loaded, the autosuggest did not work as expected, resulting in a script failure.

Here is the modified script that works as expected:

browserMob.beginStep("Step 1: United Airlines Home ");
selenium.open("http://www.united.com/");
browserMob.waitForNetworkTrafficToStop(2000,timeout);
selenium.verifyTextPresent("Travel information");
browserMob.endStep();
 
browserMob.beginStep("Step 2: Choose Flights using AutoSuggest ");
selenium.click("id=shop_from0_temp");
selenium.typeKeys("id=shop_from0_temp", "SAN ");
// wait for the network traffic to stop and the autosuggest to populate
browserMob.waitForNetworkTrafficToStop(2000,timeout);
selenium.mouseOver("css=div#autoSuggestDiv > div");
selenium.clickAt("css=div#autoSuggestDiv > div", "");
browserMob.endStep();

[Example: 1e]

I’d like to highlight one more important aspect of the ‘waitForPageToLoad/waitFor[Text/Element]Present’ and ‘waitForPageToLoad/verify[Text/Element]Present’ combination. While both work fine it is recommended to use the combination waitForPageToLoad/verify[Text/Element]Present for load testing. Why?

Consider the snippets below:

var timeout = 60000;
selenium.setTimeout(timeout);
 
browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1f]

var timeout = 60000;
selenium.setTimeout(timeout);
 
browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.waitForTextPresent("Latest news");
browserMob.endStep();

[Example: 1g]

Both would exhibit the same behavior when the page loads successfully. However, in the instance of an error, say an ‘Internal Server’ error or a ‘Connection Reset’ error, the script would declare an error immediately after the error page loads with script 1f while it’ll wait for 60 more seconds to declare the same error with script 1g as the ‘waitForTextPresent(“Latest news”) is going to wait for 60 more seconds, which is the timeout we’ve set in the script, to declare the same content error, irrespective of the page’s status code. Besides,  script 1f will help simulate more transactions in the same amount of time than script 1g, especially in the event of errors.

3. Setting Proper Timeouts

The default selenium timeout is 30 seconds (30000 milliseconds) and the default TimeUnit in use by BrowserMob is milliseconds. It must be remembered that most of the ‘waitFor’ commands like the ‘waitForElementpresent’, ‘waitForVisible’, ‘waitFortextPresent’ etc wait for however long this default selenium timeout is set to. Timeouts can be set the following way in the script:

var timeout = 60000;
selenium.setTimeout(timeout);

It is recommended to set the timeout value before the actual transaction begins in the script. Studies show that the ideal page load time is 5 seconds. Any amount of load time greater than this is considered unacceptable for real users. However, 5 seconds is too small of a timeout threshold for load testing as load times are expected to rise with increase in user load. Hence, 60 seconds to 120 seconds would be considered ideal. The upper limit on the timeout threshold set by BrowserMob is 120 seconds.

Occasionally, you may have the necessity to set a different timeout value for different page loads which can be accomplished by the solution proposed in this article. The article on waitForCondition may also make an interesting read to learn more about advanced handling of timeouts.

4. Optimization of Locators

Selenium provides several ways of locating elements on pages. It may be worthwhile to read through this before proceeding further: http://release.seleniumhq.org/selenium-remote-control/0.9.2/doc/dotnet/Selenium.html

The most commonly used locator is the xpath. FireFox plugins like xpath finder, firebug may be used to identify the correct xpath for the appropriate elements. Below are a couple of useful articles on this topic:

http://blog.browsermob.com/2008/12/using-firebug-for-load-testing/

http://blog.browsermob.com/2009/04/test-your-selenium-xpath-easily-with-firebug/



Optimization of the locators used is extremely important for script robustness and re-usability. In the screenshot below, the IDE has captured the xpath ‘link=Arrested GI had gunpowder, ammo’ when clicking on the first article under “Latest News”. Obviously, the script will work only until this article is available on the Home Page. The xpath shown by firefbug for the same is ‘/html/body/div[5]/div/div[4]/div[3]/div/div[3]/div/ul[2]/li/a’ . Again, this might work temporarily but the moment the site is updated and the DOM changes, the script could break because of this complex xpath as the hierarchy of the elements could have been altered. ‘//ul[@class='cnn_bulletbin']/li[1]/a’ would be a much more optimized and a less brittle xpath for this example. Learn more about locators and pattern matching here.

5. Pause Times

Inserting pause times in between steps is essential in order to simulate real user actions more closely. A range of pause times is recommended over a fixed time as different users browse through a web site at different paces. Both BrowserMob and Selenium API’s pause method can be used to insert think/pause times in between steps.

6. Randomization

In the context of load testing, I want to emphasize the extreme importance of having large datasets available for testing. Many important bugs simply do not surface unless you deal with very large entities such thousands of users, thousands of mail server mailboxes etc. For scenarios involving product checkout, browsing through product pages etc, consider randomizing the product being purchased or browsed through, from a list of several products in order to simulate real user behavior more realistically as not all users are going to be purchasing the same product at the same time. Randomization and parameterization can be achieved in a few ways which are explained in depth here and here.

7. Use of quickStop() and stopExpected()

The BrowserMob API has two methods that come in handy when you want to pause a test that has exceptionally lengthy scripts running, such as loops that repeat certain actions several times over the course of the script execution.

By default, when a test is paused, BrowserMob will wait for the actively running scripts to finish executing and complete the current transaction. Long running scripts with loops or scripts testing a streaming video will not be interrupted until they complete. So in order to abort such transactions immediately on pausing, it is important to include the method browserMob.quickStop() at the top of the script.

However, forcefully aborting the scripts could result in an increased error rate, skewing the results. This can be avoided by using quickStop() in conjunction with stopExpected(). This instructs BrowserMob that the script is expecting to be stopped at any time and if that happens, the sudden stop should not be considered an error.

Here is a complete script, incorporating all of the above:

browserMob.quickStop();
browserMob.stopExpected();
 
var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();
 
//  Set the timeout threshold
//  Set a different timeout threshold for local validation as it has a limit on the
// transaction length, capped at 2 minutes.
if (browserMob.isValidation()){
 	var timeout = 30000;
}
else
{
 	var timeout = 120000;
}
 
selenium.setTimeout(timeout);
 
//*************** Randomize think time *********************//
function randomXToY(minVal,maxVal)
{
 	var randVal = minVal+(Math.random()*(maxVal-minVal));
 	return Math.floor(randVal);
}
 
function sleep(x,y)
{
 	if(!browserMob.isValidation()){
  		browserMob.pause(randomXToY(x,y)*1000);
 }
 	else{
  		browserMob.pause(randomXToY(1,2)*1000);
  }
}
 
//********** Randomly pick an articleId, SearchString combination *************//
function randOption()
{
	var list = "1,PERMALINK;2,Related Links;3,Related Articles;4,Most Popular";
	var myArray = list.split(";");
	//pick a random combination from the array every time
	var position = Math.floor(Math.random()*myArray.length);
	var myArray1 = myArray[position];
	return myArray1;
}
 
var selected = randOption().split(",");
var id = selected[0];
var string = selected[1];
 
// Logging the variables being set provides a good means of troubleshooting
browserMob.log(id);
browserMob.log(string);
 
// Begin Transaction
transaction = browserMob.beginTransaction();
 
browserMob.beginStep("Open CNN Home");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();
 
sleep(2,5);
 
browserMob.beginStep("Click on a random article");
// Randomly clicking on an article with every script run
selenium.click("//ul[@class='cnn_bulletbin']/li["+id+"]/a");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent(string);
browserMob.endStep();
 
browserMob.endTransaction();

As web sites get complex, the scenarios for the load testing tend to get complex and so does the scripting and other load testing requirements such as configuration and set up. Our team of trained load testing experts is waiting to assist you with all your complex load testing requirements. Contact us right away and get all your load testing needs fulfilled!!

Hope you enjoyed reading this article as much as I enjoyed writing it. Coming up next in this Load Testing Best Practices blog series is “Test Execution” so be sure to stay tuned.

Happy Testing!!

More articlesIn the Driver’s Seat Series:
Part One: Planning
Part Two: Configuration

So you reviewed part 1 of our best practices blog series, “Planning” and now you are wondering about configuring your test.  What are all the pieces that need to be thought about and configured?  Fortunately these are not complex questions; you just need to know some basic information:

1. Where do you expect or see web traffic coming from?
2. How many users come to your site?
3. How should you apply your user load during the test?
4. Should you test with virtual users or real browser users?
5. Where do users spend the most time on your site?
6. How long do users take when navigating your site?
7. Do you use third party vendors?

Where do you expect to see web traffic from?

Whether this is a new site getting ready to go live or an existing site already on the web you probably have a feel for where your visitors will be coming from.  A site like Amazon will have visitors from around the world, whereas the government site for a province in Canada will see most of its traffic come from the local province.  You want to be sure that when you test your site you have users coming from various points attempting to emulate your normal traffic.  If most of your traffic is coming from North America finding out your pages take twice as long to load in Asia is good to know, but only if you are planning on getting more users from Asia.  But if you focus on North America all the data gathered in the load test will help you make the experience on your site better for the majority of users.

If you don’t have the capacity to apply load from the most desirable locations for your load test you can use locations that are further away.  So for example you need to add Tokyo to get the load you want in your test.  You find that from Washington, D.C. your page load time is 2 seconds when there is no load on the site.  From Tokyo the page load time is 4 seconds with no load.  So you now know that testing from Tokyo has a 2 second lag and you can use that knowledge to better interpret results from the Tokyo location of your load test.

For more information on how to set up a multi-region test using BrowserMob, feel free to check out the tutorial at the end of this article.

How many users come to your site?

How many users should you choose to test on your load test?  This question should be answered during the planning phase by deciding what the goal of your load test is (for a detailed discussion see the previous blog “Planning”).  Are you testing stability over time?  Are you attempting to find the breaking point of your site?  In general you can assume your site should be able to handle 250 users per core in your web server farm.  This can vary greatly depending on bottle necks in the code, in your network, or coming from the web to your network.  If you want to see when the site breaks, I suggest finding your peak traffic number and adding 10% during each interval until you see performance that is unacceptable.  This assumes of course that your site currently handles your peak traffic with good load times and a low error rate.    If you are testing stability over time (i.e. looking for a memory leak), then I suggest taking an average user load level and running at that level constantly over a time period beyond where you expect the issue to surface.

How should you apply your user load during the test?

Now that you have a number in mind, how are you going to apply that number of users to your load test?  Typically web traffic comes in waves.  It will surge and recede over the course of a day.  Even with a major event like a new Super Bowl commercial the surge is not instant, 1000 users appear over time not in the same second.  The best practice here is to set up your load test with intervals of ramps and constants, so that you can get granular data.  A ramp is a period where we increase/decrease users and a constant is a period where the user count remains unchanged but the users are executing scenarios on the site.  If we take a 1000 user test we could set up an hour test with intervals like this:

This basically shows 20 total intervals, 10 ramps and 10 constants.  During each ramp users are added until the Max User count is reached.  During each constant the users are executing scenarios constantly.  Whenever a user finishes a scenario or encounters an error they will immediately start another scenario continuing to load the site at a constant level.

You could apply 1000 users at the beginning and run them over the course of the full hour but you will get data that only tells you about 1000 users.  Using the ramp methodology allows you to pin point when events occur.  You might find that between 400 and 500 users the page load times begin to take longer.  When you hit 600 users you start seeing errors due to connections being dropped.  When you hit 800 users your site falls over.  All of those pieces of data would be missed if you apply 1000 users to start.  You would just know that my site will fall over at 1000 users, but you won’t end up with a lot of actionable data pointing you to problem areas to address.

Should you test with virtual users or real browser users?

First let us define what each type of user is.  A virtual user is an agent that will mimic the http requests made by your site from a browser.  A real browser user is an agent that makes use of selenium to control an actual browser and navigates your site like a real user.  It takes fewer resources to create virtual users where you can get multiple users per CPU core.  A real browser user actually uses a browser on a computer and therefore is limited to one browser per core.  Virtual Users typically are single threaded and make each http request to build in order one at a time while browsers actually will make multiple http requests at the same time.

So how do you decide what type of user your site needs?  Virtual users are best for generating load on relatively simple sites that use traditional web forms and http get and post functionality.  You can also get a much higher user load for less as virtual users are less resource intensive.  Real Browsers are best for sites that use AJAX to refresh data for the user without making a full page request or make heavy use of JavaScript to enhance the user experience.

For very high volume testing, you can also use a combination of the two types of users by creating scripts that exercise the static parts of your site with virtual users and then real browser user scripts for the interactive parts of the site.  This will up the load on the site as a whole and will help you to achieve extremely high load levels.

Where do users spend the most time on your site?

The most important parts of your site to test are the parts that users most frequently visit.  This is not to say you shouldn’t exercise the more unused portions of your web site during a load test but the point is that you should try to keep your scripts small and focused.  This allows you to concentrate on particular processes or pages and get very detailed data pointing you to problem areas.  Let’s take a basic ecommerce site as an example.  Users come to the homepage, browse, select some products, checkout and leave, right?  Actually some users come to the homepage and then leave, some browse around and then leave, some add items to their cart and then abandon the cart and leave, and some buy the product they added to the cart.  So you can write a large script that comes to the homepage, navigates every category, picks some products, and then checks out but that is not really how your site is loaded.

For this example you would want to create 4 test scripts as follows:

1) Go to homepage and exit
2) Homepage, browse a category and exit
3) Homepage, browse, add product and exit
4) Homepage, browse, add product and checkout

One that comes to the homepage and then exits, one that browses a category and then exits, one that browses for a product and adds that product to the cart and then exits, and finally one that browses the site for a product and adds the product to the cart and then checks out.  Then you apply a percent of the load to each script based on the real user traffic on your site.  Your checkout process may not be able to handle 2000 concurrent users but you may never have more than 5% of your users checking out which would mean checkout only needs to handle 100 concurrent users.

Again I want to stress simple is best when scripting.  You could set these scripts up to browse random categories and random products.  These scripts will take longer to write and will be more apt to run into products with not enough availability causing an error in the test that has nothing to do with the load.  You are better served to keep your scripts simple to start and then add functionality such that you get good coverage over the key components of your web site.  What you really want to do is eliminate unknown variables so that you can tell what page, item, or process is hanging when load is applied.

How long do users take when navigating your site?

The next thing to consider is how fast you want the scripts to execute.  By using computers we have removed the human element of looking over the page and figuring out where to click next.  We have scripted this and can click the next link the moment a page has loaded.  So again to better approximate reality you should consider adding user think time to scripts.  If you have appropriate analytics data available, you can gather information from your site to figure out what the actual time is that a user spends on the Home page before making the next selection.  As a general rule and in order to stress the site sufficiently, we use a function that randomizes the think time between 3 and 7 seconds after every page load.

Do you use third party vendors?

Please make sure your vendors are aware of the dates and times when you expect to run these load tests.  Many times when they are not notified they will see the spike in activity as an attack and may blacklist the IP’s where the traffic is coming from.  This would cause your load test to record either errors, as items wouldn’t be available for download, or you could get invalid page load time data as not all the components of your page would be downloading for the test.

Parting Thoughts

We have answered some basic questions and made several decisions regarding configuring your load test.  We know where we want user traffic to come from.  We have a target number of users, how to apply the user load during the test execution, and have picked the type of user best suited to the script.  We have allocated those users to different behaviors for the test.  We have determined a user think time to apply between actions.  We have either removed calls to third parties or notified them of the load test execution.  This has set us up for a load test that will service our needs and ensure that we get detailed data that will help us determine what areas of the site need attention.

You should now be ready for “Scripting” the next entry in this blog series.  Be sure to check back or subscribe to the RSS feed if you are interested.  Feel free to check the video below to see how to setup a multi-region test using our BrowserMob tool.  If you have any comments or suggestions for future posts you can reach us at marketing@webmetrics.com.  Thanks!

More in the In the Driver’s Seat Series:
Part One: Planning

This is the first part of a five-part blog series that will cover all the aspects of load testing, including planning, configuring, scripting, executing and analyzing. This first article focuses on the key areas and questions to consider when planning to run a test.

Why test?

I generally see customers greatly OVER-estimating their expected capacity and many are often shocked when they get the first set of test results back from us. Usually, there is a lot of work to do in order to get the website or application tuned up and ready for a full production release.

Load testing helps you scientifically determine how many customers the website or application will support – BEFORE you potentially find out the hard way (i.e. when users actually come to your site and you have to scramble to make last minute capacity improvements). In the case where your site generates online revenue, it will also help you to quantify your investment and give you specific metrics on your overall cost per transaction in terms of required infrastructure. For example, if the site is able to support 50,000 transactions under peak conditions, you can take this number and divide it by your monthly infrastructure costs (say $2,000) to determine your infrastructure’s cost per transaction (e.g. 4 cents).

The key here is to find out this information BEFORE it causes irreparable harm to your business, brand and reputation. The only reliable way of doing this is to plan a load test.

When to test?

The ideal time to run your first load test is after User Acceptance Testing (UAT) is complete, but prior to moving the application into production. Load testing should be part of your project plan and it’s important to allow for sufficient time in your plan to complete performance and load testing. Based on my experience, you need to allow for an absolute minimum of 2-3 weeks for appropriate load test planning, scripting and execution. Many customers try to compress this time down when things are running behind, but there are no tricks to accelerating this process (and generally this results in a painful experience for everyone involved and severe sleep deprivation!). The process of load testing and performance analysis is an incremental process that requires careful review and dedicated time from the development team to address any problems or recommendations from the load tests.

How many tests should I run?

Based on our work with hundreds of customers over the years, we have found that three is the magic number.

• The first test provides you a benchmark and a baseline snapshot of the current performance of the site. There are always a number of recommendations for improvement from the first test and you need time to let your team react to the results and implement the necessary changes.

• The second test is designed to validate all the changes made after the first test and identify any remaining bottlenecks. Additional tweaks to the infrastructure or application itself are then made in preparation for the third test.

• The third test is then designed to be the final validation point prior to go-live and ideally will uncover the peak performance point of the application.

Of course, your mileage may vary here. Some customers are able to complete this process with just two tests, but generally we have found it’s best to start with three tests and add additional ones as needed.

Real Browsers or Virtual Users?

With the rapid adoption and investment in Cloud Computing, it’s now possible to cost effectively launch thousands of real browsers an hour to test a website. Neustar offers this service in both a self-service platform as well as a fully managed engagement. For more information, visit the BrowserMob website. This has added a lot of realism to the process of load testing, which until recently has been limited to the testing using a “virtual user” or “virtual browser” that simply mimics the http request/response sequence issued by the browser.

Testing with real browsers provides unprecedented realism in driving user load to the site and generating a realistic load profile against the application. This is especially true with the wide prevalence of AJAX and client side plug-ins such as Flash/Flex. We have recently partnered with Gorilla Logic to make the process of load testing of Flex applications as easy as possible. Now you can quickly create automated Flex test scripts and launch thousands of real browsers to stress test your Flash/Flex applications!

Testing with real browsers also greatly speeds up the time to develop test scripts. The scripts also interact with the site from a functional standpoint by referring directly to the appropriate DOM elements that comprise the page rather than relying on mimicking of the underlying http request/response sequence. This helps to improve script re-use and provides an additional assurance that the test scripts will be accurately performing the actions of a real user when visiting the site. Virtual user testing still has its place for very high volume testing and testing of web service transactions or very simple websites, but in general load testing with real browsers is highly recommended for most sites.

How much load should I test with?

500. Just kidding.

The answer comes down to whether you need to execute a “stress test” or a “load test.” Let me explain the difference between these two terms and how they impact your overall test plan.

Stress Test
A “stress test” is designed to take the existing application infrastructure and test it to the breaking point where one or more bottlenecks are exposed. This involves sizing the test based on the amount of expected traffic the total infrastructure is expected to handle. A good general rule of thumb here is approximately 250 concurrent users per each CPU core available to process front-end web server requests. Please note that this is a very rudimentary metric and varies widely based on the type of web server, network infrastructure and most importantly the application. Performance benchmarking can help you better determine exactly what this number is, but 250 is generally a good number to start with. For example, if you had a new application with 2 front-end web servers installed on a quad-core machine, a reasonable test would be to run with 2000 concurrent users (250 users x 2 machines x 4 cores).

Load Test
A “load test” is designed to test the site to a certain volume of page views or transactions, in order to validate that the site has sufficient capacity to support expected user traffic. This type of data can be obtained from web analytics data such as Omniture or Google Analytics. Generally, the target volume is based on the peak hour of historical usage for the application.

For example, say you find that in the peak hour there are 50,000 pages being loaded and you expect a 20% increase in overall web traffic in the next several months. A good test approach would be to target a total load of 75,000 pages views so you can determine the ability of the application to serve up to and above the target page view count.

Concurrent User Calculator

When planning out your load test, it’s helpful to use a calculator to determine the total number of concurrent users necessary to reach your target load level. To illustrate and help with this process, I’ve created a simple concurrent user calculator.  Simply plug in the desired number of page views and the rate (e.g. 100,000 page views per hour), the session time and the desired test time (we generally recommend 60 minutes) and the calculator will determine approximately the number of users required to generate the associated number of page views. The calculator can be easily extended to estimate the number of transactions expected to complete during the test.

Before going too far explaining the calculator though, let me first explain a common misconception between a “concurrent user” and a “unique user” or “user”. We have lots of customers that come to us and say that they need to test their site with 50,000 “users.” Most of the time, what they are trying to do is simulate the business requirement of 50,000 unique users coming to the site and performing a specific transaction. This is very different from the idea of a “concurrent user.” A concurrent user is designed to run through a transaction from start to finish and then immediately start again, repeating the same transaction. This user will do this repeatedly, until the end of the test. A “unique user,” on the other hand is simply a single execution of a concurrent user or the completion of one transaction (execution of the test script from start to finish). Depending on a number of factors, concurrent users are able to generate the traffic equivalent of thousands of unique users over the course of a standard test. The calculator determines the number of concurrent users required to generate the expected amount of load. Now with definitions out of the way, let’s look at a specific of example of how to effectively use the calculator to plan your load test.

Say you have a 10-step check out process and you want to ensure the site can handle 1,000 checkouts in one hour. The total number of page views associated with this process is 10 steps x 1,000 transactions = 10,000 page views. Let’s also assume that we expect each page load to take 5 seconds and the user to wait 5 seconds on each page. This would give us a total session time of 10 steps x 10 seconds = 100 seconds. So plugging in 10,000 pages per hour and 100 seconds for the session time, gives us a concurrent user count of approximately 278. In this way, we can calculate how many users are needed to complete the key business transactions for the website. It’s important to note here that these are just estimations and highly dependent on the actual page load times encountered during the test. As page load times start to degrade, the overall throughput the site in terms of page views also degrades. Adding additional users at this point becomes overkill and will not result in an increase in the number of total pages loaded. This is why sizing of the load test is critical to make sure you have enough traffic at the ready to validate your objectives for the site.

Test data

Another aspect of load test planning that is often overlooked is the creation of appropriate test account data. This often takes the form of creating individual user ID’s to log in to the site or a list of products to select from, etc. Creation of a realistic and accurate data set is important to get the most out of the load testing process. Many times, errors encountered during the load test are the result of an incomplete or inaccurate test data creation process. Depending on the scope of testing, the data creation can be time consuming so it’s important to identify early on what data needs to be created and the process of creating it. Typically, data creation scripts will be needed to create the necessary data sets. Alternatively, custom user test scripts can be written as part of the load testing engagement to create the necessary data sets.

Server and Network Monitoring

Finally, you want to make sure you have a plan and resources in place to monitor the network devices, web server, and application and database servers that support the application. Load testing will give you all the key front-end metrics, and to get the most out of your test you’ll want to cross correlate this information closely with the performance counters from all the devices in the application delivery chain.

Wrap-up

I hope this article has given you a good sense of the importance of load testing, as well as the key steps and questions to consider when planning to run a load test on your website or application. Next up in this series is “Configuration” – so be sure to check back to read more, as this blog series will be taking you through the full load testing process.  Also, please feel free to drop us a line at Marketing@webmetrics.com if you have questions or comments or suggestions for future posts.  Thanks!