Google recently posted an article on their testing blog: Survival techniques for acceptance tests of web applications (Part 1). In it they note the difference between various expressions used with Selenium to get access to an object on a web page:

For each method, we need to work out how to implement it in code. How could an automated test select the compose message icon? Do alternative ways exist? An understanding of HTML, CSS, and JavaScript will help you if you plan to use browser automation tools. All the visible elements of a web application are reflected in the Document Object Model (DOM) in HTML, and they can be addressed in various ways: the directions from the root of the document to the element using xpath; unique identifiers; or characteristics possessed by the elements, such as class names, attributes, or link text. Some examples of these addressing options are shown in the Navigation Options illustration below. (Notes: navigation using xpath is much slower than using IDs; and IDs should be unique.)

We agree that IDs are usually the best approach, but sometimes (especially with AJAX.NET applications) those IDs are actually really bad to use. The reason is that some frameworks (AJAX.NET, ExtJS, etc) use ID generation techniques that aren’t the same for every page view or every web app build.

As such, keep an eye out for suspicious IDs and be prepared to switch back to XPath or DOM locators when necessary. For example, if you see an ID like ctl00_ctl00_cp_cpTab_txtFirstName (generated from AJAX.NET) consider changing it to an XPath such as //input[contains(@id, 'txtFirstName')].

Related to that topic, they also note that scripts recorded in Selenium IDE should not be considered “done” and will likely need to be optimized and tweaked before they can be considered stable:

The open-source test automation tool Selenium (http://seleniumhq.org/) includes a simple IDE record and playback tool that runs in the Firefox browser. Recorded scripts can help bootstrap your automated tests. However, don’t be tempted to consider the recorded scripts as automated tests: they’re unlikely to be useful for long. Instead, plan to design and implement your test code properly, using good software design techniques.

The author goes on to suggest Firebug as a great tool for doing this kind of work, which we tend to agree with and have covered previously. But they missed one of the most important functions: the ability to easily test XPath expressions from within Firebug!

Very few Firebug users know about this, so take note: Firebug adds a $x function that can be called from the console. It takes an XPath in the form of a String and will spit out the HTML elements that it evaluates to directly back in the console. You can then mouse over the elements and check that the XPath is working correctly:

Armed with Selenium IDE, Firebug, and $x function, you should be able to build out solid Selenium scripts in no time!

Shameless plug: these types of tips (and more) are yours for free when you sign up for a BrowserMob performance and load testing account.

Tweet This Post 

No tags

Steve Souders, a senior web performance expert at Google, recently wrote a fantastic analysis about why high performing websites should avoid CSS rules that include the @import statement:

In Chapter 5 of High Performance Web Sites, I briefly mention that @import has a negative impact on web page performance. I dug into this deeper for my talk at Web 2.0 Expo, creating several test pages and HTTP waterfall charts, all shown below. The bottomline is: use LINK instead of @import if you want stylesheets to download in parallel resulting in a faster page.

We recently worked with a customer who had eight separate CSS @imports, causing significant overhead in an otherwise well performing website. Making changes like the ones Steve advocates can lead to significant performance improvements without any investment in complex software or network changes.

Steve’s advise about CSS is important to read, but it is only one part of a broader point: high-performing websites limit the number of HTTP requests made from the browser. Keep that in mind when it comes to any CSS or JavaScript in a website.

Tweet This Post 

No tags

When doing any type of performance testing, whether it be production monitoring, load testing, stress testing, or simple performance tuning, one of the most important data metrics is the Time to First Byte (TTFB). As the name implies, TTFB is the amount of time it took for the client (usually a web browser) to receive the first byte in a given request. While the total time it took to download the object (ie: Time to Last Byte – TTLB) is also important, the TTFB usually tells a more important story, especially when it comes to software layer optimizations.

For example, consider the following two objects downloaded as seen by BrowserMob during a free load test:

Both of these requests took ~1 second to complete, but that’s where the similarities end. The JPEG is four times larger in file size than the index.jsp request. They also appear to be hosted by entirely different web servers (example.com vs your-ad-partner.com). What’s going on here?

Well, without knowing the TTFB it’s difficult to say. It could be that the web server and network connection hosting the JPEG is simply four times slower than example.com. But suppose we also knew the TTFB for each of these requests:

With this additional information, it’s much more obvious what is happening. Now we can see that in the case of index.jsp, once the first byte was received it only took an additional 10ms to get the remaining content. This is very different from some_ad.jpeg, which received the first byte fairly quickly but then took another ~950ms to to finish downloading.

What this tell us is that the bottleneck for index.jsp is likely due to some server-side processing, possibly tied up by heavy CPU usage. This is very common with requests for dynamic pages (eg: extensions with .jsp, .aspx, .php, etc). That’s because dynamic pages often will not begin sending back the any content until the internal page has completed processing. If the page needs to connect to a database or do another “expensive” operation, that could be the cause of the slow performance.

In the case of some_ad.jpeg, the situation is entirely different. The different between the TTFB and the TTLB is fairly large, meaning that the overhead is likely just a network delay caused by some slow performing connection somewhere between the client and your-ad-partner.com. It could also be due to a poor configuration of the web server hosting the image, but it definitely is not due to any dynamic content that might be taking up large amounts of CPU.

So how do you resolve these different situations?

In the case of objects with long TTFB times, like index.jsp, the solution often requires a software-level optimization. It could involve adding a database index, introducing some object-level caching, or a configuration change (such as database connection pooling). Be careful to fall in to the trap of throwing more hardware at the problem to solve these types of issues. While it might work in the short term, these issues almost always are due to sub-optimal software and throwing extra hardware at the problem will be like putting a band-aid on a bullet hole.

In the case of objects with relatively short TTFB times but overall long TTLB times, the solution is usually very different. While there may be a software solution, such as configuring Apache’s connections to be better optimized for the server it runs on, most of the time the root cause is due to network/hardware-related issues. Check with the ISP that hosts the server to confirm the max bandwidth throughput allowed. If the object response is slow during peak times but fast during off-peak times, it may need extra web servers (ie: hardware).

Alternatively, you might want to look at a Content Delivery Network (CDN) to help host the objects in a physically closer location. For a low-cost CDN, check out Amazon’s CloudFront service, which can let you host images and other static objects in nine separate locations around the world. This is a great, low-cost solution for people who want to serve static content to many different geographies but don’t have the budget or desire to open mutliple data centers.

Tweet This Post 

No tags

A couple weeks ago I was interviewed on the Startup Success Podcast about BrowserMob, testing in the cloud, and cloud computing in general. If you’re interested in any of these topics, please check out the podcast here.

While you’re at it, the same Startup Success Podcast also recently interviewed UserTesting.com about their crowd-sourced usability testing service. Also worth a listen, here.

Tweet This Post 

No tags

In the last few weeks we’ve worked with several customers who all experienced similar issues with their site under load: users were reporting that data they entered in to form fields was “disappearing” and that they were being asked to re-enter it a second time.

While no two websites are exactly the same, we were surprised to see that in each of these cases the same root issue was to blame. Specifically, these sites all used advanced AJAX techniques to make parts of a form dynamically change based on the values selected by the user earlier in the form.

Consider this simplified example from a hypothetical airline booking website:

The form has been designed to show all cities and airports that the airline services in both the departure and arrival drop-downs. However, once either an arrival or departure has been selected, the other drop down is modified to only show valid cities.

For example, if selecting “San Jose (SJC)” from the departure field, the arrival field would lose the “San Francisco (SFO)” and “Oakland (OAK)” options because the airline doesn’t do flights between those cities.

The idea behind this behavior is to simplify the user experience. But depending on how the form fields are updating and how your website scales, the user experience could actually be very poor. Here’s how that can happen…

AJAX and User Behavior Timeline Conflicts

Suppose that the departure <select> box has an onchange JavaScript event handler, which in turn makes an AJAX call with the selected city/airport. The resulting response is a bunch of <option> tags that are meant to replace the body of the arrival <select> box, as illustrated here:

This process works based on the assumption that the AJAX call will return so quickly the user could not possibly begin selecting the arrival city before the AJAX call completes. This is the normal behavior and it works just fine.

But now imagine that under heavy load the AJAX request starts taking longer and longer to return. Now the timeline looks like this:

What is happening now is that the user is actually working faster than the backend AJAX call. Now the user is able to make a selection to the arrival city and then after the selection was made the select box contents get replaced, eliminating the selection (for our web savvy readers, this is most frequently caused when the AJAX response gets assigned to the select element’s innerHTML field).

Under this scenario, the user did select a value but then the value appears to be “deselected” some time later. It doesn’t even take a very long AJAX call for this behavior to happen – a couple of seconds is enough time for some users to make selections.

Lessons Learned

So what are the lessons learned here and what can we do to prevent this type of behavior? The most important thing to understand is that when doing performance and load testing with today’s modern web applications, simply knowing how long HTTP requests take to complete won’t necessarily tell you how the end user’s experience is under such load.

It’s also important to know what those HTTP requests tie to and how they affect the user interface and the associated user interaction. This is sometimes difficult to do, since not even the web developers may think of these types of edge cases, but it’s worth thinking about.

The easiest way to test for this situation is to actually test from the end-user’s experience. Rather than doing traditional load testing, which simply simulates HTTP traffic, use a service like BrowserMob to run a load test that uses real web browsers. These will exercise the full AJAX logic inside the browser and will catch things like this.

For example, one of our customers who had a very similar issue reported seeing errors in their load test that claimed a field wasn’t filled out – just like a user would if they didn’t notice that the form field selection they made was erased before they submitted the form.

Simple UI Solution

Finally, the best thing to do always do is to tune the system so these AJAX calls don’t slow down. But you can never guarantee that the performance will always beat a user’s speed when selecting forms, so it’s good practice to include some additional UI logic to prevent this from happening. There are two common techniques you can use to solve this exact problem, though similar techniques can apply for similar problems:

• When the onchange event fires off an AJAX call that you know will change other form fields, de-activate those form fields before making the AJAX call. This will ensure that the user can’t set the value in between the request and response.
• Instead of blindly filling the select box with the contents of the AJAX response (via innerHTML), use a more granular technique, such as JSON, and iterate over each option in the select box – adding and removing fields as necessary.

In either case, it’s also a good idea to have some visual indicator that background work is happening. None of these solutions compensate for a high performance UI framework, but they are good practices regardless of your UI performance and can avoid these types of tricky load-related UI bugs.

Tweet This Post 

No tags