We’re a few weeks late to the story, but we think it’s still important to call attention to: Watir and Selenium are joining forces. For those unfamiliar with Watir or Selenium, both of them are open source browser automation frameworks. For years they have been “competing” in the open source community, with Watir winning favor among the Ruby community and those who needed strong IE support, while Selenium won favor among the Java and C# crowds and those who really valued cross-browser support.

While both projects had their pluses and minuses, it’s great to see them finally working together. On behalf of BrowserMob (which builds on top of Selenium) and as a Selenium contributor, it pleases me to know end to know that the Selenium community will now gain two huge contributions: a first-class Ruby API and all the benefits of Watir’s fantastic IE support.

Note: this is not a merger. Watir will continue to run as an independent project and it will even still work with it’s own browser backend. The important thing here is that Watir will now offer an option to wrap around Selenium WebDriver, the core cross-browser automation library that all the other Selenium projects also wrap around. This means that while the projects will remain independent there will likely be a lot more cooperation moving forward.

Tweet This Post 

Selenium · Watir · WebDriver

Today we’re extremely excited to announce another industry-first in the world of load testing: video capture.

From day one, we always believed that cloud computing ushered in an era where the majority of website load tests could and should be done using real browsers, something no one else had considered before. Doing so made scripting painless, load tests more realistic, and provided better error reporting through the use of screenshots.

Today we’ve raised the bar by also capturing video for every unique failure in a load test. This means that you can now see exactly what led up to the failure and even get a feel for what the user experience was like as your site began to break down.

In addition we also now capture the source HTML (ie: DOM structure) of the web page in the event of a failure, as well as the entire set of request and response headers for the failed transaction. This gives unparalleled information that is simply not available in the traditional load testing approach that relies on headless/virtual users.

We have a lot more great features in store for both our website load testing and website monitoring products, but we’re especially excited about this one. It’s available today so please check it our for yourself and let us know what you think.

Tweet This Post 

Load Testing · Video

One of our customers, SignalFive, recently published a case study about a project they did with Al Gore’s Alliance for Climate Protection.

The project, called the Repower America Wall, involved a massive, Flash-based interface in which thousands of user-created images and videos could be scanned through at high speed.

In the case study, they describe how they calculated the estimated site traffic and decided to use cloud services, such as Amazon and YouTube, to help the site scale. However, before launch they still needed to prove to the customer that the site was ready:

While this stress-testing was valuable, and optimizations were made based on them, none of these techniques truly simulated real-world use. Typically, a testing environment such as Selenium IDE might be used. But since the UI of the wall was built in Flash, things got tricky.

Because of the rich Flash interface, traditional load testing services based on simple HTTP protocol simulation weren’t well-suited. For that reason, they came to BrowserMob. Using our real browser technology, we were able to load the entire UI, allow the background AJAX and AMF calls to take place, and even stream the various videos:

BrowserMob’s team helped us craft some tests which we ran on the dev and production servers. The information was invaluable — based on the test results, we decided to upgrade the production server, and doubled its memory.

Below is some of the load test charts SignalFive generated using our software:

Tweet This Post 

Case Study · Load Testing

We’re happy to report that Selenium IDE 1.0.4 has been released. You can download it here and you can find the release notes here.

While this release doesn’t have many new user-facing features, it does clean up several bugs. More importantly, however, is that Adam Goucher and Jérémy Hérault did some amazing work to lay the foundation for a plugin framework. This means that soon you’ll see Selenium IDE plugins that further expand the Selenium IDE capability.

Jérémy is working on one such plugin, called Helenium (see proof of concept in action), that will allow you to do text matching against images and PDF files using optical character recognition (OCR). We’re also working on a plugin that will make it easier to upload scripts from the IDE to BrowserMob.

If you’re interested in how to build your own plugin, I recommend reading Adam’s blog, which has a series of recent posts on how the plugin framework works.

Tweet This Post 

Selenium · Selenium IDE

Late last night Amazon released an early Christmas present: a third location for which to run Amazon Web Services. This is important news for BrowserMob customers for two reasons:

• Monitoring – it will be our fifth location from where checks will run from.
• Load testing – it will be our third location from where traffic can be generated from.

We are committed to moving at the “speed of the cloud” and as such expect to have both of these options available in the next week. We’re excited about Amazon’s commitment to continue to provide geographic diversity to their cloud offering, and we look forward to their expansion to Asia early next year.

Tweet This Post 

No tags

I’m a little late to the party (dynaTrace released their product a couple weeks ago), but I wanted to still highlight their very important tool: dynaTrace AJAX Edition. It’s by far the best browser profiling tool out there – and it’s free!

If you’re familiar with Firebug, then that’ll give you a rough idea of what this product does. However, instead of being a “Swiss Army knife” like Firebug is, dynaTrace did a deep dive on JavaScript profiling.

This means that this tool can help identify hotspots and bottlenecks in your client-side JavaScript. And as applications continue to get more complex, this will continue to be a bigger and bigger area that developers need to keep an eye on. Ultimately, it doesn’t matter if your server is fast if the client application is getting bogged down due to inefficient JavaScript and DOM calls.

Rather than launching in to a full overview of what dynaTrace AJAX Edition does, I recommend the following three articles:

• A Step-by-Step Guide to dynaTrace Ajax Edition – by Andreas Grabner, technology strategist @ dynaTrace
• dynaTrace Ajax Edition: tracing JS performance – by Steve Souders, chief performance engineer @ Google
• Deep Tracing of Internet Explorer – by John Resig, creator of jQuery

If you’re working on performance optimizations for your site, especially if you are concerned about IE (and let’s be honest, who isn’t?), do yourself a favor and add this tool to your toolset. You won’t be disappointed.

Note: I should add that BrowserMob is a dynaTrace partner and we are continuing to explore ways in which BrowserMob, Selenium, and dynaTrace products can work in concert.

Tweet This Post 

No tags

A couple days ago we reported midway through the much-hyped Cyber Monday that most sites were doing fine, but that Kohl’s was having serious trouble. Now that all the traffic has mostly died down, we wanted to take a final look at how the sites did.

First, the big winner of Cyber Monday was without a doubt Amazon. According to Hitwise, prior to Cyber Monday Amazon had already received more than 15% of the market share of online shoppers. And while this is by far the largest share (the next closest was Walmart, with half the visitor share), Amazon’s servers had no trouble keeping up with the load, as evidenced by our monitoring service:

This shouldn’t surprise anyone: Amazon is so massive and so good at scale, they practically created a brand new market (Cloud Computing) by selling their technology and infrastructure in the form of Amazon Web Services, the very same stuff BrowserMob is built on top of.

Unfortunately, not everyone else fared as well. Kohl’s faced continuous performance problems all weekend and throughout the day on Monday. In fact, while the errors and massive performance problems had dissipated by Tuesday morning, performance had leveled out to 50% higher than it was prior to Cyber Monday, indicating that Kohl’s is still battling performance demons:

Note that these are hourly averages, and that the red dots indicate that there was one or more error detected on the Kohl’s home page during that hour. For an example of such an error (and, unfortunately, lost revenue opportunity), check out our post on Kohl’s from a few days ago.

In terms of uptime/availability, we counted eight full blown failures over a 14 hour period, which translates to a 4.76% downtime. Actual loss of revenue was probably even higher as some customers likely became frustrated and abandoned the site for other retailers. After all, according to a new report by Akamai and Forrester, two seconds is the new threshold for how long shoppers will wait around. If you’re curious how you stack up, run an instant test right now.

Unfortunately, Kohl’s wasn’t the only one to be crushed under a stampede of holiday shoppers: Borders also suffered. While the performance didn’t spike nearly as high, Borders too faced almost a 2X decay in response time and sixteen outright failures during the same time period:

Besides the fact that Borders was effectively down almost 10% of the time we measured, the most noteworthy part of this story was Borders’ failure to gracefully fail. While every website operator hopes to be able to handle all the traffic that comes it’s way, every site will ultimately fail – even Amazon. How you fail will determine if your customers have confidence in your business.

For example, when Kohl’s experienced too much traffic, they were able to place a cute message indicating that they were trying to keep up with the load:

While the ~5% of visitors who experienced this page probably were disappointed, they were almost certainly less disappointed than the ~10% of visitors who experienced this from the Borders website:

So while we of course always recommend load testing as a way to prepare for expected surges in traffic, it’s also a good idea to push that load beyond the limits to see if your systems fail gracefully, like Kohl’s was able to. If not, you might just leave your customers confused and frustrated.

Tweet This Post 

No tags

Using our own browser-based monitoring service, we’ve been watching some of the top retailers this morning to see how they fare during the Cyber Monday onslaught as US shoppers return to work after the Thanksgiving holiday. Of the sites that we’ve been monitoring, very few have had any problems so far.

While Amazon, Kmart, Macy’s, Walmart, and Best Buy all experienced minor site performance problems, only Kohls so far as outright failed. Starting around 7AM PST their home page response time went from 2.5 seconds to, in some cases, over 25 seconds (a 10X slowdown). At one point it was completely offline.

Check out some of the screenshots our monitoring service captured:

Tweet This Post 

No tags

We were recently delighted to see a detailed writeup about optimizing WordPress by Rob Havasy, a blogger by night and business analyst by day. He’s been running his WordPress-powered blog since May 2009 and everything had been going quite nicely until recently:

But I noticed a sudden decrease in performance earlier this week and couldn’t understand why. I was having both resource issues on the server (PHP was consuming too much processor capacity and my host automatically killing the process occasionally) and an overall slowness on the pages. How would I track down what was going on?

Rob goes on to explain how one of the plugins he was using had begun to have performance issues (apparently due to a 3rd party API slowing down) and not until he disabled the plugin did his WordPress-based site performance improve dramatically. The best part? He discovered the problem and confirmed the fix worked by using BrowserMob’s free monitoring service.

Tweet This Post 

No tags

Today dynaTrace announced that it has intergrated support of Selenium and BrowserMob to allow customers to tie their web load testing to server performance.

In my recent blog post – Get more out of functional web testing: How to correlate test reports with server side log information? – I discussed the problem that testing results are usually not linked to the log and diagnostics information captured by the application under test. The blog entry offered a way to link the two sides using HTTP Tagging via an HTTP Proxy. Tagging individual Web Requests allows linking each individual request executed by the testing tool with the transactions that are executed on the server side. Your logging framework or diagnostics solution can then take this tag and link the transaction to the originating web request. (via dyanTrace Blog)

This is a great solutiong for in depth performance diagnostics.

Tweet This Post 

No tags