Aug 142009
There are trade offs between using Virtual Users (VUs) and Real Browser Users (RBUs). The table below shows the advantages of RBUs in many situations, but also points out when a VU might be more appropriate.
| Feature | VU | RBU | Advantage of RBU |
|---|---|---|---|
| Scripting Effort | Requires JavaScript coding and some knowledge of the underlying application | Using Selenium IDE a non-programmer can easily record a script | If you can use a browser you can record an RBU script |
| Download Threads | Single thread downloads objects serially | 6 concurrent threads download objects in parallel | More realistic load characteristics |
| Ajax | May be scripted by someone knowledgeable about the application | Easily recorded using Selenium | Script creator doesn’t have to reverse engineer the application |
| Flash | No support | Can download Flash data | Has ability to test Flash players |
| SOAP/REST | Ideal for testing non-browser HTTP requests | None – The VU has the advantage | |
| Validation | Code validation logic using assert statements | Validation is natural part of application usage sequence | Since an RBU follows the actions of a user it will naturally fail if the page fails to load properly |
| Error Reporting | Error codes from server | Screenshot of failed page | One can see what a user will see during failures |
Often the right approach involves using a bit of both. VUs can be a good way to lay down base load on “read-only” pages that don’t involve form interaction or AJAX. For very large tests we suggest a 90/10 approach, where 90% of the traffic could be generated from read-only VUs and the final 10% from RBUs.
Thanks for that breakdown, it was very useful.
You mention running both VUs and RBUs which i also thought would be good. However you don’t currently support scheduling the two different types together in one test. Would I need to schedule two tests to start at the same time?
Hi Neil,
You are correct, the way to do this is to schedule two tests at the same time. Since the test types are so different we felt it made sense to separate them. It also keeps the UI from becoming overly cumbersome. Of course we are always happy to hear input from our customers. And thank you for your comment.
Good post.
I disagree in some points:
“If you can use a browser you can record an RBU script”.
I don’t think all the people that can use a Browser are skilled to record a RBU script with Selenium IDE (or others). You can check a lot of forums and mailing lists where common people (and some testers) ask about how to use recording tools.
“Ajax – Easily recorded using Selenium”
Frequently the ids for fields, textbox,…., and other objects are dynamic in Ajax, so if you don’t have programmer skills (or regular expression skills) could be very hard record that Ajax calls.
Anyway, it’s only my opinion.
Best Regards,
Jose
Hi Jose,
On the first point, I’m sure you’re right that there are some people that need assistance. But Selenium IDE is incredibly simple to use for basic scripts. Given that you don’t need to learn a programming language I’m sticking by my statement.
As to your second point, we are of the opinion that compared to the alternative RBUs are by far the easiest way to script load tests, even when it comes to difficult AJAX-based sites.
I think we are quibbling over degrees. Is Selenium a silver bullet, no. But we think it is a lot simpler than the alternatives. It makes simple things dead easy and the difficult things possible for the dedicated tester.