Today while browsing the SQA Forums, I found a question very relevant to BrowserMob: should I test on production?

There is no right/wrong answer to whether you should run a test on a production environment. In an ideal world, you’d have a staging environment that is exactly like your production one. But that is rarely actually the case.

Definitely do your initial performance testing internally first. Keep the tests simple and focussed, pinpointing specific parts of the app (certain parts of the UI, the database, etc). Sometimes these tests can be as simple as attaching a profiler (ie: YourKit for Java) to a developer’s desktop and just navigating through the app by hand a few times or playing back a Selenium IDE test 10 times in a row.

Often times you’ll find 80% of your performance problems with really simple, low-volume tests, since poor code usually stands out like a sore thumb even under low volumes.

Then it’s time to ensure the whole environment will work together. This is where a full scale load test comes in. If you have a test/stage environment that you can use, definitely start there.

If things look good, then the next question is: is my production environment different enough from my test/stage environment to warrant additional testing there?

It’s impossible to say for sure in your case, but in my experience, my test environments wouldn’t have a load balancer or redundant databases and web servers. In those cases, I considered that a large enough difference that I wanted to do a real world, production environment test. Otherwise, my load balancers could be broken and I wouldn’t necessarily know it until it’s too late. The same could be for my database clustering configuration or any number of other settings.

If you do decide to test on production, then you’ll need to analyze the risks:

• If users are allowed to access the site, they could experience major slowdown and/or errors. Is this acceptable?
• If users are allowed to access the site, it may be difficult for you to pinpoint the root cause of performance issues, especially if there are a lot of users doing different things.
• If users are not allowed to access the site, is the business prepared to allow the site to be unavailable for some time.
• If the load test generates a lot of data on the production database, it could cause slowdown that will continue even after the test is complete.

These are just some of the risks, but I hope it gives you an idea. In general, I tend to lean towards doing some sort of “real world” test. Afterall, that’s why we created BrowserMob in the first place!

Whether that test is your actual production environment or a close approximation of that environment, the key thing is that you analyze the risks and understand what you’re testing for. You’re not just testing the software now, but you’re testing the whole environment (load balancers, web servers, routers, etc).

Tweet This Post 

No tags