Over at Web Performance there is a well written blog post about issues one must be careful of when doing load testing from Amazon’s cloud. The major issues that didn’t seem to have obvious fixes in the article were:

CPU Utilization

High CPU usage can cause timing delays in page durations, skew the distribution of users as the load is ramped up, and slow down page processing (particularly extractors using regular expressions).

Bandwidth

Amazon and other cloud-computing services either do not guarantee a level of bandwidth, or guarantee a certain amount of bandwidth rather than a level.

Both of these problems are basically different forms of resource
starvation, which can certainly be a concern in a virtualized, shared,
cloud environment. In many ways, it’s no different than the concern
one would have about running any software on a virtualized layer that
might be potentially shared for other uses.

Given that there is going to be some amount of “known unknown”
variability when running load tests from the cloud, the way to solve
these issues is to work within an environment that lets you react to
the variables as they arrive. You can usually avoid the problem
entirely by provisioning more resources than you need.

For example, in BrowserMob we generally only allocate 1 CPU core and
1GB of RAM per web browser. This is of course far more than you need
to drive a browser and our CPU utilization logs show it: we typically
use no more than 5-10% of our CPU capacity. We also tend to rent out
the largest EC2 instance types (High CPU Extra Large), which gives a
much higher probability that no one else will be using the physical
bare metal.

For our Virtual User service (which is closer to what everyone else
does), we never use more than 50 VUs per CPU core. Compare that to
most commercial and open source alternatives, which encourage you to
run 500+ concurrent VUs on your computers. We can afford to use so
many resources because our expenses are only active during the test,
so overall they remain very low.

But beyond simply over-allocating resources, there can still be
outside facts that can cause resource starvation. For example,
bandwidth could decrease in a certain EC2 availability zone, or a VM’s
CPU could be starved by a different EC2 customer. That’s why we spent
a lot of time building out automated “health checks” for all of our
load testing agents. We use the data from these checks to
automatically spin up more resources as needed, potentially in other
regions that may not be suffering the resource starvation.

We think the cloud is a great place to run load tests from, but it
really requires an approach that was built from the ground up to work
with cloud environments. Traditional commercial or open source tools
simply weren’t designed to monitor cloud health and respond on-the-fly
with the appropriate actions. However, if your tool is built to do
this, you’ll find cloud-based testing is a fantastic new way to
reliably test complex websites without breaking the bank!

Tweet This Post 

No tags