You’ve heard these terms thrown around about storage, and you’re not sure what they mean, and how they relate to each other. Let’s make it simpler by using an analogy.

Say you want to measure a shipping company.

You’re trying to decide whether to use DHL, FedEx, UPS, or your local postal service.

You could measure them by sending me a gift – after all, it is the holidays, and I do a lot of stuff for you year round, and it’s probably the least you could do.

• You place one box outside
• You call the shipping company to come pick it up, and start the clock
• When I get it, I’ll call you to confirm receipt, and you stop the clock

The length of time between when you called, versus when I acknowledged it – that would be how long it takes the shipping company to move exactly one package. Let’s say – just to say – that you sent the package overnight. You called the company on Wednesday at 10AM, and I received it Friday at 10AM.

Our performance tests are:

• Input/output Operations per Second (IOPs): 1 package every 2 days
• Latency, how long an operation takes: 2 days
• Throughput, how many operations are coming through at once: 1 package at a time

That test only makes sense
if you’re sending one package at a time.

Sure, in rare cases, you really do passionately care about how long it takes to send me a Christmas gift. (But seriously, why did you pick a 55-gallon drum of Froggy’s Fog? I don’t even own a bubble machine.)

But most of the time, when you’re benchmarking shipping, you’re doing it because you want to ship a lot of things, to a lot of people, at the same time. And indeed, if you try to ship more things at a time, you’re going to get dramatically different test results.

If you set out 10 shipments of Froggy’s Fog, and call the shipping company for a pickup, your metrics magically become:

• Input/output Operations per Second (IOPs): 10 package every 2 days (goes up)
• Latency, how long an operation takes: 2 days (this stays the same)
• Throughput, how many operations are coming through at once: 10 packages at a time (goes up)

A good load test involves lots of simultaneous operations – ideally, a similar number of operations that you’d actually need in your daily work, but not too many. Because imagine what happens if you try to put 100 drums of Froggy’s Fog out for pickup, but the shipper’s default van can only hold 10 drums. The van driver will arrive, find too many drums out, and just maybe tell you, “Sorry, we’re going to have to come back tomorrow with a bigger truck to handle this much.”

Or, maybe on the other side of the test there’s a problem. Maybe my front door just doesn’t have that much space, and after dropping off 10 drums, the shipper says, “There’s no space left here – we’re gonna have to drop the rest off tomorrow.”

This isn’t necessarily a shipping company problem – instead, it’s a problem with our planning and our infrastructure.

Storage tests have the same problems.

If you only test with one storage request at a time, you’ll see artificially low IOPs and throughput numbers. Sure, the latency is roughly accurate for one request at a time – but that is rarely a useful number by itself.

If you test with too many requests at a time, you’ll surface different bottlenecks. You may not have enough storage adapters, paths, or even volumes to really take advantage of the storage device.

Even your package size selection influences the test. What if, instead of sending out 55-gallon drums of Froggy’s Fog, you chose smaller gifts, like a bunch of yodeling pickles? You might get an artificially high number of packages shipped per second, but if your workload really does involve 55-gallon drums, then the pickle test isn’t accurate. Similarly, you can see storage tests that are purposely trying to achieve artificially high IOPs requests, so they’ll do absurdly tiny I/O size requests.

It’s hard to plan storage for SQL Server. On Thursday, I’m doing a free webcast with Pure Storage to talk more about how I approach the problem when building and consolidating servers. See you there!