In preparation for a recent load testing engagement, we started talking about the company’s goals. The DBA said something really insightful:
“We want to get to 1,000 page requests per second, which is around 5,000 batch requests per second. We’re testing with all of our caching turned off, because we want to simulate what might happen on Black Friday if our caching layer fails – we still want to be able to sell stuff.”
Read that again, because there’s a lot of juicy details inside there. (I was so impressed by the DBA’s acumen that I actually wrote it down!)
The DBA knew that the real business metric isn’t queries – it’s how many pages we can deliver to customers. He also knew how many queries an average page requires in order to render.
He also knew that he can’t cheat – he can’t use all kinds of cool caching techniques in his tests because the caching layer could fail, and he has to still be able to deliver.
That’s a really admirable set of goals, but on the flip side, it means that if you’re designing hardware to satisfy peak loads like Black Friday, and you’re trying to survive even if caching is off, you’re probably only using 5% of your hardware capacity at any given time the rest of the year.
This is a business problem, not a technology problem, but it’s the reason why Infrastructure-as-a-Service (VMs in the cloud) are so doggone appealing to managers. As Black Friday approaches, you provision a new super-beefy VM, set up SQL Server on it, migrate your databases over to it, and have plenty of horsepower to survive the holiday loads. After the rush subsides, transition back over to a dramatically cheaper VM instance type, and go back to your regularly scheduled programming.
3 Comments. Leave new
We do this sort of testing regularly. We found that there’s no substitution for a really good, very large, and very complete set of test data that looks as close as possible to real data without being real data. There are great tools available for creating test databases (we use one from Red-Gate and another from DevArt) and we download lists of last names from the census department and get lists of first names from other web sites, etc.) to use with it.
This way, we can keep our (duplicate test) server just as it will be when the real load hits it – so this means caching will be turned on. If you have simplistic test data (tester1, tester2, etc.), your test results will not be representative because the caching may not work as it would when it is working with real data (just as your caching will not work the same in the US SouthWest as it will in the Northeast – try to guess why?)
Also, you might want to test VM vs. hardware side-by-side. Hold everything else constant. Then use them with your application at differing load levels.
Could you elaborate on what tools you’re using?
We’ve used Red-Gate’s “SQL Generator” and devArt’s “dbForge Studio for SQL Server” – both, in their newer versions, have gotten very flexible and allow you to create very life-like data. I don’t get paid by either of them, I just am impressed by how much better the testing process goes when you have really good test data.
It is also easier to spot errors when the test data looks like real data because the results of the tests pop out as not making sense, especially to those who know the application well. If they are looking at test data that looks like “person 1”, “person 2”, etc., errors will be missed more frequently.