Back in the early 1980s, Dr. David DeWitt – who you might know from past PASS Summit keynotes, formerly of Microsoft (LinkedIn) – was working on measuring database performance. His team wrote a benchmark, ran it against Oracle, and Oracle didn’t fare well in the results.
Oracle reacted as one would expect – they were furious, and wanted DeWitt fired. You can read DeWitt’s remembrance of it in the article DB Test Pioneer Makes History by Timothy Dyck.
To prevent it from happening again, Oracle inserted a clause in their licensing that basically said you can’t publish benchmarks without getting Oracle’s explicit approval. David Wheeler has an essay about why the DeWitt clause censorship should be illegal, and I bet a lot of you agree.
And right about now, I bet a lot of you are going, “Yeah, that nasty mean Oracle is just nasty and mean.”
Except check out SQL Server’s End User Licensing Agreement, which includes the line:
BENCHMARK TESTING. You must obtain Microsoft’s prior written approval to disclose to a third party the results of any benchmark test of the software.
You agreed to it when you installed SQL Server. If that slipped your mind, you can find it in %Program Files%\Microsoft SQL Server\140\License Terms\ (or replace 140 with whatever version number you’re on.)
Open source databases don’t have restrictions like that, so the open source world is chock full of benchmarks comparing different versions, features, hardware, cloud providers, you name it. But the closed source arena? Not so much.
I understand where Microsoft is coming from – if anybody could publish benchmarks, then people with a marketing agenda would publish biased numbers using questionable methodologies in order to further their corporate goals, and that would be bad.