Back in the early 1980s, Dr. David DeWitt – who you might know from past PASS Summit keynotes, formerly of Microsoft (LinkedIn) – was working on measuring database performance. His team wrote a benchmark, ran it against Oracle, and Oracle didn’t fare well in the results.
Oracle reacted as one would expect – they were furious, and wanted DeWitt fired. You can read DeWitt’s remembrance of it in the article DB Test Pioneer Makes History by Timothy Dyck.
To prevent it from happening again, Oracle inserted a clause in their licensing that basically said you can’t publish benchmarks without getting Oracle’s explicit approval. David Wheeler has an essay about why the DeWitt clause censorship should be illegal, and I bet a lot of you agree.
And right about now, I bet a lot of you are going, “Yeah, that nasty mean Oracle is just nasty and mean.”
Except check out SQL Server’s End User Licensing Agreement, which includes the line:
BENCHMARK TESTING. You must obtain Microsoft’s prior written approval to disclose to a third party the results of any benchmark test of the software.
You agreed to it when you installed SQL Server. If that slipped your mind, you can find it in %Program Files%\Microsoft SQL Server\140\License Terms\ (or replace 140 with whatever version number you’re on.)
Open source databases don’t have restrictions like that, so the open source world is chock full of benchmarks comparing different versions, features, hardware, cloud providers, you name it. But the closed source arena? Not so much.
I understand where Microsoft is coming from – if anybody could publish benchmarks, then people with a marketing agenda would publish biased numbers using questionable methodologies in order to further their corporate goals, and that would be bad.
14 Comments. Leave new
So, is that to say that anytime a blogger has written about SQL Server performance (or lack thereof) in specific circumstances, either to disclose the issue or to show how they fixed it, that Microsoft could penalize them for that if they didn’t get written approval first?
PM – generally speaking, benchmarks are when you compare things to each other, like different database platforms or different hardware.
Legend has it that various undocumented Oracle parameters exist only to speed up benchmarks. In any case, they appear to be used. I’m not saying other vendors don’t, I just have worked with Oracle for over 20 years and it’s been noted by others for at least that long.
The author below (Rich Niemiec) has written a number of books on Oracle performance.
https://logicalread.com/oracle-11g-top-13-undocumented-initialization-parameters-mc02/#.WvHiMIjt5hE
“Although the following warning describes well the risks associated with using these parameters, I note that the fastest RAC TPC (Transaction Processing Council) benchmark uses 17 undocumented parameters, as do many of the TPC benchmarks that I’ve seen.”
It lists some but I recall that others more dire and that shouldn’t be used even on test systems, let alone on production systems, were used in the past to get better benchmarks. As I’ve seen in the past, applications blindly ported to a different database with have wildly different performance between two different databases so TPC benchmarks aren’t the final word.
Even vendors selling monitoring tools get into the act.
https://www.itconductor.com/blog/will-hana-dominate-in-sap-performance-over-oracle
There’s no mention of the Oracle DB native compression being used or the benchmarks when memory is equal. Not that I’m claiming dishonesty but they are selling software.
Apples to oranges? Maybe.
“… people with a marketing agenda would publish biased numbers using questionable methodologies …”
Lawnmowers come to mind. In the old days, the manufacturers published horsepower ratings…until they got so inflated that no one believed ’em any more. So they were driven to publishing torque.
Sort of Gresham’s law in reverse, I suppose.
Very nice article. But exist one tip, you can measure perfomance for example in virtual parrots like in this awesome article: https://blog.jooq.org/2017/03/29/how-to-benchmark-alternative-sql-queries-to-find-the-fastest-query/
“As always, disclaimer: Some commercial databases do not allow for publishing benchmark results without prior written consent. As I never ask for permission, but always ask for forgiveness, I do not have consent, and I’m thus not publishing actual benchmark results.
I have anonymized the benchmark results by introducing hypothetical, non-comparable units of measurement, so you cannot see that PostgreSQL is totally slower than Oracle and/or SQL Server. And you cannot see that SQL Server’s procedural language is totally uglier than PostgreSQL’s and/or Oracle’s.”
Yeah, I’m going to be honest: that’s not really readable, so I don’t think anybody’s legal department is going to go after him for that, hahaha.
Great article.
So this is DeWitt. You are being a little unfair to Microsoft. When I was lucky enough to still be working for the SQL Group we actually had an active discussion (with Rohan, Nigel, and Shawn as I recall) about SQL’s “DeWitt” clause. We all felt that we should drop the clause but no one end up pushing the issue so it never got fixed.
Maybe Rohan could move the ball forward on this issue.
Recently a group of us at MIT has been running benchmarks on the various AWS offerings (Redshift, Redshift Spectrum, Presto, and Athena). It is interesting to note that none of the these services requires the user to accept a no benchmarking clause as far as I have been able to figure. We are about to publish. It will be fun to see if sparks fly once again. It is unfortunate that the Snowflake folks decided not to participate 🙂
Howdy sir! Good to hear from you.
OK, so to be fair to Microsoft – some folks have positive feelings about removing the clause, but it’s still there.
That’s kinda like how I have had a discussion with myself about not having a beer with dinner tonight, but uh, yeah. Waiter! You know what they say about the road to hell – it’s paved with good intentions about removing the DeWitt clause. 😉
I’d love to know how many organisations make a jump from one major DBMS to another for the productive systems, say between DB/2, SQL Server, Postgres & Oracle. Given how much an organisation invests in DBAs and training, the idea of swapping systems and having all affected learn a new system and having them rewrite all of the automatic processes seems very daunting and not at all profitable, especially for a 5% speed gain.
Agree with that performance is not valuable factor, but too many (also huge) organization in Russia now migrated from Oracle and MSSQL to PostgreSQL due to price cost and sanctions.
Our organization is trying to move from Oracle to SQL. We have staff skilled in both systems. But our applications are all “off-the-shelf” products, not custom-written. And for us, it comes down to two issues. Cost and complexity. (Performance is difficult to improve when it’s a closed-source product.) SQL is cheaper and, generally speaking, easier to manage.
So that’s why we look to move off Oracle anytime we are considering upgrades or replacements.
DataBricks just will leave out the Dewitt Clause and apparently Azure and AWS also are dropping these clauses. This is bold and great, or as they put it: the future is open …
Benchmarks are useless, unless they match your workload. Do your own.
A faster db engine also needs to be compared to the cost of development in that platform, as well as the additional costs for a DAL. I am expressly prohibited from publishing mine.