Recently I saw a tweet that said, “If you can’t read a query execution plan, you’re not a production DBA.”
I love execution plans, and there are really great reasons to learn to read execution plans, but I disagree completely.
Database Administration is not all about performance
Performance is only one component to database administration. Let’s take Jill as an example.
Jill is a SQL Server DBA who focuses on High Availability and Disaster Recovery for a large company. She works to keep her knowledge and skills sharp on backups and restores, Windows Failover Clustering, and High Availability Groups. She designs mission critical systems to meet high uptime requirements. She also designs and runs a lab where she proactively causes outages and troubleshoots them, corrupts databases and repairs them. She uses these experiences to build processes for front-line engineers to respond to production incidents and trains them to keep systems online as much as possible. Jill is excited about learning more about virtualization and SAN replication in the next year and has created an aggressive learning plan.
Jill’s pretty awesome, isn’t she? Is Jill magically not a Production DBA if she looks at an execution plan and doesn’t know what it means?
If you’re just getting into database administration, we don’t recommend that you start with performance as your primary focus. We recommend that you start at the base of the database Hierarchy of Needs. As you work your way up, decide where to specialize.
Don’t de-legitimize DBAs who are Different than You Are. Talk to Them!
It’s great to love what you do. Just remember that database administration is a really broad field, and the way you DBA isn’t necessarily the way everyone has to DBA. (Sure, DBAing is a verb. Why not?)
Let’s not fall into the trap of thinking “I’ve got to have this one skill so I can get into the cool club.” Instead, let’s get excited about the fact that there’s so much to learn that we all have to make tough choices. Let’s seek out the people who choose differently than us, and learn from them, too.
28 Comments. Leave new
I think perhaps the tone of the statement is too accusatory. However we shouldn’t be afraid to identify something as a core skill of a profession.
Now with regards to query plans in particular, even if you only support databases where you cannot alter sorocs or indexes, you still have to be able to tell your vendors you are wrong and this is why.
“you still have to be able to tell your vendors you are wrong and this is why”
Whoa…speaking of accusatory…
As a data architect for an enterprise ISV I can tell you I’ve had these conversations with customers’ DBAs LOTS of times. To be honest, about 80% of time we are actually right and we can explain our reasoning to the DBAs to their final satisfaction. In these cases, trade-offs were made and the DBA is seeing the negative side effect because the positive is obfuscated. In the other 20% of the cases…well, we all do dumb things now and then. I appreciate anyone who can point out when I’m wrong if it makes me more profit. It’s just a lot more pleasant to work with people who don’t immediately resort to “you’re wrong, here’s why”. Much better to say, “here’s what I see and how I would normally fix it, can you give me any reason why this would not work for your product Mr Vendor sir?”
I try to tactfully phrase “you still have to be able to tell your vendors you are wrong and this is why” in a non-accusatory manner when I approach vendors. My point was you need the tools to be able to say more than “your app is slow,” and just because your contract with the vendor says you can’t touch the database, doesn’t mean you can’t take a peak how the sausage gets made.
If one wants to grow as a human, they must be constantly peaking under the hood.
I spend very little time worry about vendor database performance tuning. It’s a waste of time and quite possibly may invalidate your support. I’ve had vendor applications perform poor, and their resolution was to disable auto-update stats and change fill-factor across the board to 70%. I suggested we first remove unnecessary indexes and perhaps compress some of the larger ones – the server was certainly not CPU bound. They wouldn’t even entertain my thoughts until I honored their suggestions, despite my actually having data to support my thoughts.
I would not go so far as to say 80% of the time vendors are right – perhaps you guys have a talented team. That would mean you are the exception to the rule when it comes to vendors.
Every vendor is different. Collegiality in interactions is most important. I enjoy when a customer DBA wants to talk to me about a perceived short-coming in our product. Like I said, anything to make a better product.
I agree with the concept of a set of core skills, but I honestly don’t think they’re always required because the job “Database Administrator” is so very broad, and people can come into the profession in many different ways.
Perhaps a developer transitions over to do database design, index tuning, and query performance tuning. The specialize on a high transaction production system that requires 5 ms latency.
But they never learn to do a backup, because in their organization a whole different team takes care of that.
I’d say that’s totally OK too: that person could be called a Database Administrator just as much as Jill in the example above, even though there’s no overlap between the two.
Where I think the concept of “core skills” is most useful is for people who are trying to break into a position and get a job, where it doesn’t happen naturally and organically. But really, core skills = common skills in that case.
Sure everyone is an accidental DBA, and there are reasons any given person might not ever need to do any given thing. I’ve always sought to be a generalists though, so I tend to seek broad knowledge to fill i nthe gaps. Depth comes naturally in the areas I actually do stuff in.
For the general title “Database Administrator,” I would agree that it’s foolish to assume a particular core skill. When you say “Production DBA,” I think you must be proficient on backup/restore. I know this is nit-picking, but that production prefix is a biggie. Just as I would expect a “Development DBA” to be able to do query tuning.
Oh, that’s interesting. In my mind, “production” often means “has sysadmin access and knows better than to violate change control rules”.
But I totally can see how in 99% of cases it would be a person who would know backup / restore.
I think that’s a great description of how 2 people can both be DBAs but have differing skills sets/areas of interest.
I have had the fortune to work with DBAs who were excellent at performance tuning, drilling down into AWR reports (Oracle) and execution plans, etc to find the smallest performance problem. However, that’s not my thing at all and I’ve found myself leaning towards clustering/replication, etc. That’s not to say I can’t do a code review from time to time but that’s the way my career has developed. Hence I’d call myself a Production DBA, compared to a Dev DBA.
SPOT ON.
Corollary…just because you can read an execution plan doesn’t mean you know how to fix it properly. I can read Ayn Rand…doesn’t mean I understand it.
Or…”database experts” who know how to query the missing index DMVs and then apply those recommended indexes to prod systems without any testing or consideration for overlapping indexes, effects on storage costs, increased deadlocking possibilities, etc.
You know, I think the thing I love most about execution plans is that there’s always something new that stumps me in them. Forces me to learn more and always makes me think, “I don’t really understand this monster yet.”
THANK YOU, Kendra. It can be quite discouraging to be faced with attitudes like that. Sometimes I feel like a fraud because I don’t know every single little thing yet.
I swear, every week I learn at least one thing that surprises me in some way. Once upon a time I thought surely that experience would stop happening if I “really” knew what I was doing. But it turns out that’s not the case!
And have you ever had that experience where you look back at something you wrote years ago and think, “Wow, I was a lot smarter than I gave myself credit for”?
There’s a surprisingly fine line between being too self-satisfied and not giving yourself enough credit, I think.
I hear ya Kendra – this week it was spinlocks and diving into auto-update logic more to fully understand it (thanks to Erin Stellato I think I have a much better understanding now!).
There are hundreds – probably thousands of things we as DBA’s are responsible for being the SME on at our places of employment. There is no way on Earth we can be experts on every single topic. Fortunately (as we’re doing now) we have a great community to collaborate with. I’m never too proud to ask the community for a second opinion on something.
The key is when you realize there is something you should know that you shouldn’t is to silently trudge forward and learn that. Yes we all suffer from impostor syndrome, but if you can channel that self doubt in a positive way, you emerge a more competent practitioner.
The heart of the post is that each of us must decide what we “should” know, based on what our company needs and the shape of our own career. We don’t have to live up to some idealistic idea of what a job title demands or go chasing every possible thing we *could* know.
Because I *think* you, Justin, should know something – and you don’t – doesn’t mean you are any less a DBA. The term “DBA” is a very broad spectrum. Sometimes you are required to know a certain portion of the profession because you have thousands of instances, and sometimes you do it all – often learning the more advanced topics on-demand. Believe me, many of us were DBA’s when you were just wrapping up high school so I’m certain we could come up with a few items that we consider “DBA” worthy – that you just might not know.
Seems the Tweet got removed? 🙂 can’t find it in search
LOL. The tweet is still out there. (Twitter search is an interesting thing, I had to find an old link to it and get to it that way.)
I didn’t link to it on purpose, I didn’t want cause any potential flood of replies to one person. And I saw a few people agreeing with the statement and saying similar things, too– it wasn’t just one person who felt that way.
I feel the same way.
It’s funny when someone invests two decades into the technology, and then upon reaching the top, sees fit to deride and ridicule everyone else who is only a few years into the same journey. “No MCM? Well you aren’t a DBA!”
Sadly this is a large part of the top-end of the SQL Server scene. Oh sure you have amazing people like Paul Randal. But you’d be surprised at how negative some of the other famous names are, to the point I feel bad even reading their articles.
Luckily for me being somewhere at the bottom I only hob knob with like-minded people. You know those who respect colleagues who have a brain and use it, instead of just judging them on whether it’s full yet or not?
I think the original statement is overreaching perhaps for dramatic effect, and can certainly convey the wrong message to people who are very legitimately under the DBA umbrella, but specialized in other areas. For situations where you aren’t highly specialized or siloed into a specific subset of DBA duties, though, or for folks just starting out, though, I think a softer version of the sentiment may be in order…maybe, a “generalist” DBA who hasn’t yet started learning and using execution plan analysis is missing a highly useful tool on his or her toolbelt. And they are missing out on the “fun” that is performance tuning! I spent most of my early time as a DBA in a “systems DBA” role and left the performance stuff to others, but I’m glad I finally started tinkering with performance tuning…its actually quite rewarding to apply that tiny little index and see the plan turn upside down and executions speed up. Also, just about every DBA job description around here includes that boilerplate blurb about mentoring developers, and even if it isn’t a specialty, its good to be able to “talk the language” on scans, seeks, lookups, etc., for when that developer walks up to your cube to ask why his or her query is so slow.
So I’m a big fan of all of us DBAs being encouraged to learn stuff we might initially think lies outside our specialty, sometimes we end up discovering things we actually enjoy and are good at. Which I guess means I should eventually force myself to learn SSAS….yikes.
Well said.
Except for the Analysis Service part. No way man! 🙂
An interesting thought experiment might be to take “If You Can’t Do X, You’re Not a Production DBA” and then solve for X. Is there any skillset that is a non-negotiable for DBAs across the board, or is our occupation so diverse now that there really isn’t one common “must have” skill? I was thinking, maybe a basic understanding of T-SQL (or PL/SQL or whatever the RDBMS dictates), but I wouldn’t be surprised if there could be an exception to that as well. I was thinking about the so-called soft skills, but I’ve met a lot of DBAs who can be very, very different with different combinations of these as well…some natural problem solvers, some great at customer service, etc.
I’m generally very cautious about people who tell you everything they know, or those who want to put you down for what you don’t know. It’s been my experience that the real experts are those who have a firm grasp of what they DON’T know.
I think part of this mentality stems from performance-oriented job descriptions; most mid-to-large size companies will have very explicit job descriptions, and may even have skill expectations based on experience (e.g., Jr DBA, DBA, Advanced DBA, Senior DBA, DBA Advisory). However, good managers (and most employees) should know that no employee should match their job description 100%; you want to give people room to grow, and you want to bring different experiences to the table. You don’t want to assume that just because a person doesn’t have a particular skill set (but they have adequate coverage of other skill sets) that they shouldn’t be in that position.
I think it’s also a disconnect between “hands-on” and “conceptual” knowledge; hands on knowledge usually implies the ability to do a task almost automatically, whereas conceptual knowledge refers to the understanding that a task is part of my responsibilities, but I can go look it up when I need to. I don’t routinely install new servers, but I could figure it out if I needed to. I still call myself a DBA, even though I haven’t done a server install in probably 5 years.
Interestingly enough, I expect more hands-on experience at the lower levels than at the higher; in my opinion, Jr. DBA’s should be able to monitor backups and restore databases with minimal guidance. An Advisory DBA should be conceptually aware of how to do a backup, but very rusty in practice. I need mechanics at the lower level, and engineers at the top. Why? Because they should be focused on understanding the big picture of the organizational database needs, and they should know where to go look something up. It’s the Kelly Bundy effect; only so much space for information. The more you know about the broad topics, the less room you have for immediate recall of specifics.
“Employee must have 5 years of experience in MongoDB, SQL Server Hekaton, and Fixing Printers.” Wait, whaaat?
Somewhat related: as a consultant I have all these weird little edge cases and symptoms I know about. But I also know that they’re edge cases and don’t happen to everyone. I actually make fun of myself if I jump to the “it must be spinlocks!” conclusion and skip the very basic steps of starting where everyone should: What’s running? What’s it waiting on? Are there any messages in the error logs?
If I had to pick just three tools to troubleshoot a performance issue, I might not even choose execution plans. (Wait Stats, Error Log, and Index Info might be it. It’s a toss up between the Index info and the Execution Plans.)
Edit: speeeling
This is an encouraging post for me to read because I am a relatively new DBA and I feel like there is soooo much I need to learn still. My company’s situations with backups could be better only by going to full recovery and doing tlog backups ever few minutes. I got us down from 24 hours of data loss to 1 hour of max data loss using differentials. I’ve learned and implemented some indexing, statistics and perfmon information I picked up from this site and SQLSkills.
I got my start in SQL through SSRS reporting because our application’s canned reports were inadequate. Later I gave input on a flattened OLTP warehouse that was being built so I did not have to join large amounts of tables together to get basic info. It came with no indexes so I had to put into practice what I knew and quick. In light of these things I have been mostly a Development DBA and not really focused on prod machines that were exhibiting regular problems.
Over the last year I have been increasingly asked to delve into problems on the prod machines, sometimes as the problems are occurring. I have little experience with this sort of thing so it feels intimidating. I need to print and make the First Aid checklist from this site my best friend so I have some steps and scripts at my disposal.
At any rate, I feel the pressure of the thought, “Can I really call myself a DBA?” because there is a lot that I know I don’t know yet. I just have to be patient with myself and learn as fast as I can. Looking back on where I was 3 years ago and how far I have come since, helps a lot too.