I know a lot of DBAs, but it’s really, really rare that I’ve seen one get fired. I don’t think it’s ever happened during one of my consulting engagements, and I’ve seen some incredibly horrific database disasters (not to mention a whole lotta ugly near-misses).
So I asked Twitter:
Thinking about a blog post. When do you fire the DBA? What offense would seriously make a manager do it, something so bad they lose it?
— Brent Ozar (@BrentO) February 26, 2014
@BrentO lost data. consistently slow restores. ever increasing hardware budget but little/zero demonstrable business gain, devs complaining
— Phil Campbell ? (@Philoushka) February 26, 2014
@BrentO I guess that's the most egregious offence, isn't it. From a mgr's POV, that's the easiest tell. Obv. multiple offences contributed.
— Phil Campbell ? (@Philoushka) February 26, 2014
Not Telling Management About Problems
@BrentO When they don't tell you (the manager) about the problems.
— Stuart Miller (@mecheph) February 26, 2014
@BrentO There are many reasons you can't fix a problem. but you darn well better know that it exists.
— Stuart Miller (@mecheph) February 26, 2014
That’s fair – I can understand if someone’s databases have problems that they don’t know about, because we can’t all know about every risk. But if you do know about a risk and you don’t inform management, that’s bad. Which leads to…
Doing Dirty Deeds
@BrentO I think the words Intentional and Malicious would most likely be involved.
— Anders Pedersen (@arrowdrive) February 26, 2014
https://twitter.com/buckwoody/status/438678817058865152
@BrentO crimes involving the data they are sworn to protect.
— Bob Pusateri (@SQLBob) February 26, 2014
What if someone came to you and offered you fifty thousand bucks for a copy of your company’s database backups? Or more realistically, just asked you to up their login permissions to see more databases as a favor?
The devil doesn’t come wearing red horns and a cape. He comes as a friend who needs a favor and will pay you handsomely for it.
Being Untrainable
@BrentO Perhaps a long line of small offenses, like not knowing how to DBA. That happened at a friend's company. Person lasted 6 weeks.
— Matt ???? (@mvelic) February 26, 2014
@BrentO They had misrepresented themselves to get a senior position and couldn't do the job.
— Matt ???? (@mvelic) February 26, 2014
@BrentO @mvelic After I tell you something for the 4th or 5th time, and the same mistake is made….
— Joe Fleming (@muad_dba) February 26, 2014
Not Having Backups
@BrentO Besides the egregious? (e.g. DBA says regular backups are all good but when pressure's on they don't have a backup to restore from)
— Kendal Van Dyke (@SQLDBA) February 26, 2014
This one’s interesting because it happens a lot. No, seriously, it’s the first thing I check when I start a consulting engagement, and maybe 1/4 to 1/2 of the time, the backups aren’t working the way the DBA assumed.
After all, think about your own databases – you’re just assuming the backups worked okay last night because you didn’t get an email. If someone walked into your cube right now and checked every database for a clean (non-corrupt) backup that made it to tape or offsite, are you ready to bet your job on it? Especially if other teams are involved in sweeping the backups off to tape?
I’m much more tolerant of this mistake now because I see it so often. DBAs get distracted by performance issues because that’s what users complain about. Nobody complains about missing backups until it’s too late.
Making a Fatal Mistake
@BrentO @mvelic I've seen folks walked out the door for doing things like rebooting a prod server during the day.
— Allan Hirt (@SQLHA) February 26, 2014
@BrentO @mvelic It was an accident. All boils down to policies, procedures, tolerances, and stated consequences.
— Allan Hirt (@SQLHA) February 26, 2014
I love those last two words from Allan – stated consequences. If the company makes it clear ahead of time that certain mistakes are unforgivable, then yep, that can be a recipe for firing. If you’re in that kind of shop, you’d be wise to print out that list of unforgivable mistakes in a really large font and stick it to your wall near your monitor as a constant reminder.
Not Being Customer-Friendly
@BrentO Chiming in- Let go a DBA who angry with user's lax passwords, reset everyone's with a 16 char varying combination without notice.
— ??Kellyn Command Line ?? (@DBAKevlar) February 26, 2014
Over on DBAreactions, I make a lot of bad jokes about developers, SAN admins, network admins, sysadmins, and your momma.
The reality, though, is that I love these people because they’re struggling just like us DBAs are. They’re working to get better, and I have to help them get better as part of my own struggles. As much as we’d like to think we DBAs know everything about everybody else’s jobs, often our attitudes create a problem that’s a bigger liability than our perceived knowledge.
Not Improving Despite Guidance
https://twitter.com/buckwoody/status/438678939234750464
Buck sums up the real thing DBAs need to be aware of.
Ideally, you build your own learning plan to up your game and keep your data safeguarded. You sharpen your own knives, and you train for the day that you have to respond to the unthinkable outage.
Less-than-ideally, your managers notice that your knives are those plastic ones you get in coach class on airplanes, and they write you a plan as part of your annual review. You make progress on it with that jump start, and you keep treading water in your career.
Or not-ideally-at-all, you put your company’s data at risk, they take you to Human Resources, and you sign off on a very urgent plan to get your learn on.
So, go get your learn on before the company notices.
16 Comments. Leave new
I see a lot of emphasis on values and integrity in the comments, and rightly so. A DBA must be trustworthy above all else. It seems like we’re heading toward the day when even private corporations are going to pay for background investigations and credit reports for anyone entrusted with company data.
Not having backups and doing anything illegal goes without saying. However, more often it’s soft skills that can do a person in.
People can be taught SQL Server and you can give people tasks to do things at their skill level as they learn. However, if someone comes across as difficult to deal with for any reason, your customers aren’t going to want to work with them. It’s obvious to know who a customer is for consultants, but many DBAs don’t know who their customers are or how to talk to them properly.
It’s easy to come across as bossy or rude. After all, your in a position to know a technology very well that many people (developers and their managers) use a lot. It’s your job to teach them how to do it right and understand why things matter. The developers are your customers.
It’s easy to overwhelm business users and managers with details. You can’t leave them waiting until you have a full solution then leave them confused with your deep dive into SQL Server’s internals. Instead you need to acknowledge their concerns immediately, tell them what’s happening using their language (not nerd lingo), be honest with them even if it’s not the answer they want to hear. Again, they’re your customers.
No one stays in business if their customers don’t want to talk to them, DBAs aren’t an exception. It may take a lot to get fired this way, but it also takes a lot to change this. I struggled with talking to business users properly for a while, and, although it wasn’t bad enough to lose my job, it held me back.
I heard, second hand, of a case where a newly hired DBA was promptly shown the door because, as the new sheriff in town, this person decided to revoke elevated rights from all accounts on a given server, without notifying anyone first, which of course caused lots of things to fail, resulting in significant business impact. While the idea of working to reduce rights was noble, in considering the several possible ways to achieve this goal, he chose … poorly.
This makes sense. Locking things down so that an environment is “controlled” is one of the first things I do even while performing an environment sweep. This simply can’t happen without buy-in, though. It looks ugly and, as that poor cowboy noob experienced, can be a three-strikes-in-one offense (breaking working stuff and having a business impact being the ultimate problem with the wellmeaning cowboy action – though the cowboy attitude isn’t healthy, either).
The answer is to not fire: managers should be working with their “reports” on a continuous basis. Doing so creates the condition that the DBA knows whether they are doing the job, and a mutual decision to part ways becomes obvious to both parties. See this great article on “Performance Reviews” (and why they are useless) for details: https://www.linkedin.com/today/post/article/20140206114808-15893932-how-adobe-got-rid-of-traditional-performance-reviews?trk=object-title
You can get fired on a 24/7 shop when you execute DB DDL – Schema changes to the Publication DB which is replicated and by mistake you replicate the changes and all systems goes down because the network speed is slow at the subscribers sites outside of the permitted time window. NIGHTMARE!!!
Spilling your coffee on the SAN
You should fire someone because you no longer trust them. In business, you should be too busy creating value to second guess the people you work with. Sure, mistakes happen, but if you still trust that person to recover, go ahead and keep them. If you no longer trust them, they have to go. Really, in every scenario I saw above, the root of the problem was lost trust.
John – interesting, so you’re saying you trust new hires by default? Or is there a period where you gradually break them in and give them more access over time? How do you track trust levels during the employee’s new hire and evaluation process?
There is a professional level of trust you start with. If the employee doesn’t have that out the gate, your hiring process is too abrupt. Beyond that, yes, trust is developed over time. As your trust of the employee grows, you give them access to more things needed to do their job well, or new responsibilities. Regarding the new hire and evaluation process, it’s pretty simple for me. If I interview you several times, and you give the same or consistent answer, this helps build the initial trust. Second to that in the matter of trust, the interview process should the approximate weighted intrinsic and extrinsic factors for why the person specifically wants to work at your establishment. Then, finally, your capability portions of your interviews should reveal ability to follow through on creating value. Some people like Dave Ramsey use as many as 12 interviews for new hires to provide an accurate measurement of trustworthiness before hiring. His hiring process includes calling family members and spouses, as well as financial background checks.
I tell the people I work with that they have my trust until they abuse it. It may sound crazy, but if I can’t trust you, I don’t want to work with you at all. This means you should never lie to me – ever, about anything.
In short, I don’t think the process to work with or fire a DBA should be inherently different than any other technology worker in your company.
John – yeah, I totally understand, but this just doesn’t work at scale. HR departments won’t fire people based on a touchy-feely truthiness-trustworthiness gauge. That’s a recipe for lawsuits. After all, trust is a two way street – what if YOU are the problem because you are biased to trust some people over others, or it’s too hard (or too easy) to win your trust?
I’ve been around companies where everybody trusted That One Guy because he was so nice, down-to-earth, friendly, admitted his mistakes – but he was utterly and completely incompetent. After all, didn’t everybody trust Bernie Madoff right up to the end?
I would agree with the loss of trust being the main thing in most cases. As for Brent’s question, you have to trust new people from the start. It’s a more extreme case (maybe), but think about companies that hire an outsider for a CEO or CIO position. Yeah, you go through a good interview process, read through references, etc., but, in the end, the new guy or the consultant is still largely an unknown that you have to trust.
In the beginning it takes less to lose that trust, and it’s up to the individual on how much trust they build up in the company’s eyes. Since this is more of a feeling, I’d say there’s no real way to track it.
There are some examples of this above, even just going through the comments. Jose Rivera’s comment, you probably wouldn’t fire a DBA for doing that once, but after the second time would you start to not trust the up-time you’ll have with that DBA in control? Chuck Rummel talks about untested and poorly thought out permission changes that sound like they had no quick and easy rollback scripted out ahead of time, which again would lose trust of the availability of the database.
However, the real reason I got on here was to ask if you’d fire a DBA for singing at work. I could imagine someone in these comments singing very odd and infectious songs at work, like “Mr. SAN-man, build me a LUN. Make it the fastest that I’ve ever seen….” This person may or may not have written this post, so this comment may or may not last very long here.
Steve – HAHAHA, I don’t delete comments, but I have totally been that guy singing at work.
Or “SQL Agent Man, SQL Agent Man, they’ve given you a login and taken away your name…”
I think that competence is the hardest thing to gauge with IT roles, in general, for most management that is not particularly familiar with the ins and outs of a highly technical role. Beyond the obvious backup/restore (DBA Job #1), it’s hard to tell competence in an interview or early on in the hire.
Ultimately, if you take a little time before you hire someone as a DBA, or any tech role, to define the job at least a little, then you can tell much better whether or not person is doing what you expect. However, every job seems to change pretty quickly, so a good manager should be paying attention to seeing if people are a good fit for the responsibilities given, and if they are happy with the responsibilities. I’ve seen guys who are great at their job, but hate customer service, and yet get drawn into a lot of BA type work that leads them to be disgruntled and they start performing poorly. For others (like me), being stuck in the same role with repetitive tasks can bore you into becoming a bad employee.
It’s important to make sure your team works well together. Even if an employee is very proficient, if they constantly stand in the way of progress, exhibit an unacceptable amount of arrogance (which is different than confidence/competence), or just don’t seem to be able to deliver on their part, they should go.
My biggest beef is that it seems that there are generally 2 ways to ensure you keep your job, either by performing above and beyond the call of duty, or by forming “political” alliances. It’s the latter group that tend to rub the first group the wrong way and generate a lot of animosity, which can bring down the whole team. It’s one thing to carry someone’s load for a while, but when that becomes an on-going issue, and management is blind to it, then the whole thing starts to fall apart.
If they make the same serious mistake multiple times and try and lay blame on others each time. If they don’t learn and don’t want to learn. If people feel they just can’t be trusted to carry their share of the load. Lose them before you lose the people who are carrying them.