Brent says: “How many people are in the on-call rotation with me, and can I give them each a technical test?”
Once you get out of the on-call rotation, it’s really hard to go back to the electronic tether. I’d be willing to do it, but only if there are other sharp people to share the burden with me. The questions on the test wouldn’t be trivia – I would just want to hear more about their troubleshooting process, learn what actions they were comfortable taking, and know that I could trust them to make situations better rather than worse.
Bonus points if they have a good change control process that avoids random changes to production servers, thereby ruining my drinking night because some yo-yo wanted to deploy a new version of their non-critical app and suddenly adds an N+1 bug that knocks out services to unrelated apps.
Erik says: “What’s the oldest version I’d have to support, and what are your upgrade plans?”
Back when I started working here, SQL Server 2014 was still pretty new. It also didn’t have what I’d call anything “groundbreaking” in it. There were some okay new doodads, but I didn’t point in awe like “THIS WILL SOLVE SO MANY PROBLEMS”.
That’s not true for SQL Server 2016, and oh-my-dear-sweet-Robert-Smith-wig, it is ever not true for SQL Server vNext. The stuff in those two releases is heart and mind changing. Like, I think people will actually hate Microsoft less because of what they’re doing these days. Maybe. As long as they leave the Start menu alone.
Knowing what’s available in newer versions makes supporting and troubleshooting (especially performance troubleshooting) older versions downright annoying for FTEs.
Tara says: “How often does the on-call DBA get called during his/her week of on-call duty? What is the most common thing that the on-call DBA gets called for in the middle of the night? Which monitoring tools do you have in place?”
Like most production DBAs at large companies, I’ve been through horrific on-call weeks where you don’t get much sleep. As long as the bad weeks don’t happen too often, you can manage it. When every week is a bad week, you may start looking for a new job.
I’ve been in on-call rotations where there are a few hundred SQL Servers. It was rare to not get called at least once while you were sleeping during your on-call week. Most of the late night calls were due to disk space issues. Most of those should have been handled during the day. It’s imperative that the production DBAs be proactive to limit the on-call work. The on-call team needs to have enough time in their day to do proactive work. If they don’t have extra time, then the team is not sufficiently staffed.
Monitoring tools are a critical component of production environments. If I get woken up at 3am for a performance issue, I want to have monitoring data that I can look at to help solve the problem quickly so that I can go back to bed. Even if your company can’t afford a really nice monitoring tool, you can create your own. At the very least, log current activity to a table with sp_WhoIsActive. There are lots of other free tools.
Richie says: “Who would I be reporting to, and what level of interaction would my manager have with the team that I’m on?”
Actually, that’s not true. My first question would be “Why in the hell am I taking a DBA job?” If you are unaware I’m the only developer on the team (official title: dataveloper) so for me taking a DBA job would be a bit out of my comfort zone. So let’s change DBA to developer.
The most important factor for me taking a new gig is who would I be working for. The manager literally defines success for their employees. If you get a good one, they’ll create an environment that encourages your success. If you get a bad one, they’ll give you unrealistic deadlines, change the definition of success, and blame you for their failures.
Modern corporate software development is also team undertaking. I may have one of the few jobs where I’m working on software all by my lonesome. The success of a project isn’t defined by your individual contributions but the completion of the team goals. I’ve been with some companies where my manager had no interaction or control over the team/project that I was a part of. When that happens the manager is getting information about your performance second hand without understanding the full circumstances. Like a child’s finger painting, the performance review becomes an ambiguous, inaccurate mess.
In the end get a manager that you can trust. I’ve worked fourteen hour days for two straight months for a manager that I trusted. I’ve also had a manager that I refused to be in a room alone with him. I think that’s why it was so easy to join Brent Ozar Unlimited. I’ve known Brent for a long time and knew I could trust him. I knew he would be honest with me and let me know when there was a problem. I knew he would create the environment for my success. Please don’t tell him that I said any of this. He’d probably have me walk over hot coals if he knew.
What about you? What would your first question about the job or company be?
45 Comments. Leave new
“How much do I get paid when called out?”.
This isn’t because I’m greedy but if you get paid peanuts for a callout and management therefore don’t care about it, a recurring issue might not get the attention it deserves and the sleep deprivation continues. If there’s a noticeable cost with each callout then you just might get the time and resources to fix it. I get my sleep, the company doesn’t have to pay more money and the system runs better.
That is an excellent point.
Once they made part of the help desk team hourly. Then the work was only assigned to the salaried employees. They complained. So yes, it seems many management people don’t seem to care about oncall hours if they don’t have to pay for it. I once point out to co-worker that if they put in all these extra hours and fought really hard they would maybe get a 3% raise. This was considered really good (still is). Then I worked out that was a very small amount per hour. Really not worth it.
My motto, make everything really reliable so oncall is as little as possible.
Steve – yeah, I do like those payout policies a lot.
” If they don’t have extra time, then the team is not sufficiently staffed.” – been saying this for many, many years during interviews and regular conversations.
My first question – “What does my typical day look like and can I meet the other DBAs today.”
Kevin – yeah, and bonus points if you can meet the DBAs individually, privately, too.
What’s the ratio of Servers per DBA?
At one time, I was the only dba supporting 182 servers…..
Monte – great point!
I took a new position in Jan with a small (staffed) but large (household name) company. They have 2 other DBA’s who are both retired but working as contractors. I haven’t had too many issues, but the one red flag was that I have to learn Progress DB… I thought it was a typo for Postgres… NOPE! (https://www.progress.com/openedge/platform)
Wes – HA! I’ve heard of that! Never seen it in real life myself though.
Oh I have… I have seen scary things, Brent… Very scary things.
Ha… I have years of practice using proutil…. Wish it translated to MS SQL a little better.
What kind of database issues are in your backlog right now?
Or something to that effect. Immensely helpful question that I use whenever I can. It gives a quick snapshot of the company. If I get an answer like, ‘Well, one of the users keeps getting an error message saying the transaction log is full and we don’t know why’ then I know the client has a fairly primitive understanding of the tech. That’s a red flag, and I know I will have a hard time getting them to adhere to some basic maintenance, configuration and performance standards.
Derek – WOW, I love that question. That’s really insightful. I can even see asking that to prospective clients, too. Nice job!
Dataveloper! Thanks Richie, I have a new title now. DBA/Developer wasn’t doing the trick.
You’re welcome. A while back I got tired of explaining what I do to folks. I do believe I came up with it for the FreeCon in 2015.
Who else besides the DBA team has access to the PRODUCTION database server and can log in to SQL Server? Do the other DBA’s use individual logins and not SA? Does anyone who is not a DBA have SA level access to production database? Does anyone have direct access to database with a login that can make changes to a system or database on a production server? What is the protocol for pushing out patches to the Database servers are these managed through the DBA team?
If I were to go back into production support……. I am not in the business of cleaning up after people who don’t know what they are doing. Just because a “Windows Administrator” can get to the server that does not mean that the “Windows Administrator” should be doing anything on the DB server without clearing it with the DBA team or any other person who is not a member of the DBA team. Leave the DBA work to the DBAs.. 🙂 Just my rant …
Yeah. I walked into a job where the company was “entrepreneurial”, which apparently meant that the devs had sysadmin rights on the prod servers and the windows admins would reboot stuff without telling the DBAs. Took me a week to strip the sysadmin rights but about 3 months to get the IT team used to consulting the DBA team on anything that they were planning to do to the SQL servers.
Stefanie – yeah, plus once folks have access to production, that’s not going to change quickly. Good call. In those kinds of environments, you’re always going to be cleaning up messes, and that’s not a lot of fun.
How many instances do you have?
What is your largest database and the application using it?
What is your most important application using a database?
Hourly vs. salaried and how many problems will probably have to wait. But you may get the above questions answered and that will give you some clues.
Ron – I’m constantly amazed how many people don’t know the answer to that first question. You ask a salesperson, “How many customers do you support?” and they’ve got a pretty good idea off the top of their head. Ask us admins, no idea how many servers we have.
Not something you could as directly, but how much turnover do they get? (Maybe ask how many years various people have been there.)
The place I work at routinely has people retiring at 20, 25,30 years. That says something about stability, and what the job is like.
Jay – true, and if you get the chance to get into a shop like that, you’re golden!
What’s your biggest pain point right now?
What are you planning on doing about it?
Also, having made the move to the DBA Team Leader role, I’d ask:
What’s you relationship with the rest of the business like?
Gavin – that leads to a good question: asking OTHER members of the business how THEIR relationships are with the DBA team, heh.
Hi,
in addition to most of the already mentioned questions I would ask how many changes / incidents they are implementing / resolving per day.
Gerald – I wonder how many folks track that number though? It seems kinda more like a help desk metric.
That depends on whither or not you receive work via a ticketing system. Combine service requests (requests for standard work) with break/fix trouble tickets and change requests, should give a pretty accurate representation that management can report on.
Incidentally, I would if they have a formal ticketing system as well as change control. IT quickly reduces the drive bys as well as documents requests.
I spent most of my career as an Oracle DBA and would also ask about the number of people in the on-call rotation, how often the on-call got paged each week, and what were the most common on-call issues. Some times the interviewer would look at me like “don’t you want this job?” As a team leader I pushed my team to take care of space issues during the day before going home. One of the issues I faced was that some of our team were contract DBA’s who got extra pay for handling the on-call issues in the middle of the night while the rest of us did not so they welcomed the extra work. Our team went from having a call per night down to a call per week but management didn’t recognize this as our effort to make things better but that we just had less work to do.
Around here they usually don’t have the contractors do oncall. Probably because…they have to pay them for those oncall hours!
True that the better job you do, it can be perceived that you just have less work. Perception is tough to manage.
After my current gig…
My first question is do you have policies regarding your SQL footprint (infrastructure, data, access etc) and are they followed. So many wasted hours and issues caused by either not having a policy or not enforcing existing policies.
Basically it leaves me holding the bag of crap all the time for trying to follow best practices.
Trav – yeah, and everything got inherited from The Last Guy.
My first question is always: “Are you comfortable with my commitment to complete transparency to all stakeholders regarding all my work efforts, technical questions, and decisions?”. I have found it the best way to set the ground rules for an environment I am professionally comfortable with among both peers and stakeholders.
Is this a new position for the company? Had a gig with a company that had four dbas in the three months before I came. 18 years of app development where dev team members were in charge of the dbs and wrote monster cursors to step through record processing with way too many items hard coded. But to make it scalable they used with nolocks on major tables. Three years later, we are stable and get an after hour call once every two or three months, but still have issues with dev members trying to dictate schema and some managers that have no idea of what we do.
“Who is in charge of what gets deployed to the database?” I’ve had almost 20 years of cleaning up after someone else’s bad design, bad code, just lack of experience (sometimes my own) or kludges put in place “temporarily” that are still a key part of the system 10 years later.
When was the most recent security threat ? Have you tested your DR plan in the last 6 months ?
DR plan? What DR plan? Huh?!
That was the answer I got. Then:
We don’t have one.
I didn’t go there. They weren’t managing licensing either and probably had like 50 SQL Servers, one for each developer, running their business. Time share company.
did you miss your chance? I thought there was a reward program in place for reporting corporate licensing violators?
Didn’t snitch. Thought about it. I think they were caught anyway. It was only a matter of time with that many developers running things and coming and going.
What’s your process for getting code changes into production? Are scripts sent to the DBA for review? IF they answer with “no dev’s deploy all code to prod” then I’d probably walk. A process oriented organisation can prevent a lot of headaches down the track.
Hi, on occasion I see a 404 server message when I arrive at your website. I thought you may wish to know, cheers
Thanks, but kinda tough to fix that without more details.
What’s your current policy and process for periodically restoring and verifying backups? An answer of “say what?” will tell you a lot about their DR readiness…
Being the only DBA, about the only time I’m not “on call” is when i’m in the middle of nowhere (cruise ship, for example). The only thing I don’t like: I’m salaried and we don’t really have a “comp time” policy, so those 16 hour days never really get compensated for and my family kinds gets irked at me (“Are you done working YET…???” is a popular question in my home). But, harvesting performance/session information and having a solid monitoring tool does simplify things. And, just doing things that are generally acceptable leading to a stable environment. You can’t always control what folks “throw” at SQL Server – that is where having a solid relationship with the developers comes into play IMO. They often reach out to me with performance related questions before things hit production. I don’t have much that happens in terms of unknowns, fortunately. Being a DBA for ~20 years has built up my experience (both good and not so good) – but I’m learn from my mistakes and eliminate them.
Piece of advice? Track those hours in a written log. If they cut you unexpectedly they may have to pay that. You should probably be hourly.
I know this is an old post, but I had it flagged and only just got around to reading it. The only thing I would add is to probe for information about additional positions and who fills them.
I was once hired as a DB Dev in an organization that had between 30-40 production instances supporting critical applications. Turns out they didn’t have any actual DBAs. Within a laughably short period of my onboarding, people are asking me to “see what’s going on” because the users are reporting things are slow.
They also hadn’t allocated any of my time for such troubleshooting or general maintenance activities. So I was de facto saddled with two roles.