DBAs often complain about SAN Administrators. “You can’t trust them.” I’ve seen established DBAs write publicly that they resort to lying about how SQL Server works to get what they want from a SAN Administrator.
That’s pretty toxic. Other DBAs start to think it’s normal to have a terrible relationship with your SAN Administrator.
But it’s not normal. And if there’s a terrible relationship between DBAs and the SAN team, that’s the DBA team’s fault, too.
The first SAN team I ever worked with
I’m pre-disposed to like SAN Administrators because I’ve worked with great people. When I started out with SQL Server at a small dot com, we had a two person SAN team and many terabytes of data across both SQL and NOSQL solutions. Our software team and our data grew insanely quickly.
The SAN team had more fun than almost anyone in the building. They had to work super hard. They got paged a lot. They were sometimes grumpy on the phone at 3 am, just like the DBAs. But they were funny and smart and the kind of people who could magically turn a crappy situation into a good time.
They didn’t always want to make everything RAID 10, and they didn’t always automatically believe the problem was the SAN when I said, “the SQL Server’s slow.” But they worked with me every time when I had a problem, and we always found a solution.
Over time, I learned ways to show them real metrics when I really needed more storage speed rather than just saying, “it’s slow.”
Most SAN Administrators I Work With Today
I still work with SAN Administrators frequently. They’re usually helpful – in fact, they’re often happy that someone would like to hear how the SAN is configured and why it’s set up that way.
Most SAN Admins I meet work alone or in small groups. They’re super busy, and sometimes mistakes get made (just like DBAs). But also like DBAs, I’ve found them to be pretty happy when there’s a real reason that justifies investing in better hardware. They’re able to admit when something’s not right, fix it, and move on.
Remember, The SAN Admin had just as much training as you did
That’s right, they probably didn’t get any training either. The storage world changes fast, and they have to try to keep up.
Yes, they get taken out to nice dinners by the SAN vendor, and you don’t. But think about how your job looks to the people over at the helpdesk. Ever gotten a bottle of scotch from the developers as a reward for saving the day? Ever had flextime and rolled into the office late? It’s not just the SAN admins who have some perks.
Your SAN Admin isn’t out to get you. They just have a lot of customers.
Your mission as a DBA is to make your databases perform as well as the business needs them, and protect the RPO and RTO of your customers. The SAN Administrator’s goal is to provide enough storage capacity and performance as the business needs. They’ve got a lot of customers– the performance of every one of your databases isn’t at the top of their list. When the database is slow, it’s hard for them to know that the issue is the storage.
I’m not saying that there aren’t bad SAN Admins out there. There certainly are.
But don’t be the person who misrepresents things and thinks someone’s out to get them. Aspire to be more like my old SAN team: the kind of person who can turn a crappy situation into a good time. That’s usually a lot more effective.
Brent says: wait a minute – I never got a bottle of Scotch from my developers. I mean, I got hit over the head with a bottle once, but there wasn’t any alcohol left in it.
9 Comments. Leave new
Kendra, this advice is so good on many levels.
You are absolutely correct that BS-ing any of one’s teammates is toxic. It should be considered anathema. Transparency and truthfulness builds trust and camaraderie.
Our SAN admin at my employer is a good guy and he listens carefully when I explain everything. I show metrics in order to give him perspective on what is going on. He appreciates it.
We had a production database on an ERP system go live and then grow fourteen times faster than projected. That put us both on edge, fighting the urge to blame the other for an emergent problem was was neither of our fault or making. However, we worked through it by being truthful and kind and cooperative.
We worked together on solutions. I made database changes, he made storage changes, and we stabilized nicely. Now if I have an issue, the SAN admin is not in the least defensive and vice versa. The end result is an infrastructure team about which management has no worries, and in which management has great trust.
JT.
Excellent article Kendra,
I whole heartedly agree that as DBAs we need to work with the SAN admins. I have had the opportunity of building that trust between groups when I have changed companys in the past. It is so nice to be able to get to a point where instead of one team or the other saying “problem isn’t my side” to “let’s sit down and see what is going on”.
I think the biggest hurdle I always have to over come is the terminology used between the two teams. I have also found that once we all agree that the external customers (non-IT folk) don’t care if it is SAN, Database or Network issues, their complaints are it is an IT issue that is slowing them down, we seem to be able to focus on solving the problem.
Great post ! It is always better to have friends than enemies.
Just curious, this post has anything to do on the post of Joe Chang in SQLblog ?
Hahaha, no, our posts are usually written a month or two in advance. Kendra penned this one before Joe’s post went live.
I was actually inspired to write this post by some students in one of our classes who are both DBAs *and* the SAN Admins. They had a really fun, can-do attitude and they knew about the problems on both sides of the fence, and it made me start thinking of all the times over the years I’ve read and heard negative statements from DBAs who just don’t know what that other side is like. Made me think about my own experience and why I have such a positive view of SAN Admins.
Being both SAN admin and DBA (along with other jobs) means I get to yell at myself.
I know what you mean Dave, you start to get a bit schizophrenic at 3 in the morning…
@Kendra – Great post
I hate to disagree, but then I’m an old-fashioned, home-grown DBA who knows the difference between good and evil, and System Administrators are pure evil. I don’t go for the “Can’t we all just get along?” BS that managers feed me. I have tried to get a long with sysadmins about SAN requests, and my knees can’t take it. I have to grovel and flatter and beg just to get an extra vCPU. Forget it.
Now I just do my job, and if there’s slow SAN issues, or if 4GB is not enough memory to run a SQL Server, I just point the customer to the IT department.
Haha, if you’re an old-fashioned, home-grown DBA, you don’t hate to disagree 😉
A couple of things about your comment jumped out at me. You point towards the IT department as if you’re fully outside it and not *part* of that team. Your comment sounds like you’re really disempowered and alienated, and that’s a really rough spot to be in, for you and your company.