It seems a like a stretch. After all, plumbers get called when sinks are running backwards and when toilets and drains are clogged. But, really, the users and developers do the same thing with DBAs that you do with a plumber – when something is broken, we get the call.

Unclog Your Database Fast

The main thing that plumbers do is fix immediate problems. When this is your job, you have to focus on diagnosing the problem, understanding the ramifications of different solutions, and coming up with the right solution. In databases, as in plumbing, we’re almost never called on to fix problems when we can do this at our leisure – there’s a mess that needs to be cleaned up quickly.

The first thing that we need is a set of tools. Don’t go scrolling down looking for scripts, you won’t find any. The first tool that you need is the ability to listen.

You might recall when Brent had to call a plumber to fix his garbage disposal. Based on Brent’s description, the plumber had a good idea of the problem and knew that there was a simple solution: just give the disposal a good crank. Afterwards, the plumber gave Brent advice on how to keep the problem from happening again. Brent happily wrote a check and the plumber made $1000 an hour for those 10 minutes. The plumber was able to use his listening skills to quickly diagnose a problem, pick the right tool, solve the problem, and leave the customer with tools to prevent the problem in the future. As a plus, by listening, he knew that he didn’t have to bring heavy tools upstairs, just a tiny wrenchette.

As a DBA, you need a similar set of tools – when there’s a problem you need to be able to understand what the problem is, come up with a solution, implement the solution, and watch to make sure things are working correctly. But you can’t just focus on the immediate problem; take some time after you’ve solved the immediate problem to focus on the system as a whole.

The Bigger Picture

While you’re busy unclogging an index, have you ever found yourself saying “You know, if this database design was different, this wouldn’t be an issue”? You’re not alone – your plumber wants your kids to stop flushing toys down the toilet. There’s always something else that needs to be fixed; whether it’s another quick fix or a larger architectural change.

Plumbers also have a different job: sometimes a plumber gets handed a set of plans for a building and is told to make it work. It doesn’t matter that the right parts may not be on hand, there’s a deadline. Walls need to be put in place and if the plumber isn’t done on time, the entire schedule is going to slip. In these situations, the plumber has to deal with the constraints of reality. There are some things that you can do without all of the parts in place – you can put plumbing in place for a washer and dryer without the appliances being on site. But you can’t put the washer and dryer in place and expect them to work without any plumbing.

When you’re designing a new system from the ground up, you don’t have the luxury of taking all the time in the world to build things The Right Way®. You focus on filling the customer’s specs with the equipment that you have on hand. Don’t have particular type of pipe fitting? You’ll either have to wait or devise a solution. This is how things end up different in the real world from the way they were devised on paper – implementation details are frequently more than just minor details. Those database architects who really screwed things up? They weren’t incompetent, they just had a deadline.

You call your plumber to fix clogs that are happening right now. Likewise, management is usually bringing you in because there’s a clogged database. You’re there to fix a problem (unclog the database) and get back to your regularly scheduled day. It’s tempting to bring up all of the design flaws but, unless you’re asked, you probably shouldn’t bring it up. Write them down, keep a list of database design issues in your wish list. You’ll feel better about it.

While you’re writing down all the big problems that you noticed in the database, make sure you write down how this problem could have been prevented. If it’s conducting stress tests to find missing indexes, talk to the development team about the tools they can use. If the developers need some query tuning help, get in there and help them with their query writing skills.

Fixing the Problem with Great Service

Users, management, and developers don’t want to know what they’re doing wrong. They want to know how it can be made right. Sure, there are problems; nobody is happy about it, but the users know that everything can’t be made perfect.

When you help the users make things better, show them the problem. Take the time to explain why it’s a problem – “the dimensions of this toy don’t mesh with the dimensions of your plumbing, it’s only natural that it get stuck”. Then explain how the problem can be prevented going forward.

Great plumbers don’t just explain how to solve a problem, they deliver great customer service while they’re doing it. A great plumber doesn’t make you feel like an idiot because your kids decide to have a toy soldier flushing contest. A great plumber fixes the problem, tells you it happens to everyone, and makes a joke with your kids about how flushing toys is how he became a plumber.

What About the Long Term Problems?

Every time you look at a system, write down all of the long term fixes that should be put in place. Keep a list of them around and build out a plan of attack for the best way to fix it with minimal disruption. If you do a good job of solving the immediate problems, there’s good chance that someone is going to bring you back to fix problems in the future.

Don’t think you need to be silent, though. Tell the users about things you can easily fix right now. If it isn’t much effort to add an extra index during an outage window, suggest the index and go on your merry way. The last time I had a plumber over, he suggested that I throw ice in the disposal once a week in order to keep the blades sharp and to free up any gunk that was floating around in there. Did he try to sell me another disposal? No. Is that plumber going to get my business the next time there’s a problem? You bet.

You’re a plumber. Be happy that you’re unclogging a drain today. Solve the problem you’re brought in to solve and there’s every chance you’ll get to implement the plumbing the next time around.