Manager: “It’s come to my attention that the application performance isn’t as good as it should be. Users are complaining that it takes too long. Is it fair to say that every query sent to the database should finish in under ten seconds?”
Developers: “Yeah.”
Database administrators: “Yeah.”
Manager: “Okay, we’re in agreement. Going forward, all queries sent to the database need to finish in under ten seconds. Meeting adjourned. Great job guys.”
…
Is everyone really in agreement?
The developers think it’s the DBAs’ fault because the server isn’t fast enough. The DBAs think it’s the developer’s fault because the queries are so poorly written. The managers think everybody’s just flat out incompetent.
Groups of people can agree on something on the surface, but ask a few questions, and all hell breaks loose.
This seems to happen a lot in database administration due to the kinds of knowledge DBAs have. People bring us lots of ideas, but they’re not really clear on what needs to be done in order to accomplish that goal – or who’s the right person to actually do the legwork. As a result, sometimes we have to break some hearts.
I struggle with this, because I’m the kind of guy who will turn right around and give a flip answer like, “John, I agree, that’s a great idea, and I can’t wait to see how it turns out for you.”
Better answers are probably along the lines of:
- “I agree – that’s a great idea. Is there anything I can do to help you get it done?” People freeze at that one because they don’t feel comfortable saying, “Yeah, you could do it for me.” It at least gets them to stop and think.
- “That sounds great! Could I help you by testing your work or finding beta testers?” Again, sets the expectation that you’ll get involved when their work is over.
- “I’ve been thinking about that same thing for quite a while, and I haven’t been able to find the time to do it. Doggone day job of mine! Who could we track down to do it?”
And still, as I write these, I know they’re all too sarcastic. <sigh> This is one professional skill I still gotta work on.
7 Comments. Leave new
Hey Brent,
Don't forget the other answer that often come out of someone's mouth, "It can't be done." or "It will never happen" or if he's been around a while and is good and chummy with the boss, "IN YOUR DREAMS!". This one person HAD to open their mouth! Everyone else rolls their eyes. Why can't Bill just appease "the man"?! Bill, really, just smile and nod your head and we'll all be back to work, or online poker, or browsing Dice.com.
Of course it can be done – policies are put in place all the time – and that's all your scenario depicted, right? Too often a manger decides on a policy and everyone who was pulled into a conference room simply agrees, "Sure, that sounds great. Can I get back to <see above> now?" The manager is proposing an internal project, but often times they don't see it that way. They're simply putting a benchmark in place, right?
(Continued….)
(…From previous)
Here's what I like to do in situations like this. Make it a project. You don't need to volunteer to PM the thing – and it does need a PM – but rather just take the quick "policy meeting" meeting and turn it into a project kick-off. Start talking about who has what time available, gathering peoples schedules for the next 2 months (as a start), discussing what it means to current resources, and what the expected gains will mean to the company's bottom line. Sometimes the manager will take a step back to consider the policy further, or even withdraw the proposal – other times he'll put you in chage (yikes). But maybe that manager will see you as the one person in the room who took the initiative to move things in the right direction for the long-term benefit of the company. Yeah, IN YOUR DREAMS!
Great post, Brent.
Howsabout (in all seriousness, unfortunately):
"That's a really great idea. How about we work together to come up with a proper definition of which queries fall into this ten second rule. We'll then create a listing of those queries, evaluate how they are currently performing, and set a timeline for optimizing the existing code. Any future additions to the program will need to be run through this benchmark process, and will need to be able to execute in under (6,7…..) seconds under a heavy load before they get put into production. A production server will naturally have periods of heavy load and lighter load, so the 3-4 second buffer should allow for those fluctuations without violating OUR SLA."
Something like that might work? It "shares" the responsibility rather than assigning it, so you're less likely to get backlash.
Sometimes sarcasm is indeed necessary.
In all honesty, I made this request myself one time. To get started I created an automated SQL Profiler trace system that ran on an old desktop against all the servers in the building – at an IT shop, not a production environment. The results were aggregated and loaded into a database where reports were generated at the end of the day and e-mailed to the database owners. It reported "Worst Offender" queries ordered by CPU suage, IO Reads and Writes, and of course, duration. It didn't fix anything, but it let developers know where potential enhancements could be made.
From the replies so far, it sounds like most DBA’s are working with a development team that is in house. The politics can become even more convoluted when software development is outsourced. The responsibility for performance tends to be trickier to pin down and collaborate on.
I've seen this happen in the real world scenario quite a few times. I'm personally on the developer side of things and everytime I see it happen it was a terribly written query by specifically one of my co-workers. No matter how many times I tried to explain to them how important performance was they just responded with "We can just throw more hardware at the problem."
It got to the point where we were using a 16-core, 32GB of RAM DB server even though we were using SQL standard which can only use 4 of those cores.
*sigh*