My favorite engagement is when I’m brought in to help a good person who’s overwhelmed. The manager knows his DBA is smart, but the DBA is just flat out overworked. Too many developers are slinging too much code on too many servers, and there’s too many signs of smoke. The manager asks the DBA who he would pick if he could pick any SQL Server guy in the world to help out, and my email goes to Inbox Plus One.
In situations like this, I have to jump aboard a moving train. My biggest value is that I can tell the DBA, “Don’t worry, I’ll take the arrows at this next meeting. You stay back in the cube and keep working.” I have to respond to questions without knowing the slightest thing about the amount of work involved. I can’t ask for precise measurements of past performance – I have to shoot from the hip, and I have to do it fast.
The Situation: The Angry User
Ripley: “We’re having problems with the Hyperdine 120-A2’s. The hand motion is pretty quick, but we need them to go faster.”
Me: “How much faster?”
Ripley: “Well, I’m not sure. I’m not sure how to measure how fast their hands move. We started this test with a knife. Put your hand on the table and I’ll bring one in to show you.”
Me: “No thanks, that’s okay. Let’s keep it simple – do you want it to be 10% faster, 100% faster, or 1,000% faster?”
What That Line Does
I purposely use the 100% and 1,000% numbers instead of the easier-to-understand “twice as fast or ten times as fast” metrics because I want them to stop and think through the metrics. I want them to think about what those percentages mean, and more importantly, think about the vast differences in those numbers.
Those numbers don’t just refer to speed improvements.
They hint at work requirements, too.
See, the more they think about what it means to get something to be 100% faster or 1,000% faster, the more they’ll understand the fundamental differences between performance tuning and rearchitecting. They’ll realize that they’re not just calling me in to tweak a few little knobs – if they need something to go ten times as fast, I might make a very big suggestion on how they’re using technology. Admitting that you need something to be 1,000% faster means you’re willing to make some radical changes.
Sometimes, a judicious use of just the right index, query tweak, or configuration can improve performance by 1,000%. When I’m doing a very first round of performance tuning for a new client, that word “sometimes” might even be replaced with the word “often.” But before I go saying a simple new index will solve everything, I want the client to tell me how high I need to aim.
What Happens Next
I consider this line a success if the other person blinks and thinks, and they almost always do.
Ripley: “Uh…well…I guess it needs to be ten times faster.”
Me (nodding and making a note): “I see. That’s a pretty big jump. Okay, what else?”
Ripley: “You can do that?”
Me (with a grin): “I can do just about anything – that’s why they brought me in. I have no baggage, no politics, and no excuses. I’m just a hired gun.”
Ripley: “Ah, so the more work I want done, the more money you make!”
Me: “Exactly. So go ahead – let’s go through everything you’re frustrated with, and how much of a difference you need.”
By simply acknowledging the size of the jump and moving on, I’m setting up a silent contract between me and the user. They realize that performance costs money, and they start taking politics out of their decisions too. They build a mental shopping list and associate it with dollar signs, and you’d be surprised at how that tempers their demands for more performance.
16 Comments. Leave new
Nice way of defining scale. Thanks for the post.
The Consulting Lines series is great stuff.
Doesn’t 100% faster mean 0 execution time and 1000% faster mean traveling back in time? 😉
The laws of physics are invariant with respect to the arrow of time, however I’ve never seen a glass of milk unspill myself 😉
“Don’t worry, I’ll take the arrows at this next meeting. You stay back in the cube and keep working.”
I’ve been a huge fan and your overall message in this blog is spot on. Having said that I would have to disagree with the above statement. C’mon you knew a but was coming…
Like you I have played both sides of the proverbial fence, i.e. consultant and employee. As an employee and senior DBA there is no way I would not attend that meeting and as an employee I doubt you would skip it either. I would attend not to protect my turf and certainly not to submarine you, but more to sit back and watch you to do your thing. It would be important to catch the nuances of the meeting and afterwards our debrief becomes much more meaningful. To borrow a much overused phrase, it becomes “a teachable moment”.
Obviously if the house is on fire one is not looking for teachable moments but I would think the wrong message could be delivered by the senior DBA not attending that meeting. Obviously it depends on the situation and the people involved. Your blogs on this topic discusses an approach that teaches the client in subtle indirect ways so why bypass teaching the DBA?
I know there are many factors that go into this type of situation but that one line jumped out at me and just screamed. No worries, I’m still a huge fan.
Cheers
Rick – I understand where you’re coming from, but meetings aren’t always teachable moments. Sometimes they’re just about people wanting to throw rocks and arrows, and that’s when I love having someone on my side who can take those blows. This post was about bringing in a consultant to help someone who is already overwhelmed with work, and sitting through meetings isn’t going to help them move forward. Hope that helps clarify things. Thanks!
Hi Brent,
Usually managers don’t understand any metrics except for profit\loss indicators.
I bet they don’t even know that
“improve performance by 100%” = make it two times faster;
“improve performance by 1,000%” = make it eleven times faster.
Pavel – I’m sorry to hear that you haven’t had the luxury of working with good folks. Hopefully that will improve over the course of your career. Good luck in your journey!
Hi Brent,
Thank you for your response.
Please note that I love reading your blog and when I say ‘love’ I mean it.
It’s not only about SQL Server itself (even though I think SQL Server is coolest things that I happen to know!).
You inspire mediocre people like me to be better.
I really appreciate your time and effort so please regard what I wrote below as a feedback from your fan.
I was blessed to work with the wonderful people and all my bosses were brilliant (BTW one of them reminds me of you, because he was genius).
But it does not change the fact managers are usually not good at math. They are good at counting money.
When people want something to run ‘faster’ it is called ‘physics’. When they want quantative measures (metrics) it is called ‘math’. You wrote “I purposely use the 100% and 1,000% numbers instead of the easier-to-understand ‘twice as fast or ten times as fast’ metrics because I want them to stop and think through the metrics.”
If they stop and think they will find out that “improve performance by 1,000%” is essentially to make it eleven times faster, not ten.
Surely you can say something like “I am not good at math, that’s why I drop out of college years ago, but now I make $$, so STFU and teach your own children math not me”.
I have seen cases when people failed their targets simply because they didn’t know exactly what measures should be achieved and how they should be calculated.
Regards,
Pavel Nefyodov
Pavel – Hmm, you lost me with this particular line:
“But it does not change the fact managers are usually not good at math. They are good at counting money.”
Counting money is math. 😀
Hope this will help you find a way out. Consider this: steering a wheel is essential part of driving. My 11 years son can steer a wheel as god. But in all reality he is not good at driving.
Love the “Alien” reference by the way. I was in high school when that movie came out and it was a mindblower for sure. As for the topic I appreciate your take on how to make “them” think about the possible magnitude of work involved in what they’re asking for and how that relates to better performance. Also using your method of framing the problem I think I could shift the focus of the issue from the all too prevalent perception that the database is broken or in some way misconfigured and therefore is poorly performing to what changes could be made in your app/process to help me increase performance. I also like the idea that an overstretched DBA can reach out for help and that’s ok. It isn’t a comment on the DBA’s capabilities as much as it is a realization that it’s an important role in the organization, one that should be adequately supported.
Hey Brent,
I came here to tell you about a keen observation I came up with and posted on my blog. It just so happens that it’s wonderfully appropos to this article of yours…..
I finally got “That DBA Look”… 😉
http://combatdba.com/2011/03/18/that-dba-look/
Hahaha, nice…
I love when tech consultant bloggers write blogs that contain link rot. DBA != Webmaster .
Yep, I’m definitely not a webmaster. I wish I had time to maintain the thousands of links in my blog. I’d appreciate it if you could point out the broken links rather than leave snarky anonymous comments, but I understand – you had a tough day at work, and Mom’s tired of you living in her basement without paying rent. I don’t blame you. Sometimes leaving comments like this is the best way to feel better about yourself. Hang in there, man. It gets better.