Blog

Next week, Microsoft MVPs from around the world will be gathering in Redmond at the Microsoft campus. This will be my second MVP Summit, and I wanted to give you a peek into what happens at this private event.  Next week, you’ll probably be hearing people tweet about how excited (or angry) they are, and they can’t tell you exactly what they’re talking about.  It’s time for you to learn why.

What are Microsoft MVPs?

The Microsoft Most Valued Professional award recognizes…well, we’re not exactly sure what they recognize.  The public PR page says the award is given to the “best and brightest from technology communities around the world,” but the criteria for selection is shrouded in privacy.  The page on Becoming an MVP hints that there’s a panel of people who judge “the quality, quantity, and level of impact of the MVP nominee’s contributions” to the community.  (Notice that they’re judging the contributions, not the person – I’ve known a couple of total jerkwads in the MVP program.)  Every year, the same panel appears to re-judge you on the same criteria, and every year, some MVPs lose their standing due to inactivity in the community.

Your first official act as an MVP is to accept a non-disclosure agreement (NDA).  Microsoft is about to give you some insider access, but in exchange for that, you have to agree to keep your yapper shut.  Microsoft can’t have future product plans or weaknesses made public.  This is an interesting challenge – they’re taking the most publicly active people, and telling them to keep quiet about something.  I can see why every now and then, Microsoft has to revoke an MVP’s status due to NDA violations.

What Kinds of NDA Things Do MVPs Learn?

Honestly, not much.  I’ve heard from very long-standing MVPs that the program used to allow much more insider access to Microsoft employees and future product plans.  These days, that’s not really the case.  As David Woods wrote when he quit the MVP program, some of the NDA’d Microsoft material is basic marketing content for future product versions.  Microsoft wants to get us psyched up about upcoming features because we’re active in the community, and we might get the community excited too.

When there’s a major new version of something coming, Microsoft lets MVPs get varying levels of access to it.  All MVPs might get early access to a community preview build, and some specialist MVPs might get even earlier access to builds that aren’t ready for general MVP consumption.  It’s a blessing and a curse – we get access to software that isn’t ready for the public yet.  We can’t run it in public, and we can’t rely on it for production.  The more we use it, though, the more we can help Microsoft do their testing for free.  To non-geeks, that would sound like masochism, but if you truly love the product you’re working with, it’s great.  I’ll take a buggy future version of SQL Server any day.

Content authors (bloggers, writers, presenters) take these preview versions and build content ahead of time.  This is how book authors are able to get new versions of their books published relatively soon after the product goes live.  They’ve been playing with pre-release versions, writing, taking screenshots.  This is also how I was able to publish my Denali high-availability blog post and my Microsoft Atlanta analysis within seconds of Microsoft announcements of those tools at the PASS Summit, and I couldn’t provide that kind of SQL Server journalism without the MVP program.  I liken it to how Engadget posts in-depth phone reviews within minutes of a phone’s release.

So How Can You Write About the MVP Program?

Very carefully.

I’m trying not to tell you anything you can’t figure out on your own, and yet give you enough of a taste that you’ll see the benefits of being an MVP.  The NDA stuff is only a small piece – another benefit is the MVP email list.  I enjoy lurking to watch conversations between smart, passionate people.  (The MCM email list is even better.)  A valuable benefit is free software licensing for not just Microsoft tools, but all kinds of stuff from software vendors like VMware, TechSmith, and every SQL Server tool vendor.  Anything we want is basically free for our own personal use.  And then there’s the free MVP Summit…

What is the Microsoft MVP Summit?

Once a year, MVPs from around the world are invited to Redmond to hang out.  Microsoft picks up the hotel tab as long as you share a room with another MVP, but you’re responsible for the airfare.  MVPs who work for a typical company usually don’t have to take vacation time, because companies see the value in having their staff get inside information from Microsoft.  Us consultants aren’t so lucky – the time I spend in Redmond is unbilled, so I lose money for the week.

During the week, Microsoft puts on sessions.  It’s like a conference, but with all Microsoft speakers.  Most of the Microsoft speakers aren’t professional trainers – they’re software developers – so they don’t specialize in entertaining the audience with lolcats slides, nor do they know how to handle hecklers well.  They do their best to show us something pretty technical. When the audience doesn’t like the feature or senses too much marketing, the audience reacts – big time.

Remember who the audience is – they’re the most active community professionals.  These people are opinionated loudmouths, myself included.  When we don’t like a feature, we don’t just raise our hand – we throw things and yell, because we passionately love our favorite Microsoft product, and we don’t want to see it take a wrong turn.  In an audience full of SQL Server professionals, it’s very hard to build a consensus on anything.  Ask three DBAs a question, and you’ll get four different answers.  The MVP sessions are like that.  It’s good for entertainment, and the fun gets even better when the sessions stop and the drinks start.

My favorite part of being an MVP isn’t the sessions or the email list or the free software – it’s the chance to spend quality time with other SQL Server professionals and the cool people at Microsoft building things I like to use.

So Why Do People Quit the MVP Program?

Over the last year, I’ve read a couple of blog posts from people who quit the program, and I’ve had the same reaction every time: “This guy couldn’t let go and enjoy the program for what it is.”

When Microsoft recognizes you with the MVP award, they’re recognizing you for the things you’re already doing.  Don’t change yourself, your blog, your presentations, etc in order to keep the MVP award.  I’ve never heard someone from Microsoft say, “Listen, you were doing good before, but now that you’re an MVP, we need you to ____.”  Microsoft just pats you on the back, hands you a crappy glass trophy, and starts giving you cool stuff.  You’re an MVP – but you’re not Steve Ballmer’s boss.

If you want to really influence Microsoft product direction, go to work there – and even that isn’t enough.  You have to work your way into a position of influence.  Microsoft’s chock full of smart, opinionated people who all want to drive programs in their own favorite direction.

If you want to help the community, just do it.  Helping the community is rewarding in and of itself.  If the MVP program folded tomorrow, or if they just kicked me out, I wouldn’t change anything.  To me, that’s the mark of a true MVP – somebody who loves helping the community just because.

So you’re lazy and you’re company’s broke – what to do?  Here’s my favorite free downloads to help manage SQL Server:

Understand execution plans with SQL Sentry Plan Explorer – I dunno about you, but viewing execution plans in SQL Server Management Studio is a pain in my rear.  The scrolling sucks, the cost numbers are painful to read, and I can’t quickly get to the root of the problem.  SQL Sentry Plan Explorer opens execution plans and gives you much better visibility into the real problem.  You do have to save the execution plan in SSMS to an XML file, then open it with Plan Explorer, but that’s the only awkward part of the process.  Once the bad plan is open in Plan Explorer, just right-click on it and the fun starts.  When I’m tuning a server with slow IO, I can show the costs by IO only, not CPU – a big timesaver for me.

Slice and dice trace files with ClearTrace – ClearTrace is my go-to tool every single time I run a trace.  It’s easy to use and helps me get to the bottom of performance problems fast.  ClearTrace is a labor of love from Bill Graziano, the Executive Vice President of the Professional Association for SQL Server.  Bill’s a consultant who gives away software to the community and gives his time, too.  What a guy!

Get deeper insights into trace files with the Qure Workload Analyzer - If any free product ever needed a better web page, it’s this one.  When the Ami Levin of DBSophic showed me this tool at SQLbits, I nearly fell out of my chair.  It does a great job of comparing trace files to show whether your performance has gotten better or worse.

Analyze traces with Quest’s ProjectLucy.com – This one isn’t a download – it’s an upload.  Register for a free account, and you can upload your trace files to Project Lucy for analysis.  It’s very much a version 1.0 product, and it doesn’t provide a lot of in-depth analysis yet, but it’s quick and easy.  To get the best results, you’ll need to use their trace file template to capture specific events.

Improve SSMS with the SSMS Tools Pack – It’s got tons of features, but this one alone should sell you: it tracks all of the queries you run, and you can search through all of your past queries quickly and easily.  You’re not bothering to save all those queries you write, and you should, but you won’t – so grab this free tool instead.  Stay lazy, my friends!

Those are my top five – what free stuff have I missed that you love?

Companies don’t bring me in when things are going great and everything’s working flawlessly, oh no.  As a consultant, I’ve seen lot of ways that things can fail.  If a tree falls in a forest and there’s no one around to see it, the owner’s probably going to call a consultant afterward to find out what the noise sounded like.  I’ve seen a lot of fallen trees, and most of the time I get asked the same question: “How could this tree have fallen?  We did everything just the way we were supposed to do!”

While I can’t always figure out what the culprit was, I’m usually involved in helping to prop the tree back up.  In the process, a lot of people are going to be involved, because there’s usually a big business built around that tree.  The same people who were in charge of keeping the tree propped up – and failed – are now responsible to get it back up again, and I can’t rely on them knowing how to do their job.  In the heat of the moment, I can’t ask them to prove their capabilities to me, but in this week’s Consulting Lines series, I’ll show you how to ask one single question that will tell you everything you need to know about their experience and their grace under pressure.

The Situation: We’re Dead in the Water

Kaylee: “It won’t start because the catalyzer on the port compression coil blew.”

Me: “Oooo, that’s bad.  Do we have a spare port compression coil that we can pull a catalyzer from?”

Kaylee: “No, but I have an extra starboard one, and I might be able to wire it backwards to make it work.”

Me: “What happens if that doesn’t work?”

What That Line Does

I love this line because it’s open-ended and simple.  I’m not saying the action we’re about to take won’t work, and I’m not even touching the percentage probability of whether it’ll work.  I’m not saying I’ve got an alternate course of action, and I’m not saying the action is a good idea.  I’m simply taking the speaker’s word and going on to the next step in the plan.

This line works whether there’s a manager in the room or not.  If a manager is present, we’ve framed the discussion in a way that isn’t putting the worker at fault if the plan doesn’t work.  Just the opposite – we’re giving them room to vent about all the things that might break, thereby letting management make a more informed decision about whether to take this course of action.

Personally, I love hearing this line from project managers because it’s my chance to shine.  It’s my chance to show management that I’ve got the experience under pressure that the company wants.  I can rattle off lots of ways my plans might fail, and all the ways I’m going to try to work around them or detect them before they hit.

My disaster recovery plans don't always involve virgins on drink coasters.

What Happens Next: The Easy Way

If the other person is calm, collected, and experienced, they won’t take it personally.  They’ll know sometimes things fail when we expect them to succeed, even when we know what we’re doing, and they’ll just answer the question.

Kaylee: “Well, if I miswire it, there’s a chance it’ll blow the other compression coil too.  I should probably have that unplugged when I test my homemade one.  And if that fails, I – ”

Brent: “Hold on a second – let me capture this in writing so I can talk to the pointy-headed bosses while you get the real work done.  Ah, yes, this drink coaster will work.*  Alright, keep talking….”

* – Yes, I’ve actually documented an urgent outage on drink coasters.  The drink coaster pictured here was just one of several that we used during a recent outage of a 10TB data warehouse (who shall remain nameless).  We had several parallel tracks of action, and I documented what we’d do if each one of them failed.  I’m rather proud to notice that I even put the failure actions above the working actions!

What Happens Next: The Hard Way

If they’re freaked out and beyond their comfort level, they’ll start lashing out.  They’ll insist that things have to work even when they’re doing something they’ve never tried before.  No matter what their reaction is, if it’s not calm, cool, and collected, your first job – before tackling tech tasks – is to address the panic level.

The most common response line I hear is:

Kaylee: “But it has to work!”

Me: “No, we need it to work, but that doesn’t mean it will work.  Before we touch it, let’s spend 30 seconds doing a brain dump about all the ways it could fail.  I’ll write, you talk.”

If they can’t come up with at least one way that their plan could fail, they don’t understand the plan well enough to execute it.  Everything has a risk, and I only want people performing actions when they understand the risks they’re taking.  They might be too freaked out to see the risks, and simply calming them down with a distraction for 30 seconds can do the trick.  If they continue to ratchet up the pressure by insisting on immediate action without comprehending consequences, it’s beyond us geeks – time to get the people professionals involved, meaning the managers.  Managers are better at directing this kind of conversation than we are.

When you need SQL Server to handle more load, there’s two big-picture ways to do it:

• Scale up – use a more powerful SQL Server
• Scale out – use more SQL Servers

Quick disclaimer first – before you try either of these solutions, the easiest thing is to circumvent the whole problem – don’t ask SQL Server to handle more load.  Tuning your existing queries, schema, and server settings is almost always faster and cheaper than scaling up or scaling out.  For today’s discussion, though, I’m going to assume you’ve already tuned the bejeezus out of your existing SQL Server setup and you’re worried about running out of options.  Today, we’re going to focus on scaling up.

Scaling up means using a SQL Server with more oomph – more CPU power, more memory, or more storage throughput.

You have to fix the right bottleneck.  Don’t go writing checks to hardware vendors without knowing the real thing holding back your server.  SQL Server 2005 & newer track the bottlenecks for all queries, all the time, and they’re yours to discover by querying wait stats.  Glenn Berry’s diagnostic queries are my favorite tool for this job because they’re well-commented – you can run his queries step-by-step, read the comments, and understand what your SQL Server is telling you.

If Memory Is Your Bottleneck

If your bottleneck is memory, the good news is that memory’s easy to scale.  Scaling up memory is one of the easier scaling solutions because in most cases, your boss didn’t max out the memory when he originally bought the server. The max amount of memory is usually leaps and bounds more expensive than the second-highest amount of memory.  Take these memory upgrades for an HP DL785 server from Crucial:

2 + 2 = $5

16GB of memory is $1,300, and 8GB is $300.  To double your memory, you have to spend four times as much.  The good news is that memory prices drop over time, so by the time a server is a year old, it’s often more cost-effective to go back and double the memory.  Unfortunately, the older a server gets, the reverse happens – it’s harder to find warranted memory for a long-out-of-production server.

If you can fit your entire database in memory, plus space for sorts & joins, you can get away with a cheaper IO subsystem.  This is the approach StackOverflow takes with their database servers.  The challenge is that the moment your data stops fitting in memory, performance falls off a cliff.  Queries that used to fly through memory suddenly crawl because they have to fetch data off disk, which is exponentially slower than moving data around in memory.  You have to stay keenly aware of your memory metrics and be willing to write checks to support new database tables and indexes, adding memory as you need space for data.

Every server has a maximum amount of memory that it can handle.  Once you hit that max, you have to upgrade to a newer server, typically one with more memory slots.  Physically small servers (like 1U rack mount pizza boxes) have a limited amount of real estate devoted to memory slots.  Larger servers have more space for memory slots – but they also have more CPU sockets as well, and that’s where even more money comes in.

I stopped scaling up in high school.

If CPU Power Is Your Bottleneck

If your bottleneck is CPU power, brace yourself to write big checks.  Scaling up works fairly well from one CPU socket to the four-socket tier, but once you break past four CPU sockets, you’ll be spending significantly more on hardware. Getting an eight-CPU-socket server is usually much more than twice as expensive as a four-CPU-socket server.

CPU scaling challenges have these phases:

1. Replace the existing server with one having the same number of sockets, but more modern processors.  Thanks to Intel and AMD pumping out the jams, we can usually get dramatically faster CPUs every year with more cores.
2. Switch from a 2-socket server to a 4-socket server.  At this jump, we double our licensing costs, but we’re still using commodity off-the-rack hardware.
3. Switch from a 4-socket server to an 8-socket server.  Suddenly, the pricetags get serious.  8-socket servers are enterprise-only.  Thankfully, though, we can still use SQL Server Enterprise Edition.
4. Switch from an 8-socket server to a monster.  Now we’re in the price tag tier where server manufacturers say, “Please call for details.”  It gets worse, though – starting with SQL Server 2008 R2, database servers with more than eight CPU sockets require SQL Server Datacenter Edition at roughly $60,000 per socket.  That’s half a million dollars just for the SQL Server license.

The first few steps are easy, but we have to figure out a solution before we get to phase 4.  Thankfully, the multitudes of zeroes involved make it easier for managers to justify fixing bad code.  If we have to choose between buying a Datacenter Edition class server or hiring a couple more programmers to fix the code, sometimes it’s easier to do the latter.  (This is often where I come in – managers are out of options before they have to write a very, very large check, and a week of me performance tuning can help avoid a seven-figure purchase.)  Speaking of seven-figure purchases….

If Storage Is Your Bottleneck

When dealing with a server that only has local storage (not a Storage Area Network), scaling storage throughput is the opposite of scaling memory.  Your boss probably loaded the server up with storage throughput already.  He might have gotten cheap with the quantity (meaning, bought 146GB drives instead of 300GB drives) but in most cases, swapping in drives twice as big isn’t going to make the server twice as fast.  There are some niche cases where we can isolate a poorly designed database to its own drives, but that’s the exception rather than the rule.

If all of the server’s drive bays are filled up, there are three solutions to improve storage throughput:

1. Add storage that bypasses the drive bays altogether – like Fusion-IO or OCZ solid state drives on PCI Express cards
2. Swap out the storage for solid state drives – like Intel X-25 consumer drives as StackOverflow is doing
3. Reduce the amount of storage throughput required – by tuning indexes, queries, or procedural code

Notice how that last one keeps sneaking in?  Over and over during scaling exercises, the cheapest answer is manpower.  It’s so often safer to make code improvements rather than introduce new pieces of hardware or software into the mix.  As a DBA, whenever I did performance tuning, I made sure to point out to my management that my work directly saved us from buying more hardware.  Sometimes I would even avoid performance tuning altogether until the project managers and developers asked for my help.  They’d say, “Brent!  Our CPU use on the SQL Server is constantly high!  What should we do?”  That gave me the ability to answer with a choice: they could either spend $X upgrading to new hardware, or they could give me two weeks of uninterrupted time to tune the application.  After those kinds of interactions, managers understood that my time really was worth money, and it helped to free up my agenda to do what I love most – performance tuning.

Continue to Part 2: Scaling Out SQL Server

Almost ten years ago, some friends of ours (Erika & me) went through a rough divorce.  We’d always thought they were completely happy, and their divorce took us completely by surprise.  The more we learned about their breakup, the more curious we got about the success prospects for our own relationship.

Don't You Dare Get Married - Period!

We stumbled across the book Don’t You Dare Get Married Until You Read This: The Book of Questions for Couples.  We dutifully filled out our own answers in the margins of the book, and in the process, we learned a lot about each other.  I was shocked at the number of issues I’d never even considered, but thankfully, almost all of our answers meshed perfectly.  Almost eleven years on, we’re still happy together.

When Kim, Paul, and I started working with our lawyers to merge our businesses together into SQLskills, I was reminded so much of that book.  Lawyers asked us all kinds of questions and provoked some pretty surprising discussions.  We were all blown away by the kinds of things that can tear companies apart – well, all of us except the lawyers, who see this kind of thing all the time.

Along the way, we discovered things about ourselves, our businesses, and our goals – things we hadn’t even thought to ask ourselves.  We unearthed situations where we had major incompatibilities, and we figured out that we might not be the best 3-some to meet at the altar.  It’s better to figure this stuff out before you get in front of the priest.

I’m still going to be doing classes & consulting with Paul & Kim, but instead of joining SQLskills as an owner, I’ll be going on my own to gradually start my own brand.  We’ll be friends with benefits, though.  Paul says I can keep the sheep costume.

I like tech editing because it sharpens my skills.  I think of it as honing my BS detector.  When I read something that surprises me, I’m forced to go dig much deeper into SQL Server to find the real truth.  I can’t just call someone’s work wrong – I have to be able to prove it, and that teaches me things.

I learned something today, too:

When Documents Go Bad

Wow.  High score!

Hey you – put down the Entity Framework book for a second.  Pop quiz time.

What’s usually the slowest thing in your application?

Yep – the database.  Damn that thing.

Doesn’t it frustrate you that you have no idea what’s going on inside that black box? Execution plans look like bad subway maps, you have no idea what the bottleneck is, and you just want to know whether it’s really a hardware problem or a query problem.  The DBAs tell you it’s your code, but you’ve got a sneaking suspicion they’re not too sure what’s going on either.  You’ve asked for proof, and they don’t know what proof to give you.

I’ve got the fix.  In our 5-day Performance Tuning class in Dallas on March 14-18, we explain things like:

• How to find your bottleneck with simple queries – forget monitoring with Perfmon, because you don’t have the time to set long-term baselines and benchmarks.  You need to find the bottleneck for your query right now, and SQL Server will tell you using wait stats.
• How to analyze traces to find the bottleneck cause – after you figure out if CPU, disk, or memory is the problem, capture traces of the queries running on the server that might be causing the problem.  We’ll show you free tools to slice & dice the server activity and point to the handful of queries you need to tune.
• How to throw the SAN guy under the bus – storage can be the Achilles heel of SQL Server.  I’ll show you how SQL Server uses storage, how to speak your SAN guy’s language, and how to prove when there’s a storage issue that can’t be fixed in your code.
• How SQL Server does threading (and why you care) – even DBAs don’t understand the intricacies of how today’s multi-core servers, NUMA architectures, and SQL Server interact.  I’ll simplify it, show how to tell when it’s a problem, and explain why so much of that CXPACKET advice out on the internet is garbage.
• How to tune without the DBA’s help – either your DBA is too busy, or you don’t trust ‘em, or your company doesn’t have one.  You just want to make your app go faster, and we’ll give you the keys.

Yes, you do know enough to attend.

You don’t have to be a DBA to attend our training, and there’s no prerequisites other than being able to write a SELECT statement.  It sounds crazy, but there’s a really good reason: see, most DBAs don’t know this stuff either!  If you look at the week’s agenda and it all sounds like gibberish, don’t be scared – the week is all about teaching you each of those topics from start to finish.

And I have candy. Sweet, delicious bacon candy.

No, I’m not one of those snobby DBAs.

Well, yeah, but I’ll let you in on a dirty little secret – I used to be a developer too, and I got out of it because I wasn’t good enough.  I used to be a web coder back in the days of Classic ASP (ha! Classic, as if it’s like Classic Coke) and I realized I had to learn another language to keep up with the industry.  I toyed with Java and VB.NET, couldn’t decide which one would suck less, and said screw it – I’ll go become a DBA instead.  After all, ANSI SQL never changes – how hard can it be to stick with one language for life?  (I’m lazy.)

I’m unusual for a DBA – I actually like developers.  You guys are my customers.  You build all the cool stuff, and usually you hate the DBAs because they get in the way.  My co-presenters feel the same way – Paul Randal was a developer for Microsoft, and Kimberly Tripp writes and tunes T-SQL code.

No, I’m not intimidating.

I’m not a rock star.  I’m the most down-to-earth, laid-back guy I know.  Tim Ford and I founded SQLcruise because we wanted to have a completely relaxed, approachable training atmosphere where you could ask us absolutely anything and we’d respond honestly.  Your company probably won’t pay for you to go to SQLcruise, but they will pay for a more structured training event, and that’s where my land-based training comes in.

Just because it’s in a classroom environment doesn’t mean it’s all stiff.  I still have slides with race cars, guys stapling themselves in the eye, and buildings on fire.  You can ask me anything you want, and I won’t think you’re asking stupid questions or that your code sucks, because believe me, my code has sucked way worse.  VBscript, man.  Seriously.

Yes, I really know what I’m doing.

You know that StackOverflow thing where you post all your T-SQL questions?  Check the bottom of the team page – I’m not an employee of theirs, but I’m behind some of the database awesomeness.  (One of my favorite career highlights is on the StackOverflow podcast #46 at 1:03:25 when, while discussing experts, Joel and Jeff call me a genius.  HOOAH!)

Remember the Google Chrome ad with the potato gun?  That’s AllRecipes.com with a SQL Server back end, and yep, they’re one of my clients.

There’s more proof at our past customers page, our student testimonials, and the Best of PASS Summit 2010 listings, where Paul, Kim, and I had 5 of the top 10 rated sessions.  We’re really serious about this stuff.

Yes, this material is really relevant to development.

It doesn’t matter whether the database is fast – it only matters if your entire application is fast.  The Performance Tuning week is all about getting the database out of your way so your app can shine.  Whether you’re using SQL Server 2005 or 2008 or 2008 R2, whether you’re using local disk or SANs, whether you’ve been coding for two years or ten, you’re going to leave this course with more useful information that you can take back to your shop and start making your app go faster.

Here’s what to do next:

• Register for the March 14-18 class. It’s the second one in the dropdown.  The other ones are good too, but this post was specifically about the March 14-18 one that I’m teaching.  We’re running an early bird special right now for $2,995 (after February 18th, it’s $3,495.)  If you need help, here’s how to ask your boss for training money.
• Book your room at the Dallas Marriott Suites Market Center.  Call 1-214-905-0050 and ask for the SQLskills Immersion Event group rate for $129/night.
• Pack your bags and bring your laptop.  If you’ve got a tough SQL Server question that’s stumping you, bring remote access (VPN or LogMeIn) so we can troubleshoot it together.  I’ll be available before & after class each day to talk.

One of the cool things about working with an established product like SQL Server is that there’s a ton of knowledge out there.  The intertubes are chock full of advice about how to manage all kinds of problems.

Unfortunately, some of them are flat out wrong.

They may not have started out wrong – they might have been right when they were initially written, but over time, we’ve figured out better ways to work.  When I started out working with SQL Server, everybody watched the Perfmon counter for Average Disk Queue Length in order to determine whether their storage subsystems were overloaded.  These days, we know that’s not a good idea, but somebody who’s just getting started with SQL Server might stumble across a lot of that advice on sites that aren’t kept up to date.  I know Brad (the author who originally wrote that article) and he was completely right at the time, but that web site hasn’t been updated by its current owners.  Some pages never should have been published to begin with – on this Perfmon page, the author says if a server’s available MB of memory is <20-25% of its installed total, it doesn’t have enough memory.  That is simply not true, period.

I run across this situation in consulting all the time – someone’s banging their head against the wall because they’re following what they think is solid advice from a good source, and they’re not getting the results they want.  They truly want to do the right thing, and they’ve done at least a little research on the web to find the right solution.  It just so happens that they found an incorrect answer, but it’s not their fault for trying.  This week in my Consulting Lines series, I’m covering how to get past those misconceptions.

The Conversation

Me: “So I understand you’re having problems taking down the alien mother ship.”

David: “Yeah, I’ve written a virus on my Mac, and – ”

Me: “Wait, what?  I thought Macs couldn’t get viruses?”

David: “I know, that’s what makes it so genius!  They’ll never see it coming!  So anyway, I’m trying to plug my Mac into the alien mainframe, and it’s not working.  I can’t find a USB port.”

Me: “A what?”

David: “A USB port – you know, Universal Serial Bus.  Alien ships are still from this universe, so they should have a USB port, right?  That’s the best way to get files from one computer to another.”

Me: “Yes, that used to be a best practice, and it worked for the longest time.  However, drives have gotten bigger and we’ve wanted to transfer more data, we’ve had to change our tactics.  Here’s the latest trick we’re using to work around it – we’re using the cloud.”

What That Line Does

XKCD - Duty Calls

It’s almost easier to explain that line in terms of what it DOESN’T do.  It doesn’t dispute the speaker’s claim, because you don’t want to go down the rathole of arguing about whether or not the point was right at the time.  You could spend the entire day arguing about why USB has the word Universal as part of the name, but that’s beside the point.  You need to get to the right solution as fast as possible, and sometimes that just means ignoring something that’s wrong.

It doesn’t chastise the other guy for not keeping up with the latest tips and tricks.  It’s hard enough to just do our jobs, let alone keep up with blogs and training, and real life interferes.  It lets the other person save face.

It doesn’t divide you against the other guy.  I try to use the word “we” whenever I use this line, because I want to take the other guy along with me on the journey to fix this problem.

What Happens Next

This is one of my favorite lines because things almost always go the easy way.  It lets everybody in the room continue to believe that they were doing the right thing, and that it’s not their fault things weren’t working.

If the other guy wants to pursue the original point, agree that he’s right.  Agree vehemently that he’ll find lots of supporting evidence on the web from very credible people.  The key is to keep saying yes, and explaining why the new technique is even better – not that the old way is bad, but the new way is better.

In the middle of last year, I left Quest Software.  It was nothing against Quest – I loved what I was doing – but a better opportunity came up that I just couldn’t ignore.  The Microsoft Certified Master program was changing, and I wanted the chance to be one of the instructors.  I truly, passionately love teaching people about SQL Server, and I wanted to be part of the new generation of the MCM program.

The choice at the time was a no-brainer – I signed on with SQLskills as a consultant and trainer, and I went to work for  Paul Randal and Kimberly Tripp.  They’re phenomenal people who I really enjoy spending time with because we share so many things in common.  I cringe when I think about the number of champagne bottles that we’ve emptied in just six months – I’m fairly sure I’ve cost them more in champagne than I’ve made them in profits.  We work hard and we laugh hard.

My SQLskills Employee Badge Photo

However, just like last year, another opportunity has come along that I just can’t ignore.  I’ll explain more about my new gig here shortly, but in the meantime, I’d like to celebrate my SQLskills experience by listing the ten reasons I no longer work for Paul & Kim:

10. They refused to let me win a game of Blokus.

9. Paul kept calling my MacBook Air “your sticker carrier.”

8. Company meetings in the hot tub. ‘Nuff said.

7. They found out about my Richard Simmons costume.

6. Paul ran out of undocumented commands to show me.

5. We haven’t hired a junior employee, and I’m tired of being the water champagne boy.

4. Kimberly keeps cooking, and I keep gaining weight.

3. Erika doesn’t understand why I keep flying out to Seattle, and I can’t tell her about the free champagne.

2. I *still* don’t understand the source code for DBCC.

1. The sheep suit itches.  A lot.

Update: The Biggest Reason I No Longer Work for Paul & Kim

is that I work with them now.  I’m now part owner of SQLskills!  What a heck of a way to start the year, eh?  I’m still pinching myself, and not just because I enjoy doing that kind of thing.  Our titles are:

• Kim – Founder/Owner
• Paul – Managing Partner/Owner
• Brent – Managing Partner/Owner

This came about because marketing is hard.  I was chock full of ideas of things I wanted to bring to the table to help market SQLskills consulting and training, but those things take a lot of work and time.  As a consultant, I only got paid when I was doing client work, so I couldn’t justify putting unpaid time in to do marketing.  Paul and Kim kept hearing my crazy ideas about how to do marketing, and they finally got fed up.  They were willing to make me part owner of the company if I’d stop talking and start unleashing my l33t marketing skillz.

This is awesome – now I get to work more without getting paid for it!  Wait, on second thought…

No, really, we’re all wildly excited about what’s coming next.  Every year, life keeps throwing me things I wouldn’t have expected just a year or two ago, and I trace it all back to one thing: community involvement.  You have to be willing to put yourself out there, get involved, and take chances.  This feels like the biggest risk I’ve ever taken, and yet the safest bet in the world.

“I’m scared.  Hold me.”

I hear that all the time from people who want to go to a conference or a training class.  They’re terrified to walk into The Boss’s office because The Boss only says one word: “No.”  Here’s some easy ways to start the conversation, handle objections, and seal the deal.

The Opening Lines

“You know that project we’ve got coming up this year?” If you’re planning a project around virtualization, SAN storage, performance tuning, or version upgrade, you can get trained before the project starts.  Bosses understand that the training needs to happen first, and you’ve got a killer line here: “If I get trained first, I bet I can save us from spending too much on the wrong storage or doing something that makes the project fail.  An ounce of prevention is worth a pound of cure.”

“I’m tired of relying on consultants who don’t know what they’re doing.” If your company has been burned lately by consultants who are fumbling their way through your systems, now’s your chance to offer yourself up as an alternative.  With seriously advanced training, you’ll be able to do seriously advanced tasks.  All of the training I do is focused on giving you things you can take back to the office and make your systems faster and more reliable.  At the very least, you’ll be able to call the consultants’ bluff when they’re faking it.  Your boss is probably sick of spending money on people they don’t trust, but they DO trust you.  Investing in you is a safer bet than spending more on strangers.

“I know we can’t get raises this year, but…” Economic realities mean some companies are still in a wage freeze.  If you’re good, your boss probably wants to keep you around.  He can’t go offering training to everyone, but if you’re one of The Good Guys and you show initiative to ask, he’ll bend the rules for you.  Don’t ask if you’ve got a training budget, because that’s likely to make him say no, and you want to get to yes.  Instead, say something like, “I know we can’t get raises this year, but you know what’d make me just as happy?  Getting training.”

How to Handle Objections Gracefully

Here’s the most common objections I’ve heard from managers:

“I can’t send you because conferences are boondoggles.” Managers have probably been to conferences themselves, and they know that conferences involve just as much partying as they do learning – if not more.  Even worse, THEIR managers believe conferences are boondoggles too, because once you get to the upper management level, “conferences” involve a whole lot of golf and hanging out with buddies.  Your manager may not want to be seen as sending their own employees to conferences.  It’s tough to fight this image (especially when the web is full of conference party pictures) – so take a different tack.  The easy response is, “That’s fair – how about I go to training instead?  SQLskills offers 5-day training classes, no partying involved.”

“Why can’t you just go to training locally?” Training companies like New Horizons have offices in most major cities and they periodically offer SQL Server classes.  Yes, I’m actually linking to one of our competitors – because people who have attended those types of classes will usually tell you there’s no competition at all.  One easy response is, “I want to learn from someone who’s actually doing this work, not somebody who’s reading from a script.  Brent Ozar PLF’s people win awards from their attendees, like Best of PASS Summit and Top Ten of TechEd.”

“I can’t have you unavailable for a week.” I remember what it’s like to be not just the DBA, but also the developer, the sysadmin, and the coffeemaker.  My favorite response to this is, “I’ll take my laptop and I can pair it with my phone to get internet access.  If there’s an emergency, I’ll leave the class.  I’d rather get some training than no training, and I’ll log in every night to do as much preventative maintenance as I can after class.”

“What’s the ROI?” Managers love Return On Investment – a calculation that shows the money they spend on something pays for itself.  Yes, training really does have an ROI, and here’s how to calculate it.  Tell your manager, “We’ll spend about $5k on the training including hotel and airfare, and here’s what we get in return:”

• “We’ll avoid spending money on new hardware.”  My clients constantly tell me that after learning my performance tuning techniques, SAN troubleshooting tips, and virtualization configuration tricks, they’re able to get by with a lot less hardware.
• “We’ll spend less on consultants.”  We focus on training you how to do the same things we do day in and day out.  You’ll be empowered to do things that you used to rely on outsiders for.
• “(Custom Quote Here)” – If you’re trying to figure out whether training is right for you, ask us.  Drop us an email with a few sentences about your job, your upcoming projects, and your experience level, and he’ll tell you which Immersions class is the right fit for you – plus why it’s a good fit.

How to Seal the Deal

“I’ll bring back the material I learned and re-present the highlights to the team.” At training events, presenters don’t always hand out their PowerPoint files, but they do give handouts with the slide content printed out.  This material is usually copyrighted, and you should never re-present this material to the public, but re-presenting it to your own team is a way to bring value back to the rest of the staff who didn’t get to attend.  You can filter the entire week’s worth of training down to just an hour or two of material that is especially relevant to your own team, helping them fix problems that are happening in your environment right now.

“I can ask questions about our environment and get the very best free help.” I’ve gotta be honest – hiring a good consultant for a few hours is really expensive.  During training sessions, I’ve fielded questions from audience members about specific issues they’re running into in their environment.  After the training, I’ve even sat with students as they VPN’d into their company and pointed out problematic servers, thereby troubleshooting with them right then and there.  Not only are we able to give good answers, but we also ask the entire class.  Our audience has decades of SQL Server experience, and sometimes the best answer is sitting next to you.

“There’s a $500 early bird discount if we register now.” Saving money quickly helps motivate managers.

I still remember the first time I got up the courage to ask my boss for training money.  I was blown away that he said yes, and I kicked myself for not asking sooner.  Don’t sit around waiting for someone else to offer you training money – it just won’t happen.  Take control of your career and use these techniques to pop the big question.  If your boss still says no, post the reason why here in the comments, and as a group, we’ll figure out how to help you get to yes!

Update: it worked! @ShadyRudy read this post and was inspired to ask his boss.  I’ll be meeting him at the Performance Tuning Immersion event in March!

El Jefe Said Yes