SQL MCM: The Exams

If the DBA answer is always, “It depends,” how can Microsoft test DBAs?

Six of the SQL MCM R8 guys

Every week of the MCM program, attendees take a multiple-choice test. Afterwards, we gathered in the hallways and vigorously debated our answers. We agreed about most of the questions, but we had loud discussions about the pros and cons of a dozen of the test questions. The test really forced us to evaluate a lot of options. How many answers are right? Sometimes they tell you to pick three – and sometimes they don’t. Sometimes it’s up to you to figure out how many answers are right. I adore tests like that because they don’t give you any help, and they make you prove things for yourself. You’d better be right, too, because there’s no partial credit for halfway right answers. Time to be a master, as the posters in the classroom say.

The questions don’t involve regurgitating memorized syntax, either. Sure, some of ‘em can be figured out by the covered material, but most of them required you to put one and one together. It wasn’t just about knowing what the book says – it was knowing what the book DOESN’T say, and justifying your rationale. Every question has a comment area, and during the tests, keyboards were clacking away. I put comments on at least a quarter of my answers, saying things like, “I picked B, but only because you didn’t say you needed HA. You should have HA for this, which means you need D, not B.” I went way, way out of the way to back up as many of the answers as possible. The tests are graded by real, live human beings. More on that in my next post.

I passed all 3 exams on my first try, and I’m really proud of that. Before I got to the Microsoft campus, I was worried about the exams, not the final six-hour hands-on lab. I was so afraid of failing the tests that I studied my brains out every night for those rather than spending time with lab scripts. Attendees can bring their scripts and documents to the lab, plus use the web, so how hard could it be?

The Final Lab

Simon Sabin (Blog – @Simon_Sabin) noticed all the studying I did during the last 3 weeks and asked me if it was possible to cram your way through the MCM. First, there’s an interview process before you even get accepted to the MCM, and you can’t cram for that. But for the sake of argument, let’s just say you fake your way through the interview, and you cram your way through the exams – there is absolutely no way to cram your way through that lab.

The lab was the best simulation I’ve ever seen of what it’s like in real life. The CIO’s standing behind you, asking you why the system’s not up yet, and why it’s not as fast as he wants it to be. The MCM lab is like the last question on Who Wants to Be a Millionaire when you’ve already used up all your lifelines. You don’t get the 50/50 option, you don’t get to phone a friend, and Regis wants to see how much you really know. The final lab on Saturday was the toughest IT experience of my life, and I’ve been through some serious nastiness. Obviously I can’t say specifics about what the lab involved, but I’ll do my best to walk the line.

As a DBA, sometimes people want me to do things that I’m not comfortable with. Sometimes they want to use SQL Server in ways that don’t make sense, like looping through records individually and doing CPU-intensive processing on them. Sometimes they want to use features I’ve never touched before, and I pad more time into my project timelines in order to get comfy with those features. Sometimes they want to combine features in ways that I’ve never thought of. I try to deal with these requests in a way that keeps the project moving forwards on time with the least amount of risk.

In the MCM lab, you don’t get the luxury of finding a common ground between what the client needs and what you’re personally capable of delivering. The client needs something, and they don’t care how SQL Server works. They don’t care if you’re comfortable. You’d better strap in and get the job done, cowboy.

At 11:30 AM, I got a several-page business requirements document that was more thought-out than what I usually get from clients, but in a very similar style. It was as if someone had parachuted in before me, interviewed the client, done a lot of the requirements gathering, and handed it off to me. It wasn’t perfect – but it was still better than what I usually get from managers. I took about five minutes to read it, formulate a game plan, and then got started working. I had a list of about half a dozen things I needed to accomplish in the client’s lab environment. I had to build some things, I had to adapt some things that were already in place, and I had to fix some things.

My heart sank as I realized what I’d have to do by 5:30 PM. I understood the business needs behind every single task I needed to do. I have big philosophical objections to doing some things in SQL Server, but not this stuff. It was all a match made in heaven – but I just didn’t know how to do some of it. They had completely legit business needs, but I couldn’t directly match some of those needs to anything in the SQL Server toolbox. It wasn’t like I could open SSMS, right-click on a few things, tweak some obscure parameters, and deploy a solution.

Or could I? As I built my list of tasks, I was absolutely positive that I was missing some obvious features or some simple DMVs. I kept thinking, “I’m gonna be so pissed afterwards. I’m going to be the only guy here who didn’t know about the Magic Beans feature that automagically does this stuff for me. I’m gonna be the only moron who rolled his own solutions out of duct tape, lint, and Cheesy Poofs.” We had access to the web, so I spent maybe two minutes searching for solutions, but came up empty. I figured out immediately that the MCM team has put a heroic amount of work building a test that simulates real client work, yet requires answers that you can’t find in Google. I will never breathe a word of the final lab contents because I think it does such a phenomenal job of gauging truly master-level expertise.

Getting Things Done

Whether you’re doing a lab or a real-life project, you have to design for the time you have. Artists ship. But should you:

• A. Plan to be 100% done with 50% of the tasks
• B. Plan to be 50% done with 100% of the tasks
• C. Plan to be 100% done with 100% of the tasks

When someone else sets your deadline and your tasks with non-negotiable limitations, you need to decide whether you want to shoot for A or B. In the case of the MCM, I took approach B. I wanted to show that I was capable of doing everything the client needed, and that I knew what best practices were, but I just didn’t have the time to do it according to best practices.

For example, when I’m importing data from a file, ideally I want to know what the real field definitions are. Sure, today the longest value in the LastName field might be 40 characters, but what if the next batch has someone with a 60-character last name? I don’t want my processes to break. However, finding this out can take time, and the cost of being technically perfect is high. In situations like that, I might make LastName a varchar(100) and call it a day. Anytime I made a compromise in the interest of time, I added it to my running task list in Notepad. If I had the choice between doing something perfectly or doing it MacGuyver style, I did it MacGuyver, then made myself a note to come back and clean it up.

Even the seemingly easy tasks were hard. This was no point-and-click GUI-fest, and canned scripts didn’t help much. I brought dozens, maybe hundreds of scripts that I’ve used in the past. I think I only used two, and they only saved me maybe fifteen minutes, max. The rest of the time, I was banging out keystrokes, testing my work in TempDB first, and hoping like hell that it all worked when I tried it with the client’s stuff.

I finished the MacGuyver round after 4 hours, and then circled back to clean up some of my work. I walked out with about 30 minutes left because I’d gotten to the point where I didn’t want to touch anything else lest it break. My solutions worked enough that I was comfortable defending them. I could have built more, but in doing so, I stood a pretty high chance of breaking something. I turned in my solutions along with my laundry list of things I’d love to do if the client would be willing to give me a week.

After The Lab

I walked out and called Erika. I was glad I got voicemail, because I was on the verge of crying. I could barely keep control of my voice when I left her a message. I was exhausted, I was proud of what I’d accomplished, and I felt like I couldn’t possibly have done any better given the skills I’ve gathered over the years. I was still sure I was missing something obvious, because there was no way candidates could be asked to this much work in 6 hours.

As the other attendees came out, the experience was nothing like the post-written-exam discussions we’d had. Rather than debate the merits of our solutions, we all tried to rationalize each others’ solutions. We didn’t want to promote why our idea was right – we wanted to figure out what the other people had done, and learn from their approach. There were no easy buttons, no quick solutions. As we spoke, we were horrified to discover that we’d overlooked or overthought various pieces. Several of us (myself included) built out whole wings of stuff that we thought were a good idea, but weren’t technically required by the client. I was awed by the work of my classmates, who came up with really brilliant ideas and features.

Here’s the part that really got me: everybody I talked to pulled off Mission Impossible. We actually did most of the stuff we needed to do in the time allotted. I’m pretty sure one guy even pulled off 100% of the tasks to 100% quality, too, and I stand in awe of his skills. As the MCM blog announces new Masters, I’ll be the first person to sing their praises. Forget the certification – there were candidates who I would hire in a heartbeat because they’re so skilled.

The non-disclosure agreement really sucks because I can’t tell you how incredibly difficult that lab was, yet so perfectly in tune with the DBA experience. If you’re a production DBA or a consultant who doesn’t freak out when someone comes running into your office with a tough challenge, you can pass this. If you enjoy tracking down difficult problems and reverse engineering things to make them do what you want, you can pass this. It’s not gonna be easy, but your job isn’t easy either. There were parts of the lab that we wish would have been even harder to show off our particular skill sets, but they would have required making the lab longer – and remember, we didn’t even finish the work we’d already been given.

Passing the exam shows you know what’s in the book. Passing the lab shows you know what to do when the book runs out of pages.

Even in this seriously talented group of candidates, we all missed things. As we poured our hearts out over beers, many of us wished we could have another go at the same lab just to time ourselves and see if we could do it better. A lab like this really shows who’s in it for the technical challenge, for the internal pressure, and the excitement. We were very, very, very hard on ourselves, and one guy said it best: “If I was the instructor, I’d fail me. I expected better.” So many of us just aren’t satisfied with our skills and our results no matter how good they get. We always want more, and we never stop pushing.

But that’s how we ended up in the Master program.

What It Takes to Pass the MCM

Microsoft Certified Masters don’t know everything – but then, you can’t. There’s too many parts in the SQL Server box, and no one human being can know how all of them work. New features come out faster than you can learn them all, so you have to be able to prioritize. And even if you know them all, it’s not enough, because things outside of SQL Server are just as important. A couple of years ago, I got into SAN administration and virtualization to make myself a better DBA, and now I’m looking forward at other technologies to improve my DBA skills too.

There is no sharply defined guide to database administration – or any job – with a first page and a last page. The act of “working” is a morphing set of tasks that isn’t limited, and even the act of building the task list is part of the art. Being a Microsoft Certified Master is about being able to recognize what’s important for you to know, knowing the bejeezus out of that stuff, and just as important, being able to quickly figure out the parts you don’t know. It’s more than knowledge, more than training, more than task management, more than priorities – it’s the sum of all those, and how they make you a better SQL Server professional on the whole.

So as hard as it was, I’m very proud to say I passed the exams and the lab on my first try.

I’m a Microsoft Certified Master for SQL Server.

Brent Ozar

Brent specializes in performance tuning for SQL Server, VMware, and storage. He's one of the very few Microsoft Certified Masters of SQL Server, a published author, and a Microsoft MVP. He likes travel, Jeeps, Apple gear, jokes, and writing about himself in the third person. Read more and contact Brent.

Website - Twitter - Facebook - More Posts