There’s only one correct answer: it depends.  Successful performance tuning boils down to a simple cycle:

• Measure the application’s performance
• Find the current bottleneck
• Improve that bottleneck so that it’s not the bottleneck anymore
• Measure to find out how much your application performance improved
• Ask the application owner if it’s good enough now. If so, move on to the next application. If not, go back to step 2.

And every one of those steps is equally important.

If You Don’t Find the Right Bottleneck

This Bottleneck Is Plenty Big Enough for You

Performance tuning isn’t about zooming in and focusing on a single number in incredible detail; rather, it’s about stepping away and getting the full picture. Time and again, I get emails asking about whether a single metric is OK, but upon questioning, the DBA has leapt to a conclusion without surveying the environment as a whole. If you spend your tuning time closely examining a single metric, trying to figure out how to improve that one metric, you might not improve the performance of your application.

Sure, your page life expectancy might be pretty bad – but is that the one thing keeping your application from performing faster?

Take a step back and gather a complete set of Perfmon metrics. Look at CPU, memory, disk, and network performance as a whole. Find the thing that’s in the absolute worst shape possible. In that link, I explain the order that I look at metrics to find which one looks like the most likely bottleneck.

If You Don’t Focus on Improving the Bottleneck

I was recently working with a client frustrated with their application performance. I found two issues:

• CPU-intensive user-defined functions were being called thousands of times per query
• The storage subsystem was nowhere near as fast in practice as the vendor had claimed

The application’s bottleneck was the CPU-intensive UDFs. The server was frequently pegged at 100% CPU, and queries just couldn’t run any faster until they were rewritten to rip out the UDFs. I put together a recommended plan of action to take those UDFs out, which would make the application an order of magnitude faster. I noted that they should probably start working on the storage performance in a second track, because the instant the UDFs were removed, storage was going to become a problem. With the CPU-burning UDFs out of the way, the server would be able to churn through more records faster, but the storage subsystem wouldn’t be able to deliver records fast enough to satisfy the users.

On our next status update call, they said they’d reworked the storage subsystem. SQLIO reported dramatically faster storage throughput, but they were only seeing a minor improvement in application performance. I had to break the bad news to them that they’d focused on the wrong problem first. After we revisited my report together, they pursued the UDFs with renewed vigor, and suddenly the application was blazing fast. Thankfully I’d documented my findings in writing, but if I’d have been an internal employee, I might have communicated that in verbal form instead. I might have lost the ensuing battle to fix the UDFs because the manager would have thought my advice was bogus.

If You Don’t Measure Your Improvements

You Get What You Measure

All DBAs are consultants.

Some of us think we’re full time company employees, but in reality, we’re delivering a service. Whether they’re developers, project managers, end users, or other DBAs, they’re looking at you just as if you were an outsider. You’re expected to stride in, identify the problem, mitigate it, and show that your work delivered a return on investment. The investment, in case you’re not following, is your paycheck.

Don’t believe me? Poke around in an application and then throw up your hands, saying you can’t find a way to make it go faster. The next phone call the project manager makes will be to an external consultant, and the project manager probably won’t call you first next time. (Sometimes, that’s not a bad thing.)

If you put in a lot of hard work to make an application go faster, but you don’t measure the before-and-after effects of your work, someone else is going to take credit. The developers will say the improvements were due to a new version of their code, because they’re working on the code at the same time you’re working on the database. The sysadmins will say they defragged the muffler bearings. The SAN guy will say he tweaked the flux capacitor. The project manager will say he made everybody worked long hours and that did the trick. The only way, the ONLY way that the DBA can ever take credit is to take clear before-and-after measurements for proof. Run the same code base before & after your tweaks, and measure application performance. Follow up with a written report, even if it’s a one-paragraph email, summing up your changes and the performance improvements, and copy your manager on it.

If You Don’t Ask the App Owner If It’s Good Enough

Never confuse what YOU think about a metric with what the BUSINESS thinks about a metric. Your CEO doesn’t care about page life expectancy. (Your CEO probably doesn’t even care about the DBA’s life expectancy.) Before you spend time or money improving an application, you have to find out whether it’s the most important thing to your business right now.

Say hello to the most important metric you will ever calculate: opportunity cost.

A Long Night at Zig Zag Has Its Own Opportunity Cost

Opportunity cost is the cost of doing something as compared to the cost of doing something else. If you spend eight hours today improving the page life expectancy of a particular server, is that worth more to the business than anything else you could be doing in those eight hours? Could you spend eight hours doing something more valuable?

I use opportunity cost whenever anyone asks me to do something.

As an employee, if a project manager asks me to tune a particular application, I bring them into my manager’s office and say, “It will take me three days to make that application faster. I’ll probably make it an order of magnitude faster, because I’ve never tuned that server before. However, if I take those three days to do it, I won’t make the deadline for Project Snazzywidget. Which one is worth more to you?” At that point, it’s a political decision and a business decision, not a technical decision. If you’re doing the best job of any employee he has, your manager will put you on the most valuable project – which in turn increases your worth again.

As a consultant, I approach the problem differently: “Here’s the thing – I could spend another three days working on this application, but from this point forward, I’m only going to be able to achieve incremental improvements, not the order-of-magnitude improvements we saw in the first few rounds of tuning. I hate to make you guys go through that for a small gain – but is there another application in-house that isn’t delivering the performance you want?” This resets the client’s expectations, and they start seeing you as a weapon that they can point at slow applications. They’ll cherish your time and focus you where your effort will pay off the most. This keeps your perceived ROI high. If you deliver jawdropping results each time you tune an application, you can justify higher billable rates.

After all – isn’t your bank balance the one metric you really want to improve?

...
Upcoming free webcasts: SQL and SSDs: A Valentine's Day Love Story and Anatomy of the SQL Server Log File.

Brent Ozar

Brent specializes in performance tuning for SQL Server, VMware, and storage. He's one of the very few Microsoft Certified Masters of SQL Server, a published author, and a Microsoft MVP. He likes travel, Jeeps, Apple gear, jokes, and writing about himself in the third person. Read more and contact Brent.

Website - Twitter - Facebook - More Posts

The Finished Product

Erika has champagne tastes; she can walk into any store and find the highest-priced item without ever seeing the price tags.  She’s just drawn to expensive stuff.  She’s had her eye on a $1,000 marble-top cocktail table (they’re not called coffee tables anymore, don’t ya know) from Room & Board for months.  Therefore, it was a complete shock when we both really liked this $50 table with exactly the same lines and shape, but a simple white veneer top instead of marble.

$950 savings?  Sold.

Granted, we won’t be passing this table down to our grandchildren, but we don’t even plan on having kids, let alone grandkids.  And this table might not last ten years, but at $50 a pop, we could buy a new one every five years for 100 years straight.  Odds are, our tastes will change over the course of the next 100 years, too.

The catch with Ikea is that the stuff is flat packed – the table comes in a flat box.  You’re responsible for hauling it through the aisles of the store, getting it home in your car, and assembling it yourself.  I’ve done this enough over the years that it doesn’t stress me out (anymore).  My tips for a successful Ikea build are:

• Read the instructions three times – which is pretty simple, since they’re pictures, not words
• Use an electric screwdriver to avoid exertion
• Drink your favorite alcoholic beverage in moderation during assembly
• Whenever you’re about to curse, stop to think about how much money you saved
• When you’re done, don’t stand on it to test it

All of those are equally important.

If you don’t read and reread the directions, it’s very easy to end up with a desk that shakes when you type on it – and I happen to be typing on one of those at this very moment.  If you don’t use an electric screwdriver, your arms will be tired before you get halfway through, and you’ll strip out the screw heads.  If you don’t drink, you won’t be mellow enough to laugh at the pictures in the directions.  If you don’t constantly remind yourself of how much money you saved, you’ll be picking up the phone to call Crate & Barrel.

And finally, unless you bought a piece of furniture that was specifically designed for you to stand on, then it wasn’t.  Don’t stand on it to prove a point, because more often than not, you’ll prove that it wasn’t designed with standing in mind.

Ikea Table - Before

SQL Server is the Marble Coffee Table

When you’re building a brand-new product, company, or web site on the Microsoft stack, SQL Server Enterprise Edition is the sexy marble-top cocktail table.  That’s the one you really want, baby.  You just know all your friends will ooh and aah when you bring ‘em over for the opening night party and say, “Yeah, I built it with SQL Server.”

“But it’s so expensive!” they’ll say.

“Yeah, but my app is worth it.  SQL Server Enterprise Edition scales like there’s no tomorrow.  Built-in backup compression, partitioning, database mirroring, active/active clustering, and all that stuff those MySQL guys can only dream about.  We’re ready for the future now, baby.  Want another glass of champagne?  My venture capital guy bought us a couple dozen cases of the good stuff.”

Riiight.  Back to reality.

SQL Azure is the Flat Pack Database

Much like Ikea furniture, Microsoft SQL Azure is a cheap way to get a reasonable facsimile of the database you really want, but can’t justify buying.

There’s drawbacks:

• Backups are not included – if you want to get your data out, get out your electric screwdriver and build it yourself
• Scaling is not included – you have to roll your own sharding, and frankly, I don’t know anybody who does a good job of that
• Database mirroring, partitioning, clustering, and batteries are not included
• You can’t stand on it – don’t use this for your data warehouse or high-throughput databases, because the data goes through your internet connection

But if you can live with those drawbacks and build your own HA/DR solution – just like you dragged home your Ikea coffee table and assembled it yourself – you can save a lot of money.

Both SQL Server and Azure can – and will – coexist.  Heck, Ikea furniture happily coexists with the good stuff in my house too.  Now, if you’ll excuse me, I’ll go refresh the Crate & Barrel web site to see when my dining room table and chaise lounge are going to arrive.

...
Upcoming free webcasts: SQL and SSDs: A Valentine's Day Love Story and Anatomy of the SQL Server Log File.

Brent Ozar

Brent specializes in performance tuning for SQL Server, VMware, and storage. He's one of the very few Microsoft Certified Masters of SQL Server, a published author, and a Microsoft MVP. He likes travel, Jeeps, Apple gear, jokes, and writing about himself in the third person. Read more and contact Brent.

Website - Twitter - Facebook - More Posts

I’ve been working as an (insertJobHere) for a few years, and I’d like to start looking for another job as an (insertBetterJobHere).  How can I get started?

It depends on the phase of your career.

Phase 1: The Kindness of Strangers

When you’re just getting started – whether you’re fresh out of college or currently enrolled – you’re not going to know anybody.  You’re going to be cold-calling strangers, sending your resume to people you’ve never met, and hoping that you can knock ‘em dead in the interview.

Headhunter Company Car

Examples of hiring methods for Phase 1 are:

• Newspaper and online ads
• Recruiters
• Consulting companies

There’s two problems with Phase 1.  First, you’re going up against complete strangers.  People in Phase 1 are desperate, and they’ll do some crazy stuff to beat you out in the job hunt.  They’ll work for less money, they’ll lie about their resume, they’ll use their mom as a reference, you name it.  Getting a better job when you’re going up against strangers can be a nightmare.

Second, you’re going to work for complete strangers, which can be a living hell.  You won’t find out that they’re raving lunatics until after you’ve already quit your current job and gone to work for them.  Every time I’ve gone to work for strangers – every single time – I’ve regretted it within a matter of months.

Strangers hire other strangers when they run out of friends willing to work for them.

Phase 2: The Buddy System

After you’ve been working for a few years, going to local user group meetings, and survived a few reorgs, you’re going to know a lot of other people working in your same field.  You’ll even know people in unrelated fields.  DBAs meet project managers, end users, report writers, developers, and so on.

Everybody Loves the Buddy System

When it’s time for you to start looking for another job, you can put out the word among your contacts.  Because they’ve worked with you before and they know what a kick-ass worker you are, they’re likely to vouch for you.  They’re likely to tell you about jobs before they go into Phase 1 recruitment, because they’d rather hire you than a stranger.  You’re a sure bet, because you’re doing such good work.

The only way to get from Phase 1 to Phase 2 is staying in touch with people.

I can’t emphasize this enough – if you don’t learn to network, you’re going to be permanently stuck in Phase 1.

Networking sounds so slimy, but it just boils down to this: meet people, and keep in touch with the ones you genuinely like.  I used to think networking meant rubbing shoulders with greaseballs who were trying to sell each other stuff, but that’s not true at all.  Just keep in touch with people you like, and go to the places where they hang out to meet more people like them.  For database professionals, that’s the upcoming PASS Summit.

Befriend Buddies, Not Just Bosses

One fast way up the corporate ladder is to hitch your wagon to a rising star.  If you do great work for a great boss, you can both go places quickly.  I’ve taken that approach a few times in my career, and it’s served me really well.  I remember getting a call from one boss as soon as he’d changed companies, and he said, “I know what you’re making now.  I’ll give you an extra $X to come work for me at the new shop, and you know I’ll be good to you.”  Sold.  However, that’s not the only way up.

The programmer next to you today might be the CIO at another company tomorrow.  A few years ago, Matt Mullenweg was just another guy I ran into at Houston Wireless meetings.  I had no idea he was building something cool in his spare time, and since then, that little blogging platform has raised tens of millions of dollars in funding.

Look around you right now.  Your coworkers and customers are the only ones who will take your career to the next level, regardless of how unskilled and antisocial they seem.  Even if they’re really, really bad at what they do, it’s your job to make them look better and leave a permanent positive impression on them.  They might be your key to a better job down the road.

If you wait to network until you need a job, it’s too late – and you’ll be stuck in Phase 1.

...
Upcoming free webcasts: SQL and SSDs: A Valentine's Day Love Story and Anatomy of the SQL Server Log File.

Brent Ozar

Brent specializes in performance tuning for SQL Server, VMware, and storage. He's one of the very few Microsoft Certified Masters of SQL Server, a published author, and a Microsoft MVP. He likes travel, Jeeps, Apple gear, jokes, and writing about himself in the third person. Read more and contact Brent.

Website - Twitter - Facebook - More Posts

I Can Quit Anytime I Want*

Grandma and some friends took a road trip to Las Vegas, and while they were gone, I threw an epic party.  We’re talking 30 gallon garbage cans filled with homemade Jungle Juice.  (I’m one of those reasons she couldn’t completely eliminate the Valium.) A couple hundred of my closest friends had a good old time.

A couple of my less-than-closest friends had a little too good of a time at my expense and started trashing the house.  They walked up the staircase smashing the picture frames of every family photo, then started to throw a couch off the second floor balcony.  My security guys (I’m telling you, it was that good of a party) carried them out before they got too carried away.

Cleaning Up After The Party

The next morning, the very-closest-friends did a fantastic job of getting things back to normal.  We replaced all the picture frame glass, got the dirt out of the sofa, vacuumed the place top to bottom, and finished the Jungle Juice.  I thought we’d hidden all our tracks, but we got busted by the tiniest of clues.

Someone had left beer bottle caps on top of door jambs all around the house.

Amazing!  Who thinks of this stuff?  You know they did it on purpose, too – they were just itching to get me into trouble.  One beer cap on one door jamb, I could understand, but all over the house?  Damn.

I was disinherited for that particular shindig.

I learned a valuable lesson: if you’re not absolutely sure you can clean up every trace of everybody else’s messes, you shouldn’t throw parties.  Sooner or later, somebody you can’t trust is going to show up at your party, and they’re going to do something that’ll get you in trouble.  Come to think of it, it’s just like being a DBA.

SQL Server 2008 Data Compression: No Inheritance Either

Microsoft SQL Server 2008′s Data Compression feature lets you compress objects – not just tables, but individual indexes.  This compression does incur a little extra CPU power to handle the compression, but that extra overhead is more than offset by increased IO performance.  Generally speaking, the database server is sitting around waiting on disk subsystems.  Adding a little CPU work while dramatically reducing IO needs results in faster query return times.  You need to test compression to see if it works well in your environment, because it may not work well in heavy-insert databases.

Today, though, I’m going to focus on the dark side of compression: a complete lack of inheritance.

When you compress tables and indexes, it’s a one-time action.  You’re only taking care of what exists today.  If someone (or even you) turns around and creates an index on that same table tomorrow, it won’t be compressed by default.  Whoever creates the index has to make sure that it’s compressed, and there’s nothing in SQL Server Management Studio that will hint to them that other parts of that same object are compressed.

To make matters worse, your development, QA and production environments might all have different compression settings, and you’d never notice it at a glance.  Compression is transparent to applications, so your developers won’t know why one environment performs much differently than another even though they have the same hardware, same indexes, same statistics.

Implementing compression is a multi-step process:

• Figure out what objects you should compress
• Plan to handle all of your environments (dev, QA, production)
• Compress them during a low-activity window
• Regularly patrol your environments checking for added objects that weren’t compressed
• Keep your environments in sync

If you don’t stay on top of all of these, you’ll need Valium too.

* – Yes, about that picture.  No, it’s not mine.  Yes, it’s licensed with Creative Commons.  No, it wasn’t even the worst jungle juice picture I could find licensed with Creative Commons.  Yes, I too am amazed that people upload their party pictures to Flickr, let alone license them with Creative Commons.

...
Upcoming free webcasts: SQL and SSDs: A Valentine's Day Love Story and Anatomy of the SQL Server Log File.

Brent Ozar

Brent specializes in performance tuning for SQL Server, VMware, and storage. He's one of the very few Microsoft Certified Masters of SQL Server, a published author, and a Microsoft MVP. He likes travel, Jeeps, Apple gear, jokes, and writing about himself in the third person. Read more and contact Brent.

Website - Twitter - Facebook - More Posts