We’ve added two new one-day pre-conference classes to our 2015 calendar:
SQLSaturday Boston Pre-Con: Developer’s Guide to SQL Server Performance – You’re stuck with a database server that’s not going fast enough. You’ve got a hunch that the biggest bottleneck is inside the database server somewhere, but where? In this one-day class, you’ll learn how to use powerful scripts to identify which queries are killing your server, what parts of the database server are holding you back, how to tackle indexing improvements, and how to identify query anti-patterns. The class is just $99 – learn more now.
SQLRally Nordic Copenhagen: Performance Tuning When You Can’t Fix Queries – Brent is flying over to Rally again! Here’s the abstract: Your users are frustrated because the app is too slow, but you can’t change the queries. Maybe it’s a third party app, or maybe you’re using generated code, or maybe you’re just not allowed to change it. Take heart – there’s still hope. Brent Ozar does this every week, and he’ll share his proven methodologies to performance tune with indexes, SQL Server configuration switches, and hardware. Learn more now.
So you’ve got a great idea for a new feature to add to sp_BlitzSomethingOrOther, what’s the best way to get started?
The Documentation Says…
If you read our code contribution guidelines, you should write new code, write a test, sign an agreement, and then send us your code. That’s technically correct, but it’s a daunting task. After all – five people are going to be looking at your code and then thousands more might be looking at your code.
The documentation is technically correct (like most documentation), but it assumes a lot.
If you go by what the documentation suggests, we’ll definitely see your code, but there’s a decent chance that we’re not going to accept your code contribution.
The best way to get started on any existing project is to start small. It’s rare to write a major feature on your first contribution to any new project or product – there’s a significant barrier to entry around code formatting and style, future features, and work in progress.
The best way to help out is to fix a bug in the code.
Right now you’re saying, “Fix someone else’s bugs? No way!”
Hear me out.
When you find and fix a bug in the code, you’re signaling a few things. The first thing you signal is that you have a better eye for detail than the moron who wrote the code. The second thing you signal is that you want to help that moron make their software a little bit better.
Build Trust and Understanding
Contributing small fixes to an existing code base goes a long way to establishing trust. It’s one of the ways that we all work together and bring new employees up to speed with our tools. We don’t throw each other into the deep end (much). Instead we get familiar with our own software tooling by looking for issues and fixing them. We build up trust in each other as we’re building up knowledge.
By fixing bugs, you’re building trust and establishing a working knowledge around a particular code base.
Beyond building trust, you’re also getting an understanding of how a particular piece of code is put together. As an example, sp_BlitzCache is riddled with dynamic SQL, XQuery, XPath, and strange DMV ordering. It’s all done for a reason, and that reason is performance. A few changes would take sp_BlitzCache from finishing in 30-90 seconds to finishing in 30-90 minutes – I should know, I’ve introduced those changes before.
As you’re in the code fixing bugs, you’ll spot places to add more features and functionality. This is a great place to strike up a conversation with the authors about adding those new features, or at least getting them on a roadmap.
Sometimes, we’re already working on a feature but we haven’t made anything about it public yet. You don’t want to spend hours writing a new feature only to see it come out in a completely different format. Building up that relationship of trust means we’ll be chatting with you about our ideas and you’ll be aware of our crazy ideas as they happen.
…But Test First
The best reason to start out by fixing bugs is that we have a very strange test set up. By testing your changes the same way we test our changes, you can rest assured that your changes will be accepted on their merit, and not rejected on a technicality.
We test our code changes on multiple versions of SQL Server and we use case sensitive instances. A simple mistake in a column name can stop a query from running, for some users, and we’d rather be safe than sorry.
Too Long; Didn’t Read
In short, the best way to get started contributing to sp_BlitzWhatever is:
- Find a bug.
- Fix the bug.
- Submit your fixes.
- Rinse. Repeat.
- Work up to implementing bigger fixes & features.
Your users want Relativity to be up at all times. What’s the first step? How much work is involved? What will it all cost? I’ll give you a simple worksheet to get management and the IT team on the same page, and then show you how to turn those specs into a rough project plan and budget.
You’ll learn how to choose between different high-availability methods, and understand why clustering is such a no-brainer choice in this 22-minute video.
Got questions? Join us for our Tuesday Q&A webcast where I’ll answer your Relativity questions.
The latest version of our free SQL Server health check adds some nifty new stuff:
- Checks for non-default database configurations like enabling forced parameterization or delayed durability
- Looks in the default trace for long file growths or serious errors like memory dumps
- Checks Hekaton memory use and transaction errors
- Warns about database files on network shares or Azure storage
- Added the server name in the output if you enable @CheckServerInfo = 1
- Discontinued the Windows app version (was prohibitively expensive to get it into the Windows app store)
- And miscellaneous bug fixes and improvements
It can be a tricky to introduce yourself to people at conferences when you read their blogs and watch their presentations and videos. You feel like you know them, but they don’t know you yet. What the heck do you say? So you take a deep breath, you head on over, and then… you say something really awkward.
I’ve totally been there too, with my foot right in my mouth. I am truly socially awkward, so much so that reviewing posts like this helps me get better at small talk before a big event.
Dodge the Backhanded Compliment
“Your presentations are great, even though ____.”
What it sounds like: “You’re not that good.”
What to say instead: “Your presentation on [folding paper towels] was great.”
It’s OK if you already knew how to fold a paper towel, they won’t assume you’ve never seen one before. Just stop before the “even though” or “but”. Less is more! And trust me, a simple concrete statement explaining that someone’s hard work helped you out will truly mean a lot to them. If you want to follow it up with a question to start a conversation, you can chase it with, “What inspired you to present on that topic?” (If you have real criticism, it’s valuable to share that too– after a few sentences. It’s kinda weird to lead with that.)
Sneak Past that Accidental Brush Off
“I wish I could come to your talk, but _____”
What it sounds like: “I’m not coming to your presentation.”
For newer speakers, it’s disheartening for them to hear this: they just hear “I’m not coming” and they immediately picture having to present to a room full of empty seats. That can have kind of a “sad Eeyore” ring because many speakers fear “what if nobody shows up?” (This is really hard to not say by accident, I’m still training myself out of it.)
What to say instead: “I like your title and abstract for [Cool Story, Bro].” It’s OK, you don’t have to attend. But you also don’t have to explain that you’re not going, unless they specifically ask. If you want to start a longer chat, it’s also great to ask them, “I’m thinking of attending Mladen’s session on [Security for Developers]. Do you know any other good sessions on that topic?” Speakers love to help you figure out what session to attend, even if the session isn’t their own.
Ooops, Did I Just Kinda Call You Ugly?
“You look so much _______ in person!”
What it sounds like: “You look short/dumpy/frumpy/bad in some context.” Even if you’re saying they look great now, this, uh, implies that’s not always the case.
What to say instead: “It’s great to meet you face to face.” Don’t worry, they think it’s great to meet you as well. If you’re in a place where you can have a chat, just ask, “How did you get started [publishing hilarious animated gifs on the internet]?”
Pack Your Bags For A Quick Guilt Trip
“Do you remember me? We met at ________.”
What it sounds like: “It makes me feel bad that you don’t remember me. And this might be a trick question and I’m totally trolling you, you won’t know till you answer.”
Some people are really good with names and faces. Oh, how I envy those people! For the rest of us, we do truly feel guilty if we met you before and you know who we are and we don’t remember your name. The real problem with this is that it’s hard for the conversation to not fall flat after this. It doesn’t go anywhere.
What to say instead: “I think we may have met at [a store selling panty hose in Texas] a few years back. It’s great to see you again!” Just adding a little “maybe” in there automatically puts the other person at ease if they don’t remember the situation for whatever reason.
Boy, My Tribe Sure Is Dumb!
“I love your blog posts. My developers are so dumb though! They always are doing _____.”
What it sounds like: “I don’t have anything nice to say about anyone. And maybe you’re a member of that club.”
I think some folks start like this because it’s a way of saying, “we must have this in common, right?” But it just doesn’t work so well. Leading with a negative statement gives the conversation a weird vibe, and you’re gambling that the person actually agrees with you. Sometimes they don’t!
What to say instead: “I can really relate to that blog post you wrote on [juggling chainsaws].” And if you don’t remember a specific post to talk about, but you can think of a topic, just mention that. Letting people know that you read their work is pretty darn exciting for them, all by itself.
I’M NOT REALLY INTERESTED IN YOU
“You talk to ___, a lot right? Where are they?”
What it sounds like: “You’re not that important to me, but your friends are.”
It’s totally normal to talk about what you’ve got in common, but don’t start with the people commonalities. Talk about what’s important to that other person – their work, presentation, family, or even just get coffee.
What to say instead: “What’s the #1 thing on your mind this week?” This lets the person share their excitement with something, and it’s contagious. You might learn something about a fun insider event too!
How to End It, Short And Sweet
Most initial conversations at conferences aren’t very long. You’ve got sessions to go to, there’s tons of people around saying hi to each other. But don’t be afraid to end it: you’ll probably run into one another again soon. Offer to trade business cards! That will help you remember each other– especially if your business card has your picture on it.
Small Talk is Just a Skill
You’ve got to start a conversation somewhere, and at a conference, you start it with small talk. We aren’t all naturally good at it, though.
Know this: even if you do end up with your foot in your mouth, it’s OK. Just smile and keep meeting new people. We’ve all been there, and it’s not nearly as big a deal as it feels like when your face turns red.
In most coding languages, functions are often-used blocks of code that can be reused from multiple locations, leading to less code – and cleaner code. SQL Server also lets us create functions that can be used the same way. They are reusable blocks of code that can be called from multiple locations. So, if you need to format phone numbers a certain way, or parse for specific characters, you can do so using a function.
The question is, how much work is SQL Server doing when you call a function? If it’s the SELECT clause, is it called once – processing all rows – or once for each row in the result set, regardless if that’s 1 row or 100,00? What if it’s in the WHERE clause?
I’ll let you in on a little secret: if a function is used in the SELECT or WHERE, the function can be called many, many times. If the function is very resource-intensive, it could be causing your query to be very slow – and you would never see the execution of the function within the execution plan of the calling query.
Yep, SQL Server’s execution plans can be a bit vague when it comes to functions – and by “a bit vague”, I mean, “They don’t show up at all”. You need to dig deeper!
I’m going to run a few demos against the AdventureWorks2012 sample database in a SQL Server 2014 instance to show this!
First, I create a scalar-value function that will return the five left-most letters of a LastName.
CREATE FUNCTION [dbo].[ParseLastName](@LastName VARCHAR(50)) RETURNS VARCHAR(5) AS -- Returns the 5 left characters of the last name BEGIN DECLARE @ret VARCHAR(5); SET @ret = LEFT(@LastName, 5) RETURN @ret END;
Then, I create an Extended Events session to track statement completion. (Note: I have only tested this on SQL Server 2014, no lower versions.) (Using SQL Server 2008 R2 or earlier? You could create a server-side trace to capture sp_statement_completed and sql_statement_completed, but it won’t give you some functionality I’ll show later.)
CREATE EVENT SESSION [CaptureFunctionExecutions] ON SERVER ADD EVENT sqlserver.sp_statement_completed( ACTION(sqlserver.sql_text,sqlserver.tsql_stack)), ADD EVENT sqlserver.sql_statement_completed( ACTION(sqlserver.sql_text,sqlserver.tsql_stack)) ADD TARGET package0.ring_buffer WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=OFF) GO
I start the Extended Events session, and then turn on actual execution plans.
I start with a simple query, which returns 19,972 rows.
SELECT LastName FROM Person.Person;
The execution plan shows an index scan and has a cost of 0.10451.
Looking at the details of the index scan, I see Estimated Number of Executions is 1, and Number of Executions is 1.
Let’s look at the same query when it performs the same calculation as the function – LEFT(LastName, 5).
SELECT LastName, LEFT(LastName, 5) FROM Person.Person;
There’s now an additional operator – a compute scalar. The cost has risen slightly to 0.106508.
Now, I will modify the query to call the function from the SELECT clause.
SELECT LastName, dbo.ParseLastName(LastName) FROM Person.Person;
Looking at the execution plan, I see an index scan and a compute scalar. The cost is the same as before – 0.106508.
Expanding the properties for the compute scalar, I see the function, but it says there is only one execution.
A quick glance at my Extended Events live feed tells a different story.
If I add grouping by statement, I can see the function was actually executed 19,972 times – once for each row in the result set.
That’s a lot more work than advertised!
Does the same thing happen if the function is in the WHERE clause?
SELECT FirstName, LastName FROM Person.Person WHERE dbo.ParseLastName(LastName) = 'McCar';
Two rows are returned. The execution plan now has an index scan, a compute scalar, and a filter. The cost is 0.118091.
The Extended Events session again shows 19,972 executions – once for each row in the index.
The data isn’t filtered out until after the function is called, so it is executed once for each row.
These examples prove that whether one or many rows are returned as the query result set, if a function is used in the SELECT or WHERE, the function can be called many, many times. It could be one of the top resource-consuming queries in your server!
How can you see if a function is bringing your server’s performance down? Look at the top queries in your plan cache using our sp_BlitzCache tool, by total CPU and by number of executions, to see if this is happening to you.
Today at Relativity Fest in Chicago, kCura Relativity 9 introduces the option to move some text storage out of Microsoft SQL Server and into kCura’s new Data Grid, a tool built atop the open source Elasticsearch.
Is kCura abandoning SQL Server? No, but understanding what’s going on will help you be a better database administrator and developer.
kCura’s Challenges with Microsoft SQL Server
To recap some of my past posts on Relativity, it creates a new SQL Server database when one of the end users creates a new workspace and starts loading data. Over the coming weeks, data pours into SQL Server at completely unpredictable rates. We have no idea how many documents are going to be acquired from subpoenaed hard drives, file servers, backup tapes, Facebook messages, you name it. I’ve seen workspaces grow from zero to ten terabytes in a single week, all without the systems administration teams even knowing it’s happening. They ran their weekly backup size report, and surprise, surprise, surprise.
Data streams into Relativity during business hours at the very same time hundreds or thousands of document reviewers are running queries against those very same tables. The entire team is under tight time deadlines, and there’s no way to take databases (let alone servers) offline for loads.
And oh yeah, we’re often contractually bound not to lose any attorney work product whatsoever out of the database.
This is all doable with traditional relational databases, but it ain’t easy. It’s made even tougher by the fact that many Relativity hosting partners are understaffed, many without even a full time database administrator.
How kCura and Jeremiah Planned for Change
For the last couple of years, Andrew Sieja has repeatedly asked me a tough question: “If you were going to redesign Relativity from the ground up, and anything was on the table, what would it look like?” He faced a classic case of Innovator’s Dilemma – his team had built a wildly successful product, but there’s innovation coming from everywhere, and sooner or later somebody was going to beat him to the next level. While SQL Server was getting the job done, alternative data storage platforms beckoned with some really cool advantages, and he needed to take advantage of them before his competitors did.
kCura brought in Jeremiah to work through the options out on the market. There are a gazillion new data storage & search options out there, and some of them claim to do a phenomenal job on absolutely everything. (Hint: they’re usually lying.) Jeremiah helped them prioritize the features they needed most, and then recommended the right fit for them.
The end result is the newly announced kCura Data Grid, an extremely scalable and performant search platform built atop the open source Elasticsearch. You might recognize Elasticsearch from my demos of Opserver, Stack Exchange‘s open source monitoring tool, because Stack also migrated their SQL Server full text search out into Elasticsearch. They’re not alone – Elasticsearch has plenty of high profile case studies.
The Benefits and Risks of Elasticsearch
We typically see 70-90% of a Relativity workspace’s space consumed by extracted text and audit logging, both of which are great fits for Elasticsearch. Pushing that data out of SQL Server potentially means:
- Reduced storage costs – while SQL Server relies on expensive shared storage (typically $5k-$10k per terabyte), ES achieves redundancy with multiple commodity boxes (typically $1k-$2k per terabyte). This adds up fast for big workspaces.
- Faster search – ES is mind-numbingly fast. Seriously.
- Easier scale-out – it’s really, really hard (and expensive) to scale out a single multi-terabyte database across multiple Microsoft SQL Servers when people can create new databases at any time. (It’s even hard enough just to scale a single known database across multiple servers!) It’s easy to add ES replicas for higher performance and availability.
It’s not a silver bullet, and as with any technology change, there are risks and limitations:
- Security – ES doesn’t have any built in, so kCura had to build their own.
- Backups – when everything is in one data platform, it’s easy to back up everything at the same moment in time. Split the data, and you run into challenges – but these aren’t really new for Relativity. The databases and the native files couldn’t be backed up to the same point in time either.
- Management – Relativity hosting partners don’t have ES expertise on staff, and they’ll need training as ES becomes a mission critical part of their infrastructure.
What This Means for SQL Server Developers and DBAs
Microsoft SQL Server is an amazing relational database, a Swiss Army knife of a persistence layer. Sure, it can handle tables and joins, but more than that, it can do things like full text search, spatial data, CLR code execution, and scale-out via multiple methods. You can build a product backed by SQL Server and go a long, long, long way.
I don’t think SQL Server ran out of capabilities here, but kCura needed to plan for orders-of-magnitude growth in storage and search capabilities over the coming years. They grew one hell of a big, powerful business solution with a single database back end, and they’ve got the luxury of a large development staff and a bunch of new data storage options.
Premature optimization is the root of all evil. When you’re building the product you need today, the right database is the one you already know well. As your product grows, keep learning your own database, plus learn the other options out there. The storage and search markets are changing so dramatically every year – don’t make a bet on one today unless you have to, because tomorrow might bring an even better solution for your needs.
For more details about Relativity 9, check out kCura’s Relativity 9 page.
If you lead a local SQL Server user group, and you’re coming to the PASS Summit in Seattle next month, we’d like to say thanks.
We’re giving away 5 seats to our Make SQL Server Apps Go Faster class that runs on Monday/Tuesday before Summit. We know you probably have duties you need to attend to on Tuesday, but don’t worry – even if you have to leave early, you’re going to learn a lot.
Email us at
Help@BrentOzar.com with the name of the user group you run. Next Saturday, October 18th, we’ll draw 5 emails and notify them about their free tickets, plus a little extra something to say thanks.
See you at Summit, and thanks for everything you do for the community.
The fine print: Only one entrant per user group, please. (If your group has multiple leaders, just nominate someone to enter.)
Update – the contest is closed!
Our web site explains who we are. It’s our storefront. The vast majority of our clients find us in Google, get to know us by reading our blogs, start to trust us by using our scripts, and then finally contact us for personalized help and training.
For that to work well, our web site needs to say, “Brent Ozar Unlimited is a boutique database consulting firm who helps you make SQL Server faster and more reliable. They’re brilliant, fun, and relatable.”
That sounds really simple now, but a few years ago, we had no idea what we wanted the web site to say. To figure it out, we hired Pixelspoke to build our brand identity, logo, and web site. They interviewed us, talked to our clients, and checked at our competitors. They painted a really clear picture of who we were, what we do, and what makes us different. The end result is the web site you see today. They did a magical job, and I think it says a lot that we love their work just as much today as we did two years ago. I still grin when I see our logo.
Last year, we upped our game by having Eric Larsen draw more funny portraits of the team, and rotating those on the site’s home page. We loved the results – the guy is crazy talented. He drew us teaching various SQL Server concepts, and his illustrations managed to capture exactly what we wanted the site to say.
Leveling Up: Explaining Our Process in 106 Seconds
Some customers don’t have time to do the long courtship of gradually getting to know us. Their SQL Server is desperately sick right now, and they don’t have the luxury of getting to know all the possible health providers out there. They’re Googling and judging people by what they see in the first minute.
So this year, we wanted to take our web game up another notch by hiring a partner to build us an amazing 2-minute video. Eric pointed us to his friends at Epipheo. Like us, they’re a company that focuses on exactly one thing – but theirs is making short videos. After examining their portfolio (my personal favorite is Projetech’s) and talking to them, we were hooked.
Epipheo’s mission was to distill who we are, what we do, and what makes us different down into a single video. I’m absolutely convinced they did it.
The video tells the viewer:
- Your SQL Server is sick, and you don’t know how to get relief.
- You could throw money at it, but that doesn’t usually work. (Many of our clients even tell us that they’ve already tried this by the time they contact us.)
- You could hire The Specialist, but he won’t explain what he’s doing, and you have no idea if it’s good or bad
- You could bring in The Large IT Firm, but it costs a fortune, and they’ll overwhelm you with irrelevant data
Or you could hire us:
- We focus on the pain that’s bothering you, not a gazillion unrelated “best practices”
- We work side by side with you, explaining your SQL Server’s symptoms as we go
- We teach you our techniques so you can repeat the process on your other SQL Servers
- And then we leave you alone, because we’re not here to drain you dry.
Yep. That’s exactly what our SQL Critical Care® is, and why it’s unique. Next up, we’ll talk about our favorite parts of the video where Epipheo really captured our personalities.
We’ll sum it up in a 106 second video about our SQL Critical Care®:
In tomorrow’s post, we’ll talk about why we made a video. In the meantime, enjoy. I know I certainly do – I’ve re-watched this dozens of times, and I still giggle. It’s on our home page, and we can’t wait for everyone to see it and giggle too.