The Way Developers and Users Interact is Broken.

This month, to mark the 20th anniversary of BrentOzar.com, I’m stepping back and looking at the big picture rather than blogging about database problems & solutions like I usually do.

Twenty years ago, I tried to switch to Linux.

Again.

That wasn’t my first attempt, nor was it my last – there were so many more – and every time, I had a terrible experience. I struggled to get the right combination of hardware, software, and configuration settings. But you know what the worst part was? Trying to get support for all of it.

I saw myself as a decently skilled sysadmin and developer with a good amount of experience and troubleshooting skills, but I had one hell of a bad time getting support. Things people said would work, didn’t, and nobody seemed terribly interested in helping me get it right. Documentation, forums, email lists – it was a byzantine mess, nobody was accountable for getting things right, and I felt firsthand the impact of gatekeepers and condescending answers. The more I tried to make it work, the more burned out I got.

In 2006, I gave up and got a Mac.

I gave up the idealistic hope of a complete open source stack, configuration via simple text files, and the feeling of being a wizard. I wasn’t an operating system wizard. I just wanted something to boot up, connect to WiFi and a VPN, install applications, and sleep & resume, and when something broke, I wanted to get support from a company that would be personally accountable for getting it working again. I was fine with handing over money and permissions in order to get simplicity.

Today, I maintain the open source
First Responder Kit.

It has SQL Server scripts like sp_Blitz that help you assess your SQL Server’s health and performance. They’re the same scripts I use in my own consulting work, and are used by data professionals around the world. Back when I used to track downloads, I was tickled pink to see that even Microsoft’s own support staff downloaded them.

Today, as an open source maintainer, I see the other side of the story.

Many users don’t read the documentation. They try to use the scripts in completely inappropriate and unsupported ways. They complain when the scripts don’t do what they want. They see all this as my own responsibility, saying things like:

Eyes up here, kid
Your call is important to us, please hold.
  • “You should make it be able to do ____”
  • “I’m very disappointed that it doesn’t ____”
  • “It needs to ____ or else it’s worthless”
  • “Please do the needful”

When I point out the readme and the documentation, which clearly states that if you want a new feature, you need to start coding for it – it almost always falls on deaf ears. I’m endlessly thankful to the folks who actually do contribute code, documentation, and support, because they’re so few and far between compared to those who see open source as free consulting & development.

This is hilarious timing, but as I was writing this, my watch vibrated with a direct message in the SQL Server community Slack from someone who wants help. I responded asking them to try in the #FirstResponderKit channel as the readme says so that other folks can help too. (I checked back an hour later – they never did.)

I totally understand that a lot of the First Responder Kit users who need help are just as qualified as I was when I was trying to get Linux help. I wasn’t an amateur, and neither are these users. But it’s just an endless flood: there are always way, way more inexperienced users than there are experienced users. There just aren’t enough experienced users with free time to help, and many of the people who see themselves as experienced…aren’t.

It happens with Microsoft, too:
try Azure Data Studio.

It’s one of Microsoft’s newest open source products for database developers, a SQL Server Management Studio replacement for people who write T-SQL rather than manage servers. When you run into a problem with Azure Data Studio, you’re expected to file it in Azure Data Studio’s Github repo.

There are over 1,000 open bugs, like serious bugs that I hit all the time, many of which have been open for years.

When you browse that list, it’s hard to determine if the software is buggy, if the users need better support, or if it’s a combination of both. Pro tip: it’s always a combination of both.

There are almost 1,000 open requests for enhancement, and many aren’t trivial requests: they’re monsters, like adding support for entirely new database platforms. Users feel free to ask for a pony because there’s no cost to them for making the request. Any bozo can ask for a pony by writing up a Github issue, and they can gather thousands of upvotes by posting it on social media or forums.

I don’t blame Microsoft for how this is going down. This is what happens when you build complex cross-platform software that runs on multiple operating systems and targets multiple databases. It’s the same story with Visual Studio Code’s issues, where there are also over 1,000 open bugs going back to 2016.

These aren’t open source problems.
They’re software problems overall.

It’s a problem with $1 phone apps, where users resort to leaving bad reviews and developers try to do tech support by responding to the review.

It’s a problem with Microsoft SQL Server, one of the most expensive database platforms in the world. You would think that when you hit a SQL Server problem and you think it might be a known bug issue, that it’d be easy to go to a web page, search for the symptoms, and see a known list of bugs and the status of whether they’ve been fixed or not.

For example, say you’re hitting issues with CMEMTHREAD waits. You go to support.microsoft.com, put CMEMTHREAD in the search box, and you get this wall of text:

There’s no sorting and filtering for database versions. You’re left to click on every single post and figure out whether it’s relevant to your patch level. There’s no commenting or questions, either. At the bottom of each post, there’s a “Need more help?” search box – which just takes you back to this search result list.

And these are only the published, fixed issues.

Not the known broken issues. Those are hidden away in secret for only Microsoft employees to access.

If you’ve been working with SQL Server for a while, you might know about feedback.azure.com, a place where you can post known bugs with SQL Server in the vain hopes that Microsoft will do something about it. In rare cases, they do – but in most cases, the requests go unaddressed. And just like its predecessor, Microsoft Connect, this feedback site is going away in 2021.

Closed source software only seems like it doesn’t suffer from the extensive bugs problem because the bug list is kept hidden by the manufacturer. We users thought the software was reliable, but if you talk to the really experienced pros in the industry, they’re often furious by just how broken common features are.

The way developers and users interact
is completely broken.

I’m hopeful that someone’s going to solve these problems during my lifetime.

Stack Overflow might be one way to solve it, but they’ve got a lot more work ahead of them. Right now, the newest incoming questions feed is an insane firehose, and the DBA.StackExchange.com new questions, while lower in volume, is still too challenging for the volunteers to solve alone. Maybe Stack could solve it by doing a better job of walking users through crafting good questions, and then recruiting (and paying) vendor employees to answer questions about their products.

Microsoft’s acquisition of Github might be another way to solve it because there’s no way they’re happy with the way this is going down, either. Microsoft could tune the new-issue process at Github and start handling paid support requests through Github issues. I’m sure I wouldn’t be the only open source maintainer who would gladly charge $10 for folks to open a support case just to cause ’em to think more before asking for a pony, and offer a higher-paid issue tier with a higher service level agreement.

Or maybe it’ll be an all-new Software-as-a-Service provider, but…I don’t think that’s going to be the case, given how Microsoft experimented with UserVoice and then pulled back to their own in-house products.

I don’t know what the right answer is, but the way we’re doing it now is completely busted. I should be thankful, though, because the fact that SQL Server support is so completely broken happens to enable a delightful career in fixing that very product’s shortcomings.

Previous Post
Updated First Responder Kit and Consultant Toolkit for March 2021
Next Post
Free Live Webcast: The First SQL Server Webcast You Should Ever Attend.

10 Comments. Leave new

  • I use VS Code, Azure Data Studio, and Azure Storage Explorer daily, all based on the VS Code core according to MS. I’m familiar with a lot of those bugs, and I’ve got the ‘learned helplessness’ of the guy who just reboots the tool when I hit one of those snags.

    The most mind boggling piece of this, to me, is that I get a notice *several times each week* that the tool wants to update itself. It really makes my OCD itch to defer it, but sometimes I need to get something done, know what I’m saying?

    But despite the absurdly frequent point releases, the big issues are still there each time…

    Reply
  • Vince Thomas
    March 24, 2021 6:18 pm

    I agree with you wholeheartedly, but just like flying somewhere these days – we are stuck in a holding pattern for now. 🙂

    Reply
  • I relate to your experience with Linux. In 1990-1991 I taught myself how to code Windows 3.0 applications in C. No small feat. Then, I taught myself SQL and PowerBuilder with no outside assistance. In 1999 I was able to stand-up a Linux Oracle box and code against it, but even using a cookbook approach it was a slog with many pitfalls. In 2004, feeling confident and full of myself, I decided to switch to Linux for all development. My long-overdue humiliation rapidly ensued, for all the reasons you mentioned. What a disaster. Decided to leave Linux to uppity guys with beards (no offense) who sneer at the lowly idiots who use Windows OS.

    Speaking to your broader topic, technological change is the cause of that firehose of questions. The speed of advancements in security, storage technologies, servers, and development tools is overwhelming. We are all spending more and more of our time trying to keep up with changes. This is not just a customer support problem. Rapid change saps productivity with time spent perpetually learning, and also introduces bugs and meager documentation faster than all the forums and blogs in the world can help people stay productive. Not to say change is bad. Much of it is very, very good. It just has a cost. Brent, you have some good ideas for keeping us productive. Thank you for that.

    Reply
  • Some of this is why my experience is not always helpful to our user support folks. When I hit a problem, I tend to work around it and keep going rather than attempting to fix it or search for a solution.

    Reply
  • Francesco Mantovani
    March 24, 2021 8:22 pm

    Regarding the Microsoft Support I suggest you to give the job to the Google search engine by typing:

    “CMEMTHREAD” “2016” site:support.microsoft.com

    In this case you will JUST have 819 links but you can reduce that if you go into Advanced Search and give a range of dates or eliminate words with the use of “-“.

    I do the same when I want to find a DBA on LinkedIn with a particular knowledge or certification in a particular region.

    Reply
    • Great. Now the server in question is running SP2 CU8. Has the issue been fixed in a subsequent CU, or one they’ve already applied, or included in SP2?

      Go ahead. I’ll open a bottle of wine. Keep me posted on how that goes for you.

      Reply
  • Stephen Morris
    March 25, 2021 9:34 am

    Interesting – reading between the lines perhaps a possible solution is to have a “virtual support currency” similar to the way that stack overflow allows you to post questions with a bounty, this might reduce the number of “Pony” questions & also provide a more nuanced request value than simple upvotes. You could get a certain number of points with your licenses, certifications, MVP, enterprise agreements etc etc

    Reply
  • danielle.paquette-harvey
    March 25, 2021 12:01 pm

    This is so true at so many levels! I was a software developer 10 years before becoming a DBA and I’ve seen my share of user requests! I’ve also tried to switch to Linux and failed lol.

    As for sp_blitz, I use it all the time and it’s great! If I ever need a new feature I’ll be glad to code it but so far, it works so well for me, I don’t need to.

    Reply
  • But, honestly, didn’t the Architect, based on basins of souls, point out that purrfection doesn’t go well with us humans?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.