Microsoft appears to have given up on patch documentation, and that’s kinda scary for a product that costs thousands of dollars per core.
Yesterday’s SQL Server 2019 Cumulative Update 6 launched with the worst documentation I’ve seen yet. It’s been steadily going downhill, but CU6 represents a new low.
For years, we’d complained that the hotfix articles weren’t documented enough, and they “fixed” that problem by giving up on the articles altogether, and just publishing a short summary:
For some bugs, these short descriptions are probably fine. However, some of them post more questions than answers:
This sounds an awful lot like a performance issue, not a security issue, and it doesn’t have nearly enough details. What features does it pertain to? Should all users be concerned, like folks who just run parallel queries? Or is it only relevant to a specific feature that isn’t installed by default? How can people know if this fix is relevant to them?
Sure, maybe short notes are okay for some updates, but if they are, they sure as hell gotta be correct. We’re now at the point where not only are we getting 3-sentence notes, but the notes aren’t even right.
To make matters worse, CUs have been having showstopper issues – but those showstoppers are buried down in the tiny print at the end of the CU, far after someone clicks on the download link, like this Cumulative Update 4 bug that talks about SQL Server not being able to start up:
To put this in perspective, Azure Data Studio’s closed issues and commits have way more details, and that product is free. Customers paying $7,000 per core for SQL Server Enterprise Edition are supposed to just take down production every 30 days and apply barely-documented updates without a clear reason.
I don’t think that’s okay. What do you think? Are you still patching every 30 days like Microsoft wants you to, or are you less likely to frequently patch given the lack of documentation?
67 Comments. Leave new
This is realy bad behavior from Microsoft.
I switched a while back to the recommendation from Microsoft to Update SQL Server with the CU as soon as it appears. Gladly i am a bit cautious and waited until the next CU is available.
Will be switching back to the good old behavior of installing a CU when it fixes a problem that i have and is safe to install if Microsoft going to keep up the poor documentation work they are doing.
I migrated an environment from 2016 to 2019 and CU3. After that no patch applied because I’m affraid of what can it cause.
In test environment I applied every single patch after that, but no courage to apply in production.
I didn’t know major companies were running 2019 in production. I would love to see that, but then patch/not patch becomes that scene in Deer Hunter with the game of Russian roulette. No patch and you’re risking dangerous bugs, patch and your risking outages. Some companies are no patch no service pack 2017, which is sad but safe from a certain perspective.
Why wouldn’t you run 2019 in production? It has been our for over a year and has good feature and enhancements.
Since 2017 and newer how do you handle the wait until SP1 is released since they no longer do SP?
Garry – that’s a great question. For me, it comes down to a rapidly evolving and not-well-documented list of bugs.
It’s been a really rough ride for folks who moved to 2019 in order to get performance fixes for scalar UDFs, for example: almost every CU has walked those improvements back: https://support.microsoft.com/en-us/help/4538581/fix-scalar-udf-inlining-issues-in-sql-server-2019
So that puts customers in the awkward spot of implementing 2019, seeing better performance, and then seeing it get worse again when they apply a CU. I can’t really recommend that a customer upgrade to 2019 to fix their scalar functions when Microsoft has been removing more and more of those fixes every 30 days.
Hi Brent – Has your opinion on this changed since your last post ?
we’re looking to upgrade from 2012 and are planning to jump to 2019, but the above has me slightly concerned.
Dar H – what has happened that you believe would have made me change my opinion?
Brent – time (I hear it heals all wounds 🙂 ) & the subsequent CU updates. Probably a naïve belief the MS have attempted to address the performance issues. Thank you Brent.
Dar – you mean like the very next update, Cumulative Update 7, which Microsoft released and then had to yank from production due to quality issues?
Hi Brent – well more so CU8, I thought all knowledge of CU7 was wiped from the archives 🙂 Hey thanks Brent.
I’m gonna need a little better than 50% success rate in order to recommend a product.
I’m glad you got involved Brent and have made your feelings known to Microsoft. If you hadn’t noticed the new 2016 SP2 CU14 has two mentions of KB articles that don’t exist. Your screen shot mentions TF 12240 and KB4497705. Go click on and you get a 404 page not found
Chris – ooo, yeah, I didn’t catch those. I usually click on most of them, and I skipped this round because I was so disappointed. Tough to see that it’s gotten even worse!
We’ve got one server here that could pay for a good tech writer for a full year, all by itself.
I agree. Blindly applying updates is risky if you don’t know what is being updated. I want to know what is being patched and how it may affect my systems. No, I don’t patch monthly. I offset updates. In odd numbered months (Jan, March, May, etc), I update my DEV servers and run for the month to ensure no surprises, then update production on even numbered months (Feb, April, June, etc).
@Brent- Thank you for raising this issue. It is indeed critical to have confidence in what’s being done to our systems. And thank you for including the reference to ADS; instead of just complaining, as many do, it’s wonderful to have an example of desired behavior.
@RobertTolfree- Your even/odd-month rollout is a great suggestion for mitigating some of the risk here.
Tim – thanks! I know I’m gonna burn some bridges at MS, but I’d already raised it a couple of times on social media, and it’s falling on deaf ears.
I’m never going to be one of the first people to install a CU. It’s looking too much like beta testing these days.
HappyDBA – well-said.
There’s a reason he’s a happy dba.
WHAT? Microsoft actually helping us? The only time I install any updates is when something is not working and it requires a call to Microsoft support. Then I quickly get the latest update and then see if this fixed the problem before I even call. Last resort is to call Microsoft for help. Most of the time I will check first with Pinal Dave, That Brent Ozar guy, Paul Randall, or even google before going to Microsoft for help!
Brent – I fully agree with you on this. Also agree with you @happydba – I will always wait a while to let the major bugs come to light before installing any of these updates.
Camaro322hp – thanks, and good to see you on the blog! I was debating whether I should write this as a post, or do it as a rant on the stream, heh.
I think you got Pedro’s attention yesterday
Oooo what happened yesterday? I missed it!
This thread:
https://twitter.com/BrentO/status/1291105388524261376
https://twitter.com/ChrisAVWood/status/1291104967701430277
Scroll up to the top of it and read down to see most of the tweets.
I have stopped my monthly patching cycle primarily for this reason. I have 16TB database that is mission critical. Even with HA, I can’t take the risk of node going down because of some obscurely (un)documented CU.
Tony – yeah, with HA scenarios, I’ve recommended that clients patch the secondary, run it for a week or two, then fail over to it. If anything goes wrong, fail back over to the other unpatched node – but don’t patch that one for at least a couple of weeks, lest you find a problem on the new CU.
We’re still on 2008 R2, though next year will likely be when we make the jump to a modern version. I imagine that we’ll stay at least a couple CUs behind once we upgrade. Even then, we’ll still probably have to do 3-4 CU jumps, just to space things out – with patch notes that bad we’ll need to be extraordinarily diligent with regression testing.
I agree. Their documentation has gone downhill over the years with the updates. We run everything from 2012 to 2019 in prod. We do a 30 day patch cycle as well as we can afford the downtime, not 24/7. Don’t look at it as burning bridges. Maybe they looked at the site hits and decided to judge their documentation based on that. Calling them on it is exactly what a good customer does.
What about windows security updates? Does that fall in same category/disappointment. I just joined a new company and found that IT have automated windows updates every Tuesday night and sql sever being rebooted. No communication being sent out.
I’m in the same boat. Weekly reboot cycles for security patches with no communication.
Dan, I am having to explain them why we shouldn’t reboot so often. Regardless this article from Kendra has been helpful to explain why
https://littlekendra.com/2016/07/28/should-i-automate-my-windows-updates-for-sql-server-dear-sql-dba-episode-10/
Microsoft, thru both the Cloud and if you didn’t disable the Error and Usage Reporting, will see a lot more than they did when we were all in our Data Centres. If you were in the Cloud you get these updates automatically
I have 600 Azure SQL Databases and I’m very concerned every time I get a “Planned Maintenance Notification for Scheduled Maintenance to Azure SQL Database” that this is going to be the time they break something that impacts us. So far as I know they almost never provide any documentation on what is in these updates. Azure customers are the canaries in the coal mine for 2019 and the CUs. Maybe we’re just lucky (or we’re conservative about which features we use), but in 3 years they haven’t hurt us with one of these updates. But maybe the next one…
For instances in VMs, we haven’t moved past 2017 yet, and even those are only for things like monitoring software repositories, load test controllers, or Octopus tentacles, not for customer facing applications. I keep those at least one CU behind just in case.
Bug fixes and performance improvements, Brent. Bug fixes and performance improvements.
These are my thoughts on the “Bug fixes and Performance improvements” type of change logs that mobile devices have made popular.
This means that you have a communication problem with your users/customers.
As a software developer, this has always been a pet peeve. If you think about it, not showing a change log is insulting to your developers, and reduces say 1 month worth of work into “Bug fixes and performance improvements” … 🙁
People will think you are doing nothing with the app, and that there is no transparency. People will think there is no plan to improve the app. People will think that the app may as well be dead.
You got me there. I laughed out loud, hard, hahaha. Nicely done.
Is M$ *trying* to promote MySQL?
My employer chose Azure SQL Database before my arrival so that is the world with which I am most immediately concerned. I have become accustomed to changes in the Azure portal which show up today but were not there yesterday. And every now and then I report on a behavior that I find out is an actual bug so I can feel the pain expressed in other comments. Reading those comments regarding current SQL Server version and CU “features” takes me back to the time of SQL 7.0 on Server NT4.0 when patches/updates were deferred as long as possible due to fear and loathing of the “blue screen of death” that was so common when those patches were applied.
Yup, noticed that as well. That second item with object-level locking optimization on secondary replicas redo is especially interesting for us, and there’s just zero additional information except that one sentence. Disappointing.
They should be ashamed of themselves! If SQL Server had a simple licensing model and correct price, this would have been less objectionable.
I have been an early adopter of CUs for years now and *knock on wood* have never been burned. However I totally agree that the lack of documentation isn’t going to build trust with admins and users. Just wondering if Covid 19 is somehow related to that decrease in documentation efforts.
So you’re saying “works on my machine,” eh? Interesting. You might like this:
https://blog.codinghorror.com/the-works-on-my-machine-certification-program/
I didn’t see anyone mention PCI compliance. Business managers know they need that box checked. If the software isn’t at ‘lastest version’ or some such language, the vulnerability scans will complain. IT staff take an ‘our hands are tied’ approach and expect every CU to be applied as soon as practical, with a minimum smoke test in dev.
I would disagree with this comment. I was part of PCI for years and as long as you have your process defined in a policy for your company they cannot really dock you. We had defined 30 days for CU’s marked critical and 90 days for normal CU’s. No governing body can force you to apply something that could be harmful without adequate time to test in other environments.
Brent – is there a blog or resource out there that does the leg work on translating / summing these updates up? Maybe gives advice on risk level for each CU? This makes me hesitant to move prod to ’19. Have you written a high-level, comprehensive blog somewhere on adoption for ’19 and some of the pitfalls etc.? Know I’ve seen a lot of your stuff outlining the new features.
Remington – well, given that Microsoft isn’t documenting them, I don’t know how we could possibly assess the risk, right?
In terms of adoption and reasoning, I have better news though – check these out:
https://www.brentozar.com/archive/2020/06/sql-constantcare-population-report-spring-2020/
https://www.brentozar.com/archive/2019/01/which-version-of-sql-server-should-you-use/
Thanks will check these out.
What I meant was someone testing these after release and summing up any findings etc. Again high-level. Sounds like you’re saying they wouldn’t know what to test given how incomplete the documentation is getting.
Exactly – with a lot of this one-line “documentation”, we don’t even know what’s changing.
Brent, I see that 13594832 fix area was updated from SQL Security to SQL Extensibility. Someone from Microsoft must be listening a little. 😮
I can remember a time when ALL of the fixes in a CU or SP had KB articles (wait…wasn’t that just a few months ago?). Next Microsoft will be introducing undocumented features!!! Things we will never know about!
Wheelmans,
Microsoft mentioned that people didn’t want to click on another link to see a KB for most simple fixes. Obviously that wasn’t any of us commenting here.
I also see for a second time SSMS has had a security fix that references a previously released build and points at the release notes that of course don’t mention the security fix
Microsoft tools team have now updated the ssms 18.6 release notes.
https://techcommunity.microsoft.com/t5/sql-server/changes-to-kb-articles-used-for-cumulative-updates/ba-p/1597145
Yep. Translation: “you caught us, we didn’t tell you about it, and you found bugs in it – but, uh, it’s a feature and we swear people love it.”
Okay. Me, I prefer accurate fix notes rather than typos, but…shrug
It does prove that documentation is the last step and has been lacking at times for many years. Was this a reason for nobody clicking through to see the details.
I still don’t immediately apply a patch unless: (a) it’s had a few weeks to bed in with no reported issues; or (b) it’s to fix a specific and pressing problem.
I know this is early days for CU7 but does it appear to be better than the last few 2019 CU’s? Do you have a good base CU to work with? I have a project to move from SQL 2012 to SQL 2019 and we are on CU4 so I am wondering if it would be best to move up from there.
Thanks
Chris
Chris – great question. I don’t manage production servers on a day-to-day basis myself, so unfortunately I just can’t answer that. What I’ll say is that I don’t often recommend that clients move to 2019 right now.
I saw your comment yesterday on the inline UDF fixes so I was hoping that you might have an opinion other than don’t go to 2019 yet.
Thanks
Chris
You probably all know that 2019 CU7 has been pulled and is not recommended anymore because of snapshot issues. The details, as they are currently, can be found at https://techcommunity.microsoft.com/t5/sql-server/cumulative-update-7-for-sql-server-2019-rtm-removed/ba-p/1629317
Have seen that. I was trying to think back on how many CU’s have now been pulled after release for 2017/2019 but that would take too much researching 🙂 It’s been at least a few though. Which I don’t like one little bit.
The latest 2016 CU2 CU has 2 fixes for AG’s without a full KB article. I feel that one of these needs a KB because if you have an AG database failure you need to know how to recover the database. We had a situation 2 years ago where a number of databases weren’t synchronized so restored the databases. It happened again the following day and a Microsoft engineer said we could restart the Primary and this fixed the problem. If a situation is fixed for AG’s then that solution needs to be in a KB.
Something weird happened when i executed same query in 2012 and 2016 SP1. The output is totally different in fact i am getting wrong output in 2016 SP1.
Here is the code which generates minute wise datetime for entire month.
Does anyone know why?
— number of calls with predefined start time, end time and interval
DECLARE @start_time DATETIME; — starting from here
DECLARE @end_time DATETIME; — until the time is under this value
DECLARE @interval CHAR(3); — interval definition (e.g. day, minute etc.)
DECLARE @increment INT; — interval increment
DECLARE @loop_time DATETIME; — variable used in the loop
DECLARE @times TABLE(start_time DATETIME);
SET @start_time = ‘2020-10-01 00:00:00’;
SET @end_time = ‘2020-10-31 23:59:59’;
SET @interval = ‘mi’;
SET @increment = 1;
SET @loop_time = @start_time;
WHILE @loop_time < @end_time
BEGIN
IF @interval = 'yy' SET @loop_time = DATEADD(yy, @increment, @loop_time); — year
IF @interval = 'qq' SET @loop_time = DATEADD(qq, @increment, @loop_time); — quarter
IF @interval = 'mm' SET @loop_time = DATEADD(mm, @increment, @loop_time); — month
IF @interval = 'dy' SET @loop_time = DATEADD(dy, @increment, @loop_time); — day of year
IF @interval = 'dd' SET @loop_time = DATEADD(dd, @increment, @loop_time); — day
IF @interval = 'wk' SET @loop_time = DATEADD(wk, @increment, @loop_time); — week
IF @interval = 'dw' SET @loop_time = DATEADD(dw, @increment, @loop_time); — weekday
IF @interval = 'hh' SET @loop_time = DATEADD(hh, @increment, @loop_time); — hour
IF @interval = 'mi' SET @loop_time = DATEADD(mi, @increment, @loop_time); — minute
IF @interval = 'ss' SET @loop_time = DATEADD(ss, @increment, @loop_time); — second
IF @interval = 'ms' SET @loop_time = DATEADD(ms, @increment, @loop_time); — millisecond
IF @interval = 'mcs' SET @loop_time = DATEADD(mcs, @increment, @loop_time); — microsecond
IF @interval = 'ns' SET @loop_time = DATEADD(ns, @increment, @loop_time); — nanosecond
INSERT INTO @times(start_time) VALUES (@loop_time);
END;
SELECT * FROM @times
For general support questions, use a Q&A site like https://dba.stackexchange.com.
How can I unsubscribe to this link?
Click on the unsubscribe link in the emails you get.