A Gentle Reminder That Corruption Stinks
Seriously
DBCC CHECKDB (YourDatabaseHere) WITH NO_INFOMSGS, ALL_ERRORMSGS;
Brent says: normally, we don’t post client screenshots on the blog because it’s hard to sanitize the sensitive data. This time, not so much. Storage took care of this for us.
Free, 3× a week
Get my new posts by email
Three posts a week, plus a Monday roundup of the best database news from around the web.
I see your client has found the new SQL Server burn-before-reading feature for truly sensitive data.
So, begging the question…what was root cause? Memory bit flipping, bad SSD, something else?
We know the root cause on this one, but I don’t think the client would be fond of us sharing that one out loud. 😀
Since we don’t know who your customer is, does it matter? Or perhaps talk in terms of how you would reproduce this error to prove the source.
This would still be hard. If it were easy, I’d have talked more about it.
Awe common I won’t tell anyone :).
It’s actually a work of art. This should be put on a t-shirt or something. 🙂
PS: Thankful it’s not *my* data….
Not to be picky but who says this is corrupt and not just someone sanitizing the data. Did the DBCC actually show the error and if it did, which one. I assume it did or why the blog post but just checking.
I was wondering the same thing. Are there error messages?
There were about 50k lines or error messages, so, yeah…
Larry – well, kinda. Some of the databases wouldn’t start, ha ha ho ho.
My Blackjack cheat bot in action!
Winner Winner, Chicken Dinner!
We went through this a couple weeks ago with a 1 TB critical production database. It was not a fun experience.
What can I learn from this post? What is the takeaway, besides “corruption is bad” (which I’m already aware of)?
DBCC CHECKDB syntax 😉
What can you learn? Have good up to date backups and don’t play with DBCC Writepage! 🙂
I’d say the consistency check worked fine. This is the most consistent data page I’ve ever seen.
I think the ascii cross symbol adds a nice touch. Have you tried running DBCC DBEXORCISE? (“THE POWER OF CODD COMPELS YOU!”)
It reminded me of the graveyards in Zelda, kind of.
Lesson 1. Don’t be complacent
Lesson 2 Check that you are CheckDB’ing
Lesson 3. Now where did I put those backups – well at least make sure you can access them.
Thanks Erik.
Thanks, Mal! I still see a lot of people running CHECKDB, but not alerting if it fails. That’s no fun!
Question: How to test that the alert actually works, given that in a “normal” situation there are no errors.
Download a corrupt database (or corrupt one yourself with a hex editor).
You have a couple options. You could download a hex editor and try to corrupt a database on your own, or you could head over here and download a sample corrupt database. If you follow along with the challenge, you can see a few different kinds of corruption that can strike, and how to fix them.
I would love to hear the generic recovery steps that were taken for this .. I presume it was restore from sound backups (if you knew which backups were sound).