I have to deliver a lot of bad news. It’s pretty much my full time job. Nobody calls us when things are going well. (Well, they tweet us, because that’s free.)
Sometimes, that bad news is very dangerous – especially for the company as a whole, or its customers.
In today’s consulting line, I’m working with a group of developers on a performance issue. We’re using sp_BlitzCache™ to check out the top resource-intensive queries on their system, and figuring out how to make them go faster.
The Conversation
Me: “It looks like this query returns the customer’s email address, password, birth date, and address.”
Larry: “Yeah, it’s the profile page. We show them their account.”
Me: “So this is stored in the database, and it’s not encrypted.”
Larry: “Yeah.”
Me: “And we’re on the development server that everybody in the company has access to, right?”
Larry: “Everybody.”
Me: “Okay, let’s stop for a second. I have to cover my own butt. Gimme a few minutes to document this, and I’ll include it as part of your written findings. I know you didn’t call me in for a security review, but I’d be a bad consultant if I didn’t put that in writing.”
What That Line Does
Putting a big security risk in writing is a career-limiting move.
This line helps you defuse that bomb.
Nobody wants to have something like this in writing – especially a written document that gets forwarded up the management chain. They’re going to want you to stop writing and just ignore it, but this line establishes that it’s not really your call to make. Nobody wants to be bad at their job – but ignoring huge, dangerous risks would make you bad at your job.
What Happens Next: The Easy Way
Larry: “OK, cool.”
Sometimes the rest of the team isn’t happy about the risk either, and they’re dying to have someone else champion the cause.
And you know what’s funny? The developers and sysadmins are often quietly high-fiving each other while you’re writing it down. But if there’s a manager in the room…
What Happens Next: The Hard Way
Moe the Manager: “Wait, that’s not why we brought you here. Let’s focus on the problem at hand.”
Me: “I totally understand. This will only take a few moments for me to document, and we can work a few extra minutes at the end of the day. I won’t bill you at all for this. I just have to write this down to cover my butt.”
Moe the Manager: “I can’t really have something like this in writing.”
Me: “Believe me, I totally understand why you’d say that. If I was in your shoes, I’d say the exact same thing. But I have to protect myself – say for a second that another Target, Anthem, AshleyMadison, or whatever happens here, and it gets out that I was the database consultant, and that the personally identifiable data was out in the wide open, and I didn’t tell you about the risk.”
Moe the Manager: “We won’t tell anybody.”
Me: “You say that now, but if you got hacked and the personal data got out, Curly the CEO would tell your lawyers to point the blame at me. They’d say I was a bad consultant because I didn’t alert you about this risk. I have to include this in my written findings to cover my butt. You can feel totally free to ignore it if everybody else in the company is comfortable with that risk, but I have to put it in writing or I’m a bad consultant.”
I’ve actually lost a couple of clients when this line went downhill, but you have to be careful with your own reputation. Nobody wants to hire the DBA who was on duty when one of these incidents went down.
For more fun, read more of my favorite consulting lines.
7 Comments. Leave new
As a consultant myself, I see this scenario quite often with many of my clients. Another common response that I have experienced is the client saying its ok to keep the data unencrypted because it’s DEV or encrypting the data is too complicated for staff so they chose not to secure it. In fact, I literally had someone ask me yesterday “What is the purpose of TDE.”
It’s funny that sometimes everybody is colluding to hide such issues. Even going so far as to fire you because you would put it in a document and create evidence. That borders on criminal behavior.
Collusion, more often than not, is criminal.
I’m actually in the process of leaving a company due to unethical practices and having my recommendations to improve said practices go unheard. I’m not willing to be the DBA on duty when something terrible happens, and it’s only a matter of time unfortunately. I’ve even compiled a full report with detailed recommendations supported by industry SME’s and best practices which I will be mailing to corporate. Sometimes you have to put your reputation ahead of an opportunity. Good post!
I respect your integrity and agree that you do the *only* right thing by carefully documenting the issue. If you have a responsible company hiring you, they just may ask you how to remedy the open SPII issue.
I ran in to such a bomb at my previous job. Sent over to a business unit and found all sorts of PII regarding minors (SSNs, medical conditions, etc.) in an unsecured Access database. Immediately went to the IT manager: we didn’t previously know this database existed, much less was such a risky data store. It took a couple of years, but it got re-written in to SQL Server, encrypted and audited.
Always CYA, it’s the only way to protect yourself and protect your client.
I’ve had a similar issue when I was a junior DBA at a company. The fact was that the developers wouldn’t listen to the things I saw like the encryption issues and there were a lot.
At the end I wrote a memo to my manager who, as you can guess, took the hard way and didn’t want to change anything.
At some point shit hit the fan and a consultant came in. He saw the exact same things I had been telling everybody and suddenly everybody listened to the consultant.
That was the point I went to look for another place to work because I wouldn’t want to work in a place where they would ignore all the security issues and only listened to consultants.