Question 1: How do you feel about Run Book Automation for databases?

In theory, I love Run Book Automation. In practice, the time it takes to implement a successful solution like this (with today’s real-world tools) outweighs the time savings of automation. One of the things I love about database administration is it’s never the same day twice – and that makes for tough automation.

I’ve had this discussion with a few people around PowerShell. There’s a growing community of PowerShell scripts, and some people think we should be able to automate ourselves out of a job. However, when I look at the staggeringly large community of T-SQL scripts, and I look at how we’re all still working, that doesn’t add up for me. Just because the scripts and tools are out there doesn’t mean our jobs are over. It takes time to save time, and sometimes it takes even more time than you save. The ROI isn’t there for me in Run Book Automation until you get to the enterprise where you’ve got thousands of SQL Server instances. Even there, you’ll need full-time staff to manage the automation solution, and you’d better be *really* certain it’s going to work before you automagically change a SQL Server’s configuration.

Like Paul Randal says, the answer is always It Depends, and it’s hard to code all those rules into a tool. Heck, we’ve got simple backup and recovery tools that aren’t even able to handle every permutation of error messages and configuration problems. Doing Run Book Automation scares the pants off me. (Literally. I’m in bed with no pants on. It’s 10pm on a Sunday, though, cut me some slack.)

Question 2: Do you have multiple copies of the same database?

As Joel suspected, my answer is yes, heh. My needs, though, only revolved around homegrown applications. We had several in-house-built apps that we needed to sync between production, QA/test, and development environments. We did not do this syncing (either data or schema) with third party applications – we left that up to the third party apps to manage. Whenever we touched a third party app’s database, we voided the warranty, as they say.

Question 3: If yes, how do you feel about automated patch and configuration management of databases?

In our homegrown apps, we had two separate problems: we needed to migrate data from prod to QA to dev, and we needed to migrate schemas from dev to QA to prod. Whenever we did either of these, it was managed in combination with the developers. We never migrated data or schemas in a vacuum: we did it hand in hand with code changes by the development team.

Even when we had a gold standard schema, we couldn’t handle variations in an automated fashion. If a schema strayed from the gold standard, it’s because somebody changed it – whether it was an added index, an added field, additional child tables, whatever. These changes are made by humans, and we wanted a human to get involved to decide what to do next. We never wanted to lose someone’s schema changes, and we never, ever, ever wanted to lose someone’s data.

I love automated patch and config management for Windows servers because developers don’t put time and effort into Windows configuration settings. People just accidentally (or purposely) break configurations, and we want to get them back to a known standard ASAP to prevent hack attacks. Databases aren’t as easy to change, so when changes are made, we need to be more careful about automagically undoing them.

Yes, we have multiple copies of the same image/core db. In one environment, due to security and confidentialiy of client HR types of data, we have to do this so that client data isn’t comingled. So each project is it’s own data and we start with a common schema and default data but from there it can customized by the implementation teams.

The change management is a bugger with this one. So when we release updates or bug fixes, it is a major headache because we may have as many as 400 databases to test and update. What I would love to see is some sort of VSS within the DBMS itself. We have tried VSS whereby the developers script the objects but it just doesn’t work when you have a large number developing db objects.

We have one DB that is just for the developers to code against and test with but we have an image database that is read only and the Sr DBA can apply scripts/changes to it after all the kinks have been worked out. That in essence is our gold standard (or build version as we call it).

As for change management, I prefer automated not automatic. It’s that whole trust but verify thing.

But stating that, you just answered one of the questions You admit you have answers to all three, and that implies that your answer to #2 is “Yes.” Of course, I’m sure you’ll put a lot more detail behind it later, but I kind of thought that was funny.