I work with clients with a variety of change management practices— everything from, “I just email the team when I’m going to change something” to “I need to submit formal changes each Tuesday by 2pm for weekly review for the next approved change window.” And occasionally, “Just Do It.”
[Photo credit]
When planning changes, successful teams typically cover these questions in one way or another:
- What are you doing?
- Why are you doing it?
- What steps will you be doing?
Sometimes people even include, “How do you undo the change?” I’m a huge fan of that— if there’s no way to undo the change, that’s important to know! It’s also important to know when it’s dead simple to undo the change.
There’s one question which almost nobody asks, however, and it’s one of the most important questions.
What’s the Worst Thing That Could Happen?
With every change you do in production, ask what realistically is the worst case scenario. We’re not talking about space trash striking the datacenter and causing company-wide failure, we’re asking the question, “What specifically would fail if this change went really bad?”
This is a simple question to ask, but you learn a lot from answering it. It’s a much more informative question than asking something like “is this low, medium, or high risk?”
Why this Question Saves Your Business
The “worst case” question gives you insight about how you run your business. It shows you what levels of risk you regularly take. It also shows you how well your team understands the possible impacts of changes.
Asking this question also builds your teams in an interesting way. When I first started tracking the “worst case” scenario for changes, I found that I needed to talk to my colleagues more. I could easily tell that a change might produce incorrect data in a given table or cause a problem in a given area of data processing, but I wasn’t sure exactly what that might mean to the end users. To work out that question I needed to work with others. Sometimes I got information from our developers, sometimes from our level 2 customer support team, and sometimes I worked out the information with business users.
Great Change Management Builds Documentation the Easy Way
Asking “what’s the worst thing that could happen?” was a natural and easy way for my team to document the role of our database servers and points of impact on the business when they went offline. As we collected this information in changes we also documented it in our troubleshooting guides.
My team went from having a vague idea of the level of risk of changes to have a clear, documented list of the roles of all our databases. We worked in an extremely active environment and it was always safe to assume we didn’t have a complete list of impacts: we had to keep communicating. But this added a huge efficiency to planning changes, responding to incidents, and also demonstrating the value of our investment in data management.
Don’t wait for Management to Demand Better Change Management
Sure, improving change management can be a great win for managers. But if you’re reading this and you’re a database administrator, improving change management can be an even bigger win for you.
If your team’s change management can be improved, you have a great opportunity. Start with your own changes: try out a new method of documenting your changes and getting them approved. Use it to iteratively improve your documentation of the system. Show your management what you’ve been doing and the value it provides, and soon you’ll see the process take off. Know what you just did? You just contributed to your business in a huge way.
11 Comments. Leave new
One additional piece of information I always like to collect on Change Managements documentation is “Who will the change affect?” meaning if you have to reboot, or if there is a failure in the change who will be most impacted if the system is down temporarily.
Ah, yes, absolutely. For the most part I’ve covered this in the “worst case” scenario, but I can absolutely see calling it out separately when you’re worried people might not hit it. Thanks, Robert!
Thanks for this, Kendra. It’s very apropos, as I’ve been thinking a lot about change management recently.
A question for you (and other readers/commenters): what kind of change management software/processes are you using? I’m looking for something more than just CVS/Git/etc, as I’d like to include management for configuration changes and hardware/networking changes. Any recommendations? I’ve seen suggestions online to use CVS/Subversion/Git/etc, and I’ve seen some suggest something as simple as a wiki for the config changes (with one system per page and all changes documented on that given page). I just want to be sure I’ve got all the options before I decide what we need to do.
Thanks!
I have used a bunch of different applications successfully. These include:
* A home-written application written by a developer who eventually became the head of software development. It was basic, but met the company’s needs for many years! (Several of my customers also use the “home grown” solution and are happy with it.)
* Sharepoint based applications
* Custom “ticketing” applications that are sold by vendors for incident and change management. I used one called FootPrints for a while which was really quite nice and very customizable– no idea what it costs, though!
When first starting out, I have found that super basic things work pretty well, as you say — wiki’s can do a lot!
Thanks for the ideas, Kendra!
To start with, if you have the developers, I would say homegrown. I have actually developed a PCR (Production Change Request) system in the past. It is a simple application really. We used email notifications so the “approver” would know when a PCR was submitted and the IT Manager had to approve before implementation, unless it was an emergency, then the PCR was usually done after the fact.
But for SOX we had to take it one step further and if it was a “critical system” (i.e. payroll, financials, etc.) we had to also get the CFO approval before executing the PCR. All done through a website and email.
One additional critical question that I like to ask is “How will you know this change was successful?” I really agree with you on how to gather the interactivity with other systems to prepare for the change and plan contingencies. If I wanted to share horror stories about my current employer, we’d be here all day. I’ve gone way above and beyond my pay grade to try to improve our processes for Change Management, and I totally love your approach to resolution with or without management buy in… leading by example is so important, especially in immature organizations.
Oh, that’s a great point, Jason!
Validation can be trickier than it sounds, too. For database changes I like to have a validation test that’s run from a separate connection that processed the change at minimum– I know of one case we’ve seen where a transaction was left open in a change and validation appeared to work, but it was because it was being run in the session with the open transaction. (Ouch!)
Thanks Kendra for “evangelizing” us DBA to Change Management: I am already “converted” to ITIL change management (and in fact I have become Change Analyst) and I would like to had a couple of comments to your nice article.
Which change management process should one implement?
Why should one reinvent the wheel? There are several streams of change management, most of which are perfectly suited for IT. We have chosen ITIL which is some sort of “Industry Standard” for IT Service Management (a subpart of which is Change management), so you’ll find manuals, blogs, trainings, certifications, etc. Of course one can adapt an existing change management process to its own needs and to the Company’s maturity and it is still easier and better than starting from scratch.
Info about ITIL can be found here: http://www.itil-officialsite.com/AboutITIL/WhatisITIL.aspx
Which tool should I use?
First of all, an SVN is not a change management tool in itself (I am answering Gavin here), but it is necessary to store code and files and will allow a Rollback or will be part of a Disaster Recovery Plan. Changes can be in architecture, in server configuration or in application configuration, it can also be applying the monthly Microsoft’s security patches. There are dedicated tools that can be expensive and to start without a huge budget you can look at free tools (a nice review can be found here http://www.itframeworks.org/wiki/ITIL_tools_(free_software) , just look for “change management”) or maybe you can build yourself a SharePoint site with a change calendar, a change request form and a simple Workflow for approval (all that without writing one line of code!).
Why should I implement a Change Management process?
You were exhaustive about reducing risk and about improving documentation, which are two very important points. I would like to add that implementing Change management allows reducing costs, both real and potential ones. You reduce real costs by reducing downtime because the process will try to address changes in a more industrial way, like consolidating changes during planned change time frames, and cleverly planning changes that may interact or interfere; you reduce costs also because Change Management process will help you improve the change quality because it requires thorough testing before implementation , so you’ll need less “changes to repair previous changes”. You reduce potential costs (i.e. service disruption) because you manage and reduce risk.
Another reason is to improve IT maturity: not only you improve people professional capacity (follow a process of tests, redact change documentation, ability to manage risk, etc.) but you’ll also may get better organized, you’ll get less slave of user urgencies by defining SLAs and by defining what is the “technical” meaning of urgent.
Can I put a Change Management process in place all by myself?
Sure, at least to start with and to understand what Change management means. But if you want to have all the benefits described above, you’ll need to involve the people you work with and for that you’ll need to have a convinced sponsor in your IT Manager. In the company I work, we have implemented ITIL since a little less than two years: my manager wants everyone in IT to improve its commitment to ITIL and its maturity and asked me to define KPIs to measure this.
I am sorry for the very long comment, I hope this will help some of the people that are interested in Change management.