Like a DBA batter’s box
While we were hanging out in between webcasts at Dell, we got to talking about stuff we used to do before maintenance windows. Things that brought us good luck, warded off blue screens and failed patches, and just generally made us feel better about working well into the early AM on a weekend.
You know, things that made staring at a command prompt until PING … -t tells you that your server is back up. Hooray.
We were also all pretty grateful that we don’t have to do that stuff anymore. If you want to learn how to not do that stuff, too, check out Brent’s 24hop session on career internals.
My personal ritual was to listen to Black Celebration and make sure my lucky server upgrade mustache was properly groomed. Not a lot of fuss. My Divining Disc of Pergamon has been on loan to a museum for a while, and apparently it doesn’t work the same via Skype, so I work with that I got.
What’s your ritual?
Whether it’s installing updates, rolling out database changes, new software builds for your in-house or 3rd party apps, or just turning OFF Priority Boost, what do you find yourself doing while preparing to make changes to your servers?
If you have a particularly amusing ritual, you might just win yourself a book.
Brent says: my favorite was a big steak dinner right before we started patch night. That way, even if things went to hell in a handbasket, we still had a good memory from that day.
27 Comments. Leave new
have a sizeable upgrade coming in exactly a week…all the normal rituals are out, since we start at 6am. So just backups and verify lots of coffee are on hand 🙂
Possibly a silly question, but can we watch Brent’s Pass session, or is it just a pdf?
Thanks
Lizzie – yep, that’s a silly question given that it’s on Erik’s blog post, but I’ll forgive you. 😀 Yes, you can watch the recordings when PASS posts them. I don’t know when that is, though.
Sorry- he did link right to it so I thought it was fair game.
Thanks for the blog post, Erik.
Update: don’t worry; I just whizzed through the pdf really fast, and it was almost like a recording.
My ritual consists of taking the Harley for a spin (is therapeudic and clears my mind) weather permitting (Otherwise I wipe her down and spend a few minutes sitting on her). Then, clean my office up and rid it of distractions. While doing this, I put the Rolling Stones on the stereo. Finally, grab enough Mountain Dew and cigs to give an army a caffeine buzz and nicotine high and settle into my chair for the long haul…
Some upbeat music is a must, a charged cell phone battery (and charging cable handy anyway) and the right assortment of chips, candy, and drinks. And of course, plenty of battle testing on my scripts so they handle first contact with the enemy without rolling up in a ball in the corner sucking their thumbs.
Weird how similar this is to prep for a D&D gaming session…
Oh, and after release, visiting http://www.chucknorrisfacts.com/ is a must.
My ritual for patching is log in early, prep for everything (set up everything I’d need for the process), then play WoW until the impact time while playing classical music. 🙂 If the maintenance is early in the morning, I’d get up early, prep for everything (usual), then do Pilates before impact time.
Database Team Checklist for Ritual Sacrifice to the SQL Deities Prior to Any Major Database Server Changes (a.k.a. DTC-RSSDPA-MDSC):
1. Check the planetary alignment. If it’s out of whack, set legacy application code on fire as an offering. While it burns, dance around it repeatedly chanting “Death to RBAR!” Douse the fire with a bottle of developers’ tears hopefully before triggering the smoke alarm.
2. Sprinkle the database servers with pixie dust.
3. Take a swig or two from a bottle of developers’ tears. Pass it around.
4. Initiate co-workers’ performance of the chicken dance to Rob Zombie’s “Dragula”.
5. Release a bag or two of unicorn farts.
6. Place gnombies at cubicle entrance to ward off any stray developers.
7. Light sage to ward off the spirits of the DBAs who failed before us.
8. Quickly douse the sage in bucket of developers’ tears before the smoke alarm goes off.
9. Oh yeah! Don’t forget to do the backups!
If all else fails, eat bacon… lots and lots of bacon…
In reality, we normally just load up on coffee, tea, energy drinks, and pretzels. 🙂
What do you find is the best method for extracting the tears of developers? I’ve tried foul language, air horns, a vacuum cleaner, code reviews, and the music of Britney Spears. All of which worked but not as well as hoped.
Cut the project deadline in half, then put them in looonnnggg administrative meetings that have nothing to do with their project. Ban use of phones/laptops during the meeting, and make them give presentations.
Did this come from a management book? I’m curious if my manager has read the same one.
it getsss the coffee and it doesss the thingsss… or it getss the rbar again…
In my experience, the best method I’ve found so far is to teach them set-based thinking… I know, I know… impossible, right? No! It’s not impossible! I’m living proof it can be done! … then have them review their own SQL code and their co-workers’ SQL code! So far, it’s been fairly fool-proof.. except for that one time… well, we don’t talk about that anymore…
My comment about my experience in getting developers’ tears is better when read in the voice of Morgan Freeman… 🙂
Isn’t everything? https://www.youtube.com/watch?v=NRKZh-2j4PY
I would don my Black Knight costume and chant “None shall fail.”
Well I make sure my top secret rollback plan is in place.
I also try to have a plan b and possibly a plan c.
I also keep trying to find ways to make it faster and more reliable to do the implementation.
If possible I have run through some kind of dry run (as circumstances permit) to get some idea of possible problems and if things are likely to go as I expect.
Get ready a bit before and clear my mind of other things if possible. Best if possible to focus on the one main thing at a time. Distraction can lead to errors but not always avoidable.
I try to make my own task hit list of what I need to do and in what order and notes.
My Maintenance Window Ritual:
1) Punish myself for mistakes I might make with push-ups (good to get it out of the way early).
2) Take a quick trip to the liquor store to purchase two bottles of whiskey. A premium bottle for success and a bottom shelf bottle of pure punishment for failure.
3) Power nap.
4) Show up early and start coffee, prechecks, benchmarking and preparation.
5) Do work.
6) Send final maintenance status email.
7) Go home turn ringer up to full volume on cell phone and consume the proper whiskey based on results.
8) Grab a few hours of shuteye and return to work.
mispelled the word RUM
I can’t go into specifics but it involves white chalk, candles, a sharp object, a small four legged creature, heavy duty trash bags, and a case of carpet cleaner for afterwards. It’s worked well for me so far, but I’m afraid I’ve jinxed it now and might have to step up my game.
Beer.
It’s not so much a pre-, but I never, ever draft my success email while the maintenance is still ongoing.
Put on my silver necklace and adjust monitors and keyboards til I feel like Link in The Matrix Reloaded.
Watch a good hacking/computing movie. Hackers, War Games, Swordfish, Mr Robot, Matrix series
Update resume, because it gets the fingers nimble and, you know. Might need it later.
After: celebrate with “touchdown” pose like Link in The Matrix Reloaded
No rituals here … put enough planning, preparation and testing in beforehand and you should be able to go into each change calm and confident. Plus having your body full of caffeine/alcohol/nicotine/developers tears probably doesn’t help your brain troubleshoot.
I know that the place I currently work is really good but if you make an unprepared change and it goes wrong then be prepared to be hauled over hot coals (metaphorically speaking as they banned this practice a few years ago)
Plus ALWAYS have a planned+scripted+tested backout plan for each step of the change – even if that backout plan is re-install and restore.
We did a major upgrade for our critical line-of-business app this year (both SQL Server version and app version) and by the time we were ready for production go-live, the whole process was so thoroughly scripted and rehearsed it went exceptionally well, but I am not above a little superstition. A bit of Wagner at the moments preceding cutover (Flight of the Valkyries from third act of Die Walkuere, of course), a recitation of Henry V’s speech before the gates of Harfleur to team members (“dishonour not your mothers!”), and then when the cnames change and logins are disabled, ritually quote Obi-Wan: “I have felt a great disturbance in production…as if millions of SPIDs cried out in terror, and were suddenly silenced.”
Lets see. I pray the night before. The day of I tell wife that I will be home late, then I get the developers to pray with me. Remind myself I don’t know what I don’t know and then convince myself not to fear the unknown unknown. I then get something healthy to eat (no burgers etc…) and after that, push the button baby.