Blog

Apple retail stores have a Genius Bar where you can get tech support.  I walked over to the Apple Store on Lincoln Road (damn, I love living in Miami Beach) and bellied up to the bar.  The bartender, wearing a black t-shirt with the label “Genius” on the front, helped out a couple of customers.  While I waited, another Apple employee noticed me waiting, asked if I had a reservation, and then helped me make one with a nearby iMac.  Wow.

I sat watching the Genius tend to his customers.  He was helpful, patient, and fair.  One of the laptops needed serious repair, and he explained the customer’s options as to how to back up their data before sending in the laptop for service.  The customer was clearly in over his head, but the Genius helped him make the right decision without trying to fleece the guy for the $50 backup charge.

When my turn came up about ten minutes later, the Genius looked at my Macbook Pro for less than sixty seconds, agreed that the battery was hosed, and went into the back to fetch me a new battery – free under warranty.  He installed it, made sure the system recognized the additional charge capacity, and then sent me on my merry way.

Damn.  From here on out, I’m insisting that all of my relatives get Apples.

Yesterday was Take Your Sons and Daughters To Work Day, and I got roped into leading sessions. I ran a presentation on desktops & servers: we opened up desktops, explained what the parts did, showed off a couple of servers, and finished things off with a round of LAN games. We were a big hit with the kids as well as the parents because evidently ours was the least boring and most interactive session, and I took pride out of that.

I learned some lessons, though, and I’m passing those along here in case anybody else needs to run one of these sessions for kids:

10 kids is a great group, 16 kids is too many. The 16-kid groups quickly devolved into screamfests, and we couldn‮t get everybody to stay quiet long enough to learn something. The 10-12-kid groups were more manageable – 2-3 kids per volunteer seemed about right. Remember, we‮re not professional teachers – it‮s harder for us to manage this number of kids.

Know how to get problem kids out fast. I had a kid try to slice his arm open with a stick of RAM, and then tried to jam a CPU into his skin to see if it would leave a mark. He was actively causing disruptions with other kids, trying to get them to do the same thing, and I had no idea what on earth to do. It was an unsafe situation for him as well as the other kids. Looking back, I wish I would have asked him to step into the back of the room and had the chaperone go drop him off with his dad, but since it all happened so fast, I didn‮t know what to do. Thank God he didn‮t break his skin, because I‮d have freaked out.

The chaperones need to be trained, and then be actively involved. We had some chaperones that sat in the back quietly while the kids yelled and screamed, and we had other chaperones who actively helped to keep the kids under control. We had much better results with the latter.

Don‮t let parents leave with the kids during breaks. During lunch and the snack break, the parents took kids off to various places, and they didn‮t all come back at the start time. As a result, we had to manage 10 bored kids for 15-20 minutes. We couldn‮t start the lessons because we had to wait for all of the kids. The kids got uncontrollable while they sat and waited for the latecomers. An alternative would be having the chaperones running some kind of activity until all of the kids got together again.

We need separate tracks for 8 year olds versus 12 year olds. 8‮s are old enough to get something out of it, but the material has to be completely different than what you present to a 12 year old. My 11-12 year old folks wanted to know more and more and more, but the 8-10 year olds needed shorter, more visual lessons.

It’s 7:30 AM, and I’m sitting in an empty conference room. Why? Because Mike Culver from Amazon Web Services is about to present sessions on S3 and EC2, two of my favorite upcoming technologies. I’ve been getting more and more involved with virtualization over the last year – Southern is running 2/3 of their Windows servers under VMware – and Amazon Web Services attracts me for its tie-ins with scalability and virtualization. For the life of me, I can’t figure out how to use the technologies at Southern, but I’m still looking.

There’s actually a keynote starting shortly in another room, but I’m already camped out here in Mike’s conference room just to ensure a good seat in here.

Update after the sessions: Wow, this guy was great – funny, informative, and motivational.  Most of the attendees were interested right away, and asked great, pointed questions.  I didn’t learn much, but seeing EC2 in action made me abandon the rest of the day’s sessions and start playing with EC2 instead!  I’m curious to see if I can get it working as a low-powered DRP SQL Server using asynchronous database mirroring.

Tara Hunt blogged about the Boutique Generation, saying:

“You may be a member of the Boutique Generation if you:

1. Enjoy shopping at your local pharmacy, grocer, clothing store where
   you know the owners and feel yourself going out of your way to give
   them business even if their prices are higher because you prefer to
   have the personalized service over cost reduction
2. Enjoy shopping at specialty stores, like Cheese Emporiums,
   Perfumeries, Tea Shops, Sephora (drooool), etc. that carry a single,
   narrow type of product…”

The list goes on, and as I read through it, everything really rang true. It rang so close to home that I had to stop myself and ask, “I thought everybody felt this way?” I guess not, though, because when I drive past Wal-Mart parking lots, they’re almost always packed. That place is a zoo, and I can’t stand to step foot in there because it’s so utterly impersonal, so bland, so assembly-line. It’s not that I’m financially loaded and I can afford to shop at Saks – I can’t – but at least I can choose Target, where they offer more products with the types of personal stories that Tara’s article touches on.

Tara’s article is a real thought-provoker. The only thing I would suggest is that the word “generation” probably doesn’t apply. For example, my mom exhibits these exact same personality traits, and by definition, she’s a member of a different generation. (Don’t take offense to that, Mom.)

Technorati Tags: boutiquegeneration

Ben and I finishing the Run for Miracles

I ran a 5k this morning in Delray Beach with Ben, the guy from work who got me started on the whole running thing.

At our post-race breakfast at Cracker Barrel (helllooo, calories), I said that I don’t enjoy running at all, but I’m doing it for the long run. When I’m 50, and I’m not having a heart attack, I’m going to thank myself. Fingers crossed.

Word is getting out at the office about me using a Mac, and at least once a week now, someone comes over to my cube to ask a few questions. Here are the most common ones:

Q: Is that your personal computer, or your work one?
It’s my work one, and it’s the only computer I use.

Q: Is that a normal Microsoft keyboard & mouse, and a normal Dell flat panel?
Yes. These days, Apples use the same peripherals as regular laptops.

Q: So how’s it different from a normal laptop?
Aside from OSX, the engineering of the laptop itself is just awesome. The DVD burner loads via slot – no gimmicky sliding plastic tray. The power adapter has a magnetic connection that just pops loose if it snags on something. The keyboard glows at night, so you can see each key and its corresponding letter/number. The tiny webcam above the monitor blows everybody away. Everything you want is built in, instead of being optional, and since it’s a standard piece of hardware, all of the drivers work perfectly from the get-go.

Q: How much did it cost?
Around 10-15% more than a similarly equipped Dell laptop on our corporate discount, with the exception of add-on memory. Apple marks up memory like crazy, so it makes sense to get that from Crucial.com or another high-quality memory vendor. We get our Microsoft licensing for Windows and Office through our Microsoft Enterprise Agreement, so there’s no extra cost for software licensing over and above what we’d pay on a Dell.

Q: It runs Windows and Office?
Yep, there’s a Mac version of Office and Outlook (called Entourage). Since it can run both Apple OS X and Microsoft Windows simultaneously using Parallels or VMware, it can do everything I need to do at the office.

Q: Are you on the domain?
My Windows virtual machine is, but my Mac isn’t. I could add it to the domain, but I didn’t see any advantages to that, so I’m an island for now. I can access everything on the domain, though – the Mac just prompts for my username & password once, and saves it in my keychain for reuse each time I try to access a domain resource again.

Q: Do you really use that many Mac programs?
Surprisingly, yes. When I first started to switch, I vowed to try to switch over to Mac entirely. I do my email, remote desktop, Word, Excel, presentations, network diagnostics (ping, traceroute, etc), instant messaging, iTunes, and more all in the Mac side of things. I do run Windows to use SQL Server 2005 Management Studio, VMware VirtualCenter,
and IBM Director, and I couldn’t do my job (or at least as good of a job) without those tools. On the weekends, though, I take my Macbook Pro home and I don’t even start Windows on it. There’s even a Mac client for our VPN.

Q: What about that iLife, iDVD, iPhoto, iWeb, GarageBand stuff?
I don’t actually use any of that stuff. I blog using the Performancing tools for Firefox, and I manage my photos with Flickr, but that’s about as close as I get to creativity. I admire those artsy folks, but I’m not one of them.

Q: How long did it take before you were productive?
What the hell are you talking about? I’ve been productive for years! Oh, wait, you mean with the Mac, sorry. It took about two weeks, and I couldn’t have done it without the excellent book Mac OS X Support Essentials. It answers a lot of tech support questions that you’ll experience along the way – things that seem obvious in Windows, but are just handled a little differently in OS X. Not saying OS X is better or worse, just a few things are different, like where your files are stored or how to set an IP address. This book is aimed at technical users who need to solve specific problems, not a how-do-I-get-started approach.

Q: How long is the battery life?
I turn the display brightness way down when I’m working off the grid, so I get around 3-3.5 hours.

Q: What are the drawbacks?

• There’s no connector for a docking station, so I have to plug/unplug the same six cables every time I bring the machine into the office. (Power, audio, USB, ethernet, video, USB2 hard drive).
• Microsoft Entourage (the Outlook equivalent) doesn’t do a very good job of detecting free/busy time in the company calendar, or scheduling meeting rooms as resources. Fortunately I don’t schedule many meetings myself.
• Syncing with a handheld is easier than Windows – but only as long as you have a supported handheld. My Cingular 8125 isn’t supported, but thankfully I use Direct Push with our Exchange server anyway, so that doesn’t really matter to me. It might matter for some folks, though.
• Since only 3 of the 10,000 employees use Macs, it’s much harder to get peer support. For example, the authentication on our Cisco wireless access points doesn’t seem to like Macs, and the network guys don’t have the time to find out why. I’m guessing I could figure it out if I spent enough time on it, but I don’t roam with my laptop anyway (I use my 8125 for meeting notes) so I haven’t taken the time either.

Q: Would you recommend buying one?
For home use – yes, absolutely. For office use – a qualified yes, but check your applications first. If you spend most of your day in a Windows app like, say, the SAP GUI, then it probably makes sense to stick with Windows.

Technorati Tags: apple, osx, switchers

DBAs need an in-office computer to run overnight and weekend performance logs, traces, and long-running queries. For example, let’s say you need to run a long series of maintenance queries to rebuild indexes over the weekend. You shouldn’t be remote-desktopping into your production servers to run this query: instead, you just need a simple workstation that you can leave logged-in, with the desktop locked, that runs the query.

If you were assigned a laptop as your primary workstation, then you’ll need an alternate desktop computer or an existing development server (never a production server). It doesn’t have to be 24/7 reliable or high-powered – a previous generation desktop or a virtual machine is fine. It should be a server that you and only you manage. Otherwise, someone else will restart the server at the most inopportune times, thereby shutting off your traces and performance logs.

Picking the Operating System

If your company uses SQL Server 2000, then this DBA machine should not run Windows Vista, since the 2000 tools don’t run on Vista. You can manage SQL Server 2000 using the 2005’s Management Studio, but sometimes managing a 2000 server is easier with the 2000 tools.

If the machine runs a workstation version of Windows (Windows 2000 Pro, Windows XP, Windows Vista) be aware that only one person will be able to remote into this machine at a time. If it runs a server version of Windows, then two people (or three, depending on how savvy they are) can remote into the machine. If you’ve got two or more database administrators, either use a server, or give each DBA their own workstation for management.

If the security folks ask tough questions about whether you need to be an administrator on the box, thank them for doing their job, and agree with them. (Hopefully they’ve been this tough about your production servers as well.) The only rights you need are the ability to remote-desktop in and run Perfmon and SQL. You don’t need to install anything else, start & stop services, etc.

Applications to Install

It does not need to be a SQL Server. It only needs the management tools installed, like 2000’s Enterprise Manager and 2005’s Management Studio. When you run traces (capturing queries and events from a SQL Server), you will need to save those events to another SQL Server, but it doesn’t have to be this machine. If you don’t have a development server, then – well, you’ve got other problems, but let’s start by fixing them and installing SQL Server 2005 Development Edition on this machine. During installation, change the database to point to a different drive, not the C drive. If you leave them on the C drive, and you save a large trace to this machine, it’s easy to run out of drive space and crash your machine altogether. That would not be good. If the databases are on the D drive, for example, you can still fill up the drive and crash SQL Server, but at least the entire machine won’t crash.

On this server, set up a shared folder called PerformanceLogs, and give read-only access to everyone. When you run performance logs, set them up to be saved in this directory. That way, when other users (developers, junior DBAs, network staff) need to interpret the performance numbers, you can point them to that share and they can always grab the latest copies without your assistance.

Install the free version of LogMeIn.com‘s remote control application.  This helps you remote control the machine from your house even when the company’s VPN is down or when you’re at a location where you can’t get the VPN software installed.  I can’t say enough good stuff about LogMeIn – and in fact, I wrote a separate blog post about LogMeIn.com too.

Next Steps

Once you’ve got your always-on workstation, here’s a couple of articles that you can read for your next step:

• Performance Monitoring with Perfmon – Perfmon is free, and you can use it from your always-on workstation to investigate performance problems with database servers.
• Free SQL Server tools – a list of utilities that you’ll want to install on your always-on workstation to do performance tuning, alerting, and so on.

It’s that beautiful time of year when everybody watches the colors. No, not the changing browns and reds of the foliage, but the red and black numbers on P&L statements. That’s right – it’s budget time.

In order to build a good database budget, we have to ask our managers a few key questions:

Are we going to archive old data, or keep it online?

Database growth can be controlled by keeping a limited amount of history in the database. As DBAs, though, our job isn’t to limit the business, but to give them options. The business can decide whether keeping more data online is worth the additional expenditure for hard drives.

Example: take a data warehouse with 3 years of history. Near year-end, DBAs may be tasked with pruning this data out, but users often complain that they’ve decided they want the extra data online forever. Budget time is the perfect time to present this question to the end users. If the warehouse needs to go from 3 years to 4 years of storage, that’s an increase of 33%. Take the current storage costs, add 33%, and present that as the cost to keep an additional year of history online. Every year, recalculate storage costs and ask the same question again.

Include backup costs in these calculations, too. A truly savvy DBA will organize their filegroups so that old, historical data lives on read-only filegroups that rarely get backed up, but if we were all that good, we wouldn’t be having this conversation.

How has sales growth changed over the last year?

Database size can sometimes be changed based on business growth. If an e-commerce book store sold 15% more books this year than last year, and if they expect to grow by another 15% next year, then that’s a lot more transactions and customers that have to be stored in the database.

I’ve found that it’s easier to budget by past growth instead of future predicted growth because – well, let’s face it, predictions are hard. It’s much easier for a manager to understand and agree that we’ve already grown by 15% over the last year, and that our infrastructure has to keep pace.

Has our company headcount grown this year, and what’s the projected growth?

If a database application supports 1,000 users, and the company projects a growth of 5% next year, then that’s 5% more users that the application has to support. Granted, most applications are built in a way that scales easily, but big growth happens fast. Changes in headcount give DBAs a clear, concise way to show why their hardware has to be able to support more load.

What pilot applications are being rolled out to more users this year?

In small to midsize companies, database administrators aren’t always brought into meetings early enough during the planning phase of application development. Proof-of-concept applications become pilot applications, which become widely-rolled-out applications. At budget time, survey the landscape to see which small applications are loved by the users, and what kinds of wide adoption their managers expect to see.

This is where being a DBA gets hard: database servers are often shared between multiple applications. Being a good DBA means knowing what percentage of the database server load is being consumed by a particular application. A new application might be a nasty resource hog, and it’s not obvious until the number of users scales up. Now is the time to run profiles and performance logs to get insight into each pilot application. The more a DBA knows about each pilot application, the better job they can do budgeting for the future.

Finally, what more could you do with more time?

Make a list of day-to-day and month-to-month DBA chores that are getting overlooked due to a lack of time. Everybody has them – whether it’s fire-drill test restores, performance tuning, or even plain old career training. Give management a list of things that the department could accomplish with one or more additional head count. The list should be unbiased, matter-of-fact items, not “give me another body or I’m going to die.” The less emotion and the more raw facts the list includes, the easier it is for managers to forward it on to their managers and get approval.

Even if another person isn’t needed, make the list anyway. That way, a year or two from now when the department really does need another DBA, it’s easy to compare last year’s list with this year’s list and see that the company’s needs are changing.

The ever-brilliant Jeff Atwood wrote a fascinating blog post about why building software is hard: it’s never been built before.  He was following up on a similar blog post, but in a nutshell, software projects are harder than construction projects because they’re nothing like projects at all.  Every time we build a new piece of software, it’s like building an experimental plane. We don’t even know if it will fly.  Construction projects, on the other hand, are fairly repeatable.  Sure, every now and then somebody builds an Eiffel Tower or the first Frank Gehry building, but otherwise, it’s the same basic things done over and over.

I can completely relate to this because I’m in the midst of bringing a new project – err, experiment – to life as we speak.  I’m being asked to budget dollars for a database server and guarantee that the queries will be returned within X seconds.  The problem has two parts: the queries haven’t been designed yet, and nobody has even documented exactly what they’re going to do.  In order to guarantee that an unknown query will finish in a known amount of time, I’m going to need an exorbitant amount of money – or else, I’m going to have to build an experimental airplane.  I can’t stand spending money without a really good reason, so it’s time to play Skunkworks.

This is why I own a t-shirt with Jeff’s blog logo.

I finished my first 5k race last night, and I look forward to more of ’em.
My official time was 37 minutes 46 seconds. I placed 40th out of 42 in my division (30-34 year old males), 415th out of 496 males, and 866th overall out of 1163 runners. The average male ran 30 minutes 17 seconds, so I’m above average. (ahem) The overall winner did it in 14 minutes 41 seconds, and all but 2 guys did it under an hour. I would like to meet the two guys who took over an hour just to shake their hands for finishing.

You can also slice and dice the results online.

Here’s how the run looked in NikePlus:

I screwed up my iPod settings where you see the first couple of red blips. I was running with a new iPod armband with plastic over the jog wheel, and I couldn’t get my jogging-training podcast back on track.  After that, I had a tough time keeping my pace up.  Lesson learned for next time – cut out the clear plastic, touch the jog wheel directly, and have a full 45 minute playlist set up so I don’t have to screw around with it.
The statistics don’t mean much to me. I hadn’t expected to do as well as I did, and I’d expected to be much more exhausted and pained. I feel good, I’m not sore, and I know I could have pushed myself a lot harder. But why? I’m not doing this to win anything, just to get up off my lazy ass.

I totally understand how these things get addictive. I feel great, and I enjoyed myself. I can see myself running several of these a year just to get the t-shirts.

Today, I signed up for my first 5k race – well, my first race period – the 12th Annual Fantasy of Lights 5k.

While this will technically be a race, I will in no way be judging my results by the time it takes me to finish, or where I finish in the pack.  I feel like a winner just registering for the thing, because if I finish, I’ll be “ahead” of everybody else in the city who didn’t run that day.  That’s the way I like to think of it, anyway.

I have to thank my coworker, Ben, for even mentioning the race.  I wouldn’t have thought to look up upcoming 5k’s, but when he mentioned it, I said to myself, why the hell not?  5k is only 3 miles.  I’ve been running/walking about a mile and a half, and it’s not like 5k is some huge insurmountable obstacle.  Why not?  I could do that.  And hey, it comes with a t-shirt, and I’m all about t-shirts.

The description says: “The 5K course is flat, fast, and ideal for setting a personal record.”  I burst out laughing when I read that one.  I will definitely be setting a personal record, that record being a race completion.

I’ve been loosely following the Cool Running Couch to 5k Plan for the last couple of weeks, but I didn’t think I’d actually be doing a 5k run.  The plan is supposed to take nine weeks.  The race is in a week and a half, but I’m not worried because I’m sure I’ll be walking some of it anyway.

The below screenshot is what you see when you try to go to Google.com with Flock:

When I go to a web site, and software tries to hijack my session to go to a different web site, that’s bad software. It doesn’t matter how well-intentioned the offer is – if the software has the gumption to change my destination, that’s bad software. Bad software, bad. Go to the corner.

I first noticed this in the recent Flock builds, and I thought whoa, this can’t be right. This has to be somebody’s idea of a funny experiment. I mentioned it in the Flock mailing lists when they asked for the community’s top ten concerns, and I didn’t get an official response. I figured okay, maybe they’ll get rid of that in some coming builds. So I just downloaded a tinderbox build, gave it a shot, and it’s still there.

That means they know, and they’re okay with it.

I’m not.

So, here’s the next page I went to:

And just like that, my Flock experience is over. I’ll miss it. I had a personal tie to it – heck, my name was even in the credits for a while – but I just will not stand for software that tries, even just once, to subvert my browsing habits from one provider to another. Not gonna do it. Wouldn’t be prudent.

Now, to find the best del.icio.us plugin…

This weekend, I was determined to set up my Cingular 8125 PocketPC phone to play movies, TV shows, and music videos. It was faster and easier than I’d even expected.

Step 1: get the free PocketDivxEncoder. Install this on your desktop computer. It converts your existing videos to PocketPC-friendly formats that take less space, and fit your PocketPC’s screen better. It has presets for a lot of common devices like smartphones, iPods, PlayStationPortables, and more. They’ll try to upsell you on their new product, Lathe, for $6, but the older PocketDivxEncoder works fine. Amazingly, PocketDivxEncoder can even use Quicktime files as input, which I love since I’ve got a bunch of music videos in that format.

Step 2: get the free The Core Pocket Media Player. This installs on your PocketPC to play videos. No additional codecs or plugins are needed if you’re using the PocketDivxEncoder.

Step 3: convert your movies and copy them to your PocketPC. Just open each one with PocketDivxEncoder, and use the default settings for PocketPC. It converts the video to 320 pixels wide and resamples the audio down to mono to save space. If you’re a music nut like me, you’ll probably want to crank up the default audio quality to a higher setting to keep the tracks in stereo for headphone-quality audio. Music videos ended up taking 7-12mb each, and movies take around 200mb. Works like a champ! The quality is beautiful for such a small size.

Upon viewing The Sheep Market, Erika remarked:

“People on the internet will work for nothing, and they buy everything.”

I got a good laugh out of that. It’s pretty much true, and The Sheep Market is a great piece of evidence. Aaron Koblin used Amazon’s Mechanical Turk to get 10,000 drawings of sheep at a cost of $.02 each. He’s now reselling them in 20-packs in sticker form for $20, netting him a tidy profit. Brilliant and funny idea.

Tara Hunt posted her BayCHI presentation notes, and she touches just a bit on why startup companies (heck, any companies) need to focus on building active user communities and listening closely to them.

The community phenomenon is something that everyday project planners need to recognize and acknowledge. Internal IT projects benefit by building this kind of an excited user base. Project managers who try to push bloated, overwrought projects down the throats of their users will … well, it won’t be pretty. The end result is a “successful” project deployment with a tiny amount of users.

Instead, take the approach Tara encourages startup companies to take. Listen closely to your perceived user base, and tailor the project to their needs as the project moves along. Respond quickly to their requests, and adapt your plan. Your own early adopters will make your project more successful, get the word out faster, and turn your project into a vital part of their everyday routine.

Plus, the more you make your users feel valuable and the more they feel appreciated, there’s a better chance they’ll tolerate your mistakes along the way.

I like encouraging people to think of their projects as little businesses. They need to show a positive ROI so that the business understands where the money’s going, what the tangible return is, and maybe best of all, they need to see your excited customer community. This is where internal projects are a bit different than standalone businesses: an internal project with a big fan base can “lose money” for years as long as the execs understand how much it’s contributing to the success of their customers.

IT project managers – treat your projects like startup companies. Read marketing tips from these Silicom Valley veterans, and apply their startup techniques to your own projects. It’s all about building your business – or your project – into something successful.

You wonder why I’m so damn chipper on the mornings of Mondays, Wednesdays and Fridays? Blame Penny Arcade, my favorite web comic. It’s chock full of in-crowd jokes, but the commentary by the authors is almost – but not quite – accessible to everybody. Take a line from today’s commentary where Tycho is talking about an author that he holds in less than high esteem:

Awesome! It’s a Goddamn retard rodeo over there. It’s fun to think about the author tippity-tappin’ this one in while he eats light bulbs in the last car of the circus train.

Be still, my laughing heart.

If you scan hard enough through their history, there’s a few non-computer-related comics like this one, but most of the time you’re not going to “get it”. My apologies in advance.

For more a more accessible, more oftenly updated, and yet just as obscene comic, check out Questionable Content.

I’ve had so many hilarious things going on lately, I don’t even know where to begin.

Last weekend, IBM Director started emailing us that a couple of servers were overheating. In a matter of minutes, the heat problems were spreading across different racks, and the IT crew all started driving into the office simultaneously. Our two redundant AC units for the datacenter both failed within minutes of each other.

We shut down non-essential servers as fast as we could, but the room temps were still climbing over 120. Infrared thermometers were showing the rack equipment over 150. We ended up shutting down everything but our really, really mission-critical stuff, and we were thinking about failing that over to our alternate datacenter. Hours later, the ACs came back online.

No lost data. Hooah! Gotta love happy endings. Still, the whole event was a heck of a hard way to learn some ugly lessons, and I’m looking forward to the lessons-learned meeting on Monday.

Mostly, I’m just relieved that we didn’t have to disassemble server racks, carry huge servers out into the office area, plug them in, wire them up, and run a distributed datacenter out of a bunch of cubicles. That would have been so embarrassing. I can live with long hours in the datacenter trying to rebuild machines with fried drives, I can deal with failing things over to other datacenters, but I don’t like users seeing the datacenter in disarray.