Blog

I try not to wear too many computer-related t-shirts because they emphasize my fashion ineptitude, but when Jeff Atwood’s Coding Horror blog came out with their logo t-shirts, I could not resist. The logo is great, the blog is funny and spot-on, and let’s face it, every geek should have at least one good inside joke t-shirt.

Besides, if I didn’t buy one of these, I’d probably be buying Apple t-shirts, and I don’t want to go down that road just yet. The mere ownership of a Macbook Pro does not yet entitle me to all the coolness perks.

Ernie and I took a trip this morning to Animal General Hospital, and Dr. Campbell says she’s got a rash. Might be scabies, but probably not, and he didn’t want to do the required skin scrape test on her right now with the skin being so sensitive.

She got a shot in the bottom to prevent the itching short-term, plus a couple of prescriptions for anti-itch and antibiotics.

I’m watching the World Cup finals between Italy and France. The rest of the world is on to something – soccer is a lot of fun to watch. The action is nonstop, the athletes are really skilled, and there’s less time for commercials. Woohoo!

I spent some time yesterday going through an evaluation of the upcoming release of Microsoft Operations Manager 2007, and figured I’d share my evaluation notes with you, dear reader.  I’ve used MOM before, and I wanted to see what was new.

I’m looking at SCOM 2007 because I’d run into a showstopper with UniCenter: it can’t monitor my 64-bit machines. Since we’re going into production with 64-bit SQL Server for the data warehouses, I need a short-term (6-12 month) solution that will handle the 64-bit monitoring until UniCenter has 64-bit agents ready.

Server Monitoring Features I Need

My main criteria in order of importance:

64-bit OS and Application Monitoring

I’m looking for both operating system monitoring and application statistics monitoring. It’s not enough to just see that a 64-bit system is up – I need to know application events, like when SQL Server is running out of log space for a particular database. UniCenter is able to gather this data, and so should my short-term solution.

Easy To Deploy & Maintain

This system has to reduce our workload, not increase it. My goals for deployment are:

• Monitoring system deployment in 1 day
• Install agents on each server in 1 hour
• Add & edit users and alerts in 5 minutes
• Have everyone on the Windows group able to do those functions in less than one hour of training

There are plenty of monitoring products out there that achieve the rest of my goals, but miss on this ease-of-use one. I’m intimately familiar with ServersAlive, for example, but if I’m the only guy in-house who can maintain it, and it takes me one day a week, then we’re worse off than when we started.

Simple Monitoring Dashboard

Email alerting is good, but when a whole bunch of things go wrong at once and then get fixed, like during a network outage, I want to be able to see what’s still broken at a glance. A good example of this is UniCenter’s display up in the CORE. Bonus points if that console is accessible via a PDA.

Server Performance Trending

Right now, I’m manually gathering statistics on my machines via Windows Performance Monitor one week per month. I assemble that data into Excel and run calculations on it to stay on top of my machine health. Unfortunately, that takes about a day per month to collect, report and analyze. I’d like to automate that process. (UniCenter does this.)

With these goals in mind, I evaluated Microsoft Operations Manager 2007 Release Candidate 1. The product is mostly done, but things like documentation and polish are still hit-or-miss.

SCOM 2007 Review: Positives

SCOM 2007 is 64-bit capable.

It monitors 64-bit operating systems and applications seamlessly. The agents install and work fine without a hitch. OM2007 automatically detects and installs the appropriate 32bit or 64bit agents, including SQL Server 2005 64-bit.

SCOM autoinstalls the right agents.

To add a new server, right-click in the Administration console and click Discovery Wizard. By default, it will scan your AD forest, but that’s not a good idea here since we’re just evaluating, so instead, click Advanced Discovery, click Next, click Browse For Computers, and click the Browse button to search the AD forest. It will ask for an admin account to install the agents, and click Other Account to use your own account if you’re an admin on that box (or a domain admin).

It then checks your servers to make sure they’re not already managed, and then asks you which management mode you’d like to use (agent or agentless). If you choose to use an agent, it will ask which account the agent should use – just use the system account.

SCOM2007 detects the operating system (Win2000 vs 2003 vs 2003 64-bit) and installs the appropriate agents.

Even better, it automatically detects supported applications like IIS, Exchange, and SQL Server, and installs the right agents for those as well. The whole process is very transparent – I didn’t even have to tell it which servers were running SQL, and which ones were not.

Bottom line: I can add a new server within 1 minute, which is way better than I’d expected. Note that after the agent is installed, it takes several minutes before health data is reported.

It’s easy to add and edit email notifications.

Adding new users takes less than a minute and requires no special skills. To add a new user, go into the console and click Administration, Notifications, Recipients, then right-click on Recipients and click New Notification Recipient.

Adding new subscriptions (alerts) is also easy. Right-click in the Administration console and click New Subscription Notification. A wizard steps you through all of the choices like which applications to monitor (Windows, IIS, SQL, etc), what levels of alerts (health, maintenance, performance, etc), and how often to send reminders.

It has a great monitoring dashboard.

The console can be installed on laptops easily, and can run over the VPN. It gives a great heads-up view of the servers.

It has in-depth application health reporting.

It has deep visibility into SQL Server health statistics, and even the default settings gave me useful information. For example, it alerted me when a database shrank itself, which is a major no-no for enterprise databases.

It has a maintenance mode.

I can put servers in maintenance mode before doing work on them. That way, when I’m rebooting a machine to fix something, no false alarm emails go out. I actively used this feature in ServersAlive at my last company, and I’ve asked for the same ability in UniCenter too. When a sysadmin group is really living by these alert emails, every reduced false alarm means a lot.

System Center Operations Manager 2007 Negatives

It still has annoying bugs.

For example, I can’t always edit my email subscription – when I right-click on it and click Properties, the entire System Center console crashes and quits. Another example is that personalized views aren’t always saved correctly, and the sort order is reset when you go from one screen to another. Maybe these bugs will be fixed in the final release, but again, maybe not.

You have to know your application’s vital stats.

Like UniCenter, SCOM2007 can send a dizzying array of notifications. For a successful implementation, the sysadmin of each box needs to understand what they’re monitoring. When I first told it to send me everything, despite only monitoring a handful of servers, I got about 150-200 emails per day – only about 10 of which were useful. I’m now in the process of whittling down the notifications.

The console requires horsepower.

It runs about the same speed as UniCenter’s tools on my P4 laptop, which is to say slow.

The dashboard is not web-accessible.

In order to manage SCOM2007, you have to install the console or remote desktop into a machine running the console. Long-term, I would prefer a web-accessible portal like UniCenter’s, and in a perfect world, it wouldn’t require Java either, so I could run it from my handheld. But now we’re just talking crazy.

What I Didn’t Evaluate

Server Uptime Reporting

It’s integrated with SQL 2005 Reporting Services and does some data warehousing as well. This was way outside of my scope of needs, though: I don’t anticipate long-term use of this as an uptime tracking tool or statistics analysis system. I’d rather get back to UniCenter for performance monitoring when its agents become available.

Agentless Auditing

OM2007 has the ability to monitor systems without installing the OM2007 agent, and it can even audit event logs without an agent. For the purposes of this evaluation, I wanted to install the agents because I wanted in-depth reporting on the applications (SQL Server and IIS).

My Recommendation

Operations Manager 2007 achieves the requirements so well and requires so little effort that I would recommend we deploy it for a short-term 64-bit monitoring solution.

.flickr-photo { border: solid 2px #000000; }
.flickr-yourcomment { }
.flickr-frame { text-align: left; padding: 3px; }
.flickr-caption { font-size: 0.8em; margin-top: 0px; }

Ernie goes to church, originally uploaded by BrentOzar.

On Lincoln Road, no less.

This past week, we had an IBM guy come in for a quick-start session on implementing IBM Director. Maintaining dozens or hundreds of servers can be a chore, and Director is supposed to make it easier by automating hardware checks, firmware updates, and even phoning home when there’s a problem.

Part of this package is IBM UpdateXpress Server, a central repository for all of the IBM firmware updates. Admins are supposed to set up Director, point it to their UpdateXpress repository, and then match up to see which servers need which updates – and deploy them easily from inside Director. This beats doing a sneaker shuffle around the datacenter, carrying CDs and floppy disks from machine to machine.

The reality didn’t match up with the promise. UpdateXpress Server is horridly out of date. It fetches internet updates from a different web server than the public hits, and that server is way behind what the public server has. For example, our x460’s have had several critical bios updates in the last three months, and none of those updates were available to the UpdateXpress server.

That immediately set off alarms with the IBM guy, who couldn’t believe what he was seeing. We repeated the same update tests with other servers (x346’s and x366’s) only to get the same results. None of the firmware updates that had come out in the last 6-9 months were available inside UpdateXpress Server.

To make matters worse, IBM Director only allows Windows software packages to be deployed – not floppy images. When a bios update comes out only on floppies, like the CPLD updates for the x460’s, Director can’t do anything about it. It’s still time to do the sneaker shuffle. Since these updates often come out simultaneously, some in Windows and some in floppies, then I’m really not saving any time at all.

My experiences have been echoed by others who had bad experiences with UpdateXpress Server. I only wish I’d seen these on IBM’s forum before we bought the Jump Start package.

We were all disappointed with the results, and I’m posting this just to warn anybody else who’s considering the IBM Director jump start program. IBM’s servers are phenomenal, but the firmware update system is just as bad as the hardware is good.

Oh yeah, I gotta have me one of these.

Here’s the deal: as of about a year ago, Apple started moving their entire lineup over to Intel processors instead of Motorola ones, which means Apple computers can run Windows. During the boot process, the user can pick which operating system they’d like to run – Apple OSX or Windows. That’s great for those of us who still have to use Microsoft programs at work, like me. I make my living working with Microsoft SQL Server, and I can’t imagine trying to administer a database without the new SQL Server Management Studio. It doesn’t run run on Macs, doesn’t run on Linux, etc – ya gotta run Windows.

So now, I could either boot into Windows, or I could use Parallels, a slick piece of VMware-like software that lets you run Windows inside a Mac.

This was all good, but until last week, the only Intel-based Mac laptop was $2k – a little too pricey for me. Now there’s the new $1100 MacBook, and that’s a heck of a cheap price point. They make a black model for $1500, but I’d rather have the white one, actually.

The only question now: how can I get work to pay for it? Heh….alright, who in the company needs a new (used) Dell D810 laptop?

Yeah, it’s been a while since my last blog post, sorry about that. Life is a blast here, having a ton of fun with things at work, but I’m going to try to catch up on the blog over the weekend.

First things first: BarCamp South Beach! Woohoo! I can’t believe it. My first kinda-Valley-ish experience. I’m excited about it. I haven’t been involved in the geek meatspace since I stopped going to HoustonWireless last year, although I still keep in touch with a few of those guys.

But anyway, back to BarCamp. If you’re in the Miami area, you should check out the planned BarCamp experience and consider signing up. It’s demo-centric, meaning you’re expected to share as well as learn. I’m going to be frantically brainstorming about what I could show off. I doubt the attendees would be much interested in the finer points of database systems that cost $25k per CPU, but I might see if I can put together some demos of how we use Microsoft SQL Server on handheld computers. At the very least, I can help out by bringing projectors or servers from work.

Technorati tags: BarCamp, BarCampSouthBeach, BarCampSoBe

I’m a big fan of the relatively inexpensive Case Studio as an alternative to the big database design tools. Case Studio does a very competent job at reverse engineering databases, modeling, UML, etc.

The reverse engineering just got a whole lot better with the recent InferRI add-in that can infer relationships between tables even when no relationship is defined in the database. It looks at what fields are the primary keys in each table, and then looks for similarly named fields in other tables. This makes reverse-engineering databases a lot easier for database administrators who don’t like to force the database to enforce relationships, or who take over databases that have never been managed by a DBA before. This thing just saved me a couple hours of work.

The only drawback is that the field names have to be identical. If the Customer table has a primary key of CustomerID, and the Orders table has a field called PurchasingCustomerID, CaseStudio won’t infer the join between the two tables.

Pretty damn good for a first version of the add-in, though.

The latest newsletter from the excellent SQL-Server-Performance.com had a tip about doing performance tuning on high-powered machines. If you need to test large queries running inside the server room (meaning, the app or web server runs a big query against the db server), you may not get the speed results you’re expecting if you try to run that same query on your desktop. SQL-Server-Performance attributes this to low resources on the desktop, but there can also be another reason: network bandwidth and latency.

Last week, I was doing performance tuning on a SQL Server that supports users across the United States. One of the applications exhibited very high Duration numbers in the profiles that I ran, and I couldn’t figure out why. Even when I tuned the tables by adding appropriate indexes, the load on the server didn’t improve the way I’d expected. Finally, I found out that this particular application was running on machines in another state – not application servers inside our data centers like we’d typically do. The query results were being piped over their slow VPN connection down to their workstations, and so the Duration statistics were artificially high. Sure, it looked like the queries were taking forever, but in reality, it was just a slow network connection, and there was nothing I could do on the database side to speed that up.

The morale of that first story: never performance tune by looking at a single statistic alone, like Duration, Reads, Writes, or CPU. Use all of them together to gauge what queries are really having an impact.

Technorati tags: PerformanceTuning, SQLServer

.flickr-photo { border: solid 2px #000000; }
.flickr-yourcomment { }
.flickr-frame { text-align: left; padding: 3px; }
.flickr-caption { font-size: 0.8em; margin-top: 0px; }

Raising the dead…servers, originally uploaded by BrentOzar.

I’ve blogged about more of the downsides than the upsides of my new Cingular 8125, and it’s time for some opposing views.

Stuck in the datacenter working on a dead server, this thing proves its worth. I’m able to take full advantage of some spare time while waiting for stuff to work. I’ve caught up on my emails. I’ve caught up on my voicemail, thanks to our slick VoIP phones that send our voicemails to email. I’m even blogging! Hooah!

Now I just need a pair of Bluetooth headphones so I can whistle while I work.

For the past few days, I’ve complained a couple of times about the lack of testing on my new Cingular 8125. My troubles continue with the MMS integration. The device has a great digital camera, and it has great email support, but do the two work together? Not so much.

STEPS TO REPRODUCE:
1. Take a picture. (I took this particular one yesterday afternoon while walking Ernie down the boardwalk, and I’m only now getting the damn thing into my blog because this process is such a pain in the ass.)
2. Click Send at the bottom of the screen.
3. You get options for each of your email accounts – but not Goodlink, and I can almost understand that because it’s a third-party email client. However, Cingular touts its support, so Cingular should really support it. Bottom line, you can’t send camera phone attachments with Goodlink. But I digress – back to this particular bug. Choose MMS, a picture message.
4. The MMS window pops up, and the first thing is the “To” address.
5. Start typing a person’s name, an email address, or a phone number.

ACTUAL BEHAVIOR:
Nothing happens.

EXPECTED BEHAVIOR:
When I start typing an email address or a phone number, it should pick from my list of contacts and complete the address for me. Or at the very least, it should pop up a clumsy additional window and let me pick an address from my address book. No such luck. It’s as if my address book on the device didn’t even exist. Come on, man, this device fits in the palm of my hand – why are all of the parts so damned independent?

This brings me to testing. The bug reporting format I used above was copy/pasted from the Flock wiki entry on submitting Flock bugs. If I have a problem with the browser, I can simply submit a bug report like I did this morning. I caught a minor grammatical error and submitted a bug for it. I can then watch the Flock crew react to it, hopefully fix it, and publish the fixed code. I can download the fixed browser and see my contribution in action. The very fact that they’re open to my opinion gets me even more excited about using their product.

Now, contrast that with my Cingular 8125. I shudder to think at what would happen if I tried to report a bug to Cingular. I know they have to deal with a lot more users, and they have to insulate their developers better from their end users, but there should still be an open forum at some point where I can go to find problems other people are having with this device and its firmware. Oh, sure, there are forums alright, but they’re independent third-party phone forums that have sprung up out of desperation because they can’t get what they want from the big companies. But that doesn’t make me more excited about Cingular – it makes me more excited about the forums. Cingular has a chance to build their brand here, build a community of involved users, and they’re missing the boat.

Flock, on the other hand, Gets It. They understand that by building a community of involved users, they encourage their product to grow and spread wings.

After working with my new Cingular 8125 for a few days, I can say it’s a keeper. As soon as my memory card gets here, I’ll probably even stop carrying my laptop home during the week. I haven’t even started putting 3rd party applications on it yet, but I can say that it’s going to be my primary electronic device outside of the office.

Every single good feature, though, has the oddest drawbacks that point to insufficient testing.

Good: the keyboard buttons are huge and make typing a breeze. Bad: there are so many buttons scattered all over the damn thing that it’s almost impossible to hold without accidentally triggering the camera, voice dialing, comm manager, etc. It’s great for short-term typing, say one paragraph or less, but anything longer than that is risking disaster. It’s just too easy to accidentally start another application right in the middle of typing a paragraph, and then depending on what keys you’re in the middle of pressing, you might trigger disaster. I’ve started phone calls to people, deleted files, taken pictures, etc, all while trying to just type a letter.

Furthermore, with so many multi-purpose buttons liberally slathered all over the phone, why take up onscreen space with visual buttons that have the same purpose? There’s a dedicated hardware button for the Comm Manager – why take up an entire line of the “Today” screen to show one tiny icon for it as well? And don’t get me started about the location of the Power button: when the phone is holstered, the power button is precisely positioned to hit the seat belt socket in a car. Every time I get into the car, I turn the phone on without knowing it, and then as I buckle the seat belt, it places a call. I’m to the point now where I just take the phone off before I get in the car.

Good: two soft keys make it easy to get around in most screens, and HTC even duplicated them above the keyboard for use when the unit is in landscape mode. Bad: these two buttons are so close to the keys that they’re almost impossible to avoid. When composing email, the soft key right above the E will instantly send the email without asking if you’re sure. I looked like an idiot when I sent out three partially completed emails in a row to a group of execs. Not good. If this setup was tested, it was tested by somebody with needles for fingers.

Good: the 8125 comes with a stereo earbud headset with microphone, and it makes a killer wireless walkman. I love walking the dog while listening to Jimmy Buffett’s Radio Margaritaville over the internet, all live. Bad: the earbuds are huge. We’re talking mammoth. I don’t have small ears by any means, but I can’t keep these earbuds in my ears. They pop out at the slightest provocation because they’re apparently designed to fit in Ernie’s ears. There’s no way anybody tested these before deploying them.

Good: the camera takes great pictures for a PDA/phone. The picture you see here was taken outside at dusk with minimal lighting. The camera is very light-sensitive for a simple PDA, and a $300 PDA at that. Bad: despite buttons scattered all over the surface of this device, it’s impossible to change most of the camera settings without getting out the stylus.

Good: underground firmware available on the web enables Microsoft Direct Push email with Exchange. Companies using Exchange Server 2003 with SP2 can enable push email out to PDA’s and phones without paying for GoodLink or Blackberry Enterprise Server setups. It works pretty well out of the box with no configuration. Bad: while the push email does work, it isn’t anywhere near the speed of GoodLink or Blackberry devices. I routinely got emails on my PDA several minutes after they arrived in my inbox – not acceptable for “instant” email. Thankfully, our company uses GoodLink as well, and one of the network guys hooked me up with a GoodLink account. Presto, the emails arrived on my 8125 even before they hit my laptop. That’s what I call push email!

Good: wifi reception is better than I expected. Bad: the phone doesn’t intelligently switch over to WiFi for browsing when available. I’ve even had to manually shut off the GPRS connection just to force all communications to go over the available WiFi connection just to get the fast speed. Totally worth it, though – this device makes for a completely workable web browser from the couch during TV commercials.

Good: the device works pretty well as a phone, and the big screen means the onscreen buttons are large enough to actually work for one-handed dialing. Bad: it doesn’t come with support for A2DP, the new stereo Bluetooth profile for headsets like the droolworthy Jabra BT620. It doesn’t even come with some of the more basic features I expect in an intelligent phone, like the ability to set custom ringtones based on the caller. Come on, man, don’t let my shiny new PocketPC phone get outsmarted by a $50 Nokia.

All of the annoyances I’ve mentioned are minor. The best review comes from everybody who sees this thing in action. They all seem to say the same word: “Whoa.” The screen is big and bright, the keyboard buttons are better than a Treo’s, and it runs Windows Mobile 5. At $300 from Cingular, it’s a heck of a deal. I don’t buy new gadgets as often as I used to, and I’m completely confident in buying this thing right now. It’s the best PocketPC phone on the market today, and it’ll be a long time before anything better hits the stores at this price point.

Take two.

It’s hard to write a glowing review of a piece of consumer electronics when the damn thing crashes every half hour or so.

As I was saying when I was so rudely interrupted by a frozen screen, my new Cingular 8125 has everything from WiFi to Bluetooth to a full-blown keyboard.

Make that a half-baked keyboard. The keyboard, like the rest of this rebadged HTC Wizard, has a few annoyances that mar an otherwise amazing experience.

Take the keyboard backlighting, for example. At first, I was amazed that the keyboard lighting worked so well. After a few minutes, though, I got aggravated by how quickly the backlight turned off. The keyboard doubles up every letter key with punctuation or numbers. The only punctuation mark with its own key is the period. To use a question mark, I have to find it on the keyboard, then hit the modifier key and the key. If I don’t find it in time, the backlight cuts off, and I have to hit some random key to turn it back on, then backspace, and start looking again for my punctuation mark. Typing becomes a sort of masochistic race against the clock.

But who cares, right? I’m lounging on the couch blogging from my phone! I’m even listening to Jimmy Buffett’s Radio Margaritaville streaming audio out of this thing’s stereo speakers WHILE I TYPE! It’s a freaking multitasking phone! This little machine is way more fun than a $300 phone has a right to be. And the camera quality is spectacular. Just look at at the picture attached to this blog entry!

Oh, wait. That’s right. I can’t get the thing to email a decently sized picture because it keeps timing out trying to send an mms message. Another annoyance.

Every feature I’ve tried so far has a similar drawback. Thankfully, it looks like everything has a workaround so far. It crashes unexpectedly, so I’ve already learned to pen my blog entries in Word instead of the web browser, saving them every few minutes as I go. Like any good relationship, it’s all about compromise. You never get exactly what you want. You have to put up with a little flakiness in order to get the good stuff. So far, the Cingular 8125’s a good balance between the two.

My new job’s going extremely well, having the time of my life. I’m working with a group of very competent programmers who’ve produced a whole lot of code, but without the luxury of having their own database administrator. As a result, the database is in truly horrendous shape. As an example, I spent about an hour tuning indexes for a nightly job that’s been taking 4-6 hours to run. After tuning, it takes 1-3 minutes. MINUTES.

Does that make the programmers incompetent? Of course not. They’re very good at what they do, but they’re not database administrators. They know enough to be dangerous, but not enough to scale enterprise apps.

Another guy in the company (not you, Carlos, if you’re reading this – it’s another guy) asked me how I became a database administrator. I explained that I got started programming, and eventually took on enough database administration duties to do it full time. It’s a good fit for me because I’m much better with databases than I am with programming – debugging sucks – and I really enjoy it.

The guy then asked, “So if I was going to start becoming a database administrator, what would I do?”

I was kinda caught off guard. I don’t have a good answer for that. I mean, I have plenty of good answers, but only for people who are already doing network administration or programming. If you can do either of those halfway well, then you can be a database administrator. But to start with DBA work, with no prior programming or network administration? I wasn’t quite sure how to answer that. It struck me as a get-rich-quick scheme, and further conversations with him have proven that to be his motive.

Today, I read Peter Norvig’s excellent Teach Yourself Programming in Ten Years, and it does a fantastic job of explaining my feelings on this subject. Don’t expect to jump into anything – programming, network administration, database administration, etc – and be an expert in 24 hours, regardless of the book titles down at Barnes & Noble.

Change of plans. Southern Wine & Spirits, where I’ve been consulting since October, decided at the last minute that they wanted to bring me on permanently to work directly for them. They’re a fantastic group of people, and they made me an offer I couldn’t refuse.

Pictured here is my new cube. It may not even be my permanent cube – there’s a slightly quieter cube available right outside Don’s office, and I’ll probably swipe that one instead.

Squint closely at the picture and you might be able to make out my Dell Latitude D810, Dell flat panel, Cisco VoIP desk phone, and Cisco 802.11b cordless handset. Muhahaha, new toys….

I’ll probably be slacking off on the blog entries this week and next while I get up to speed in my new duties. It’s the same company, but a completely different set of duties.