Software Development Meme

Jason Massie tagged me, so it’s my turn to answer the questionnaire….

How old were you when you first started programming?

Dad and Mom upgraded us from an Atari 2600 to a Commodore 64 when those came out, so I must have been around 9-10.  I don’t remember much from those early attempts at programming, but I remember being really frustrated that there was so much typing involved to copy the stuff from magazines into my own computer.

See, I have this mental problem where I don’t want to do something unless I can be really successful at it in a short period of time.  That problem is often defined as laziness, but I’m not lazy - I don’t mind working really hard, but I don’t want to STRUGGLE really hard.  I will work 20-hour days, but I wanna know that I’m actually achieving great things, not trying to accomplish something basic.

Typing long, boring lines into a computer, especially lines that I didn’t think up, definitely fell into the category of time wasters.  It’s one thing to pour your ideas out into a keyboard, but it’s another thing to transcribe somebody else’s ideas, character for character, and then try to hunt down your typos without a debugger.

(That same mental problem is what kept me out of sports!  Practice for weeks just to be able to shoot a free throw?  You’re out of your mind….)

How did you get started in programming?

Lemme tell you how I didn’t get started: I vividly remember Mom forcing me to do piano lessons, and Dad forcing me to do soccer.  Neither of those hobbies stuck in any way, shape or form.  I don’t remember how I got started in programming, but I remember poking and peeking around on my own, so I bet I ran across it in a magazine.  I was a voracious reader.

What was your first language?

BASIC.  (Makes me chuckle because future generations will respond to that question so differently.)  I think my second language, if you can call it that, would have been DOS batch files, though.  I really, really enjoyed MS-DOS.

What was the first real program you wrote?

To me, it’s not a real program until the first user signs on.  That’s when you find the real bugs, the real shortcomings.  My first real program was a help desk front end written in classic ASP.

Our company was growing by leaps and bounds, and McAfee wanted absurd amounts of money for more user licenses for their help desk software.  That software sucked - I mean, reeeeeally sucked - and I said to myself, I could write something better using the same SQL Server back end, save the company a lot of money, and the users would love me because it’d be so much easier to use.  Plus, if I had any ugly bugs, they could use the old Windows console version while I sorted my bugs out.  The company still uses that system today, and it’s had over 100k help desk tickets.  Hooah!

That’s still my favorite rush in programming: walking past somebody’s computer and seeing my stuff on their screen as they interact with it.  I really love knowing that somebody could choose to use any piece of software out there, and they’re choosing to use mine.  That rocks.  It’s like being the cool kid in school.  (Only without the chicks.)

What languages have you used since you started programming?

Trying to think of these in the order of my career:

  • BASIC on a Commodore 64
  • DOS (yes, I consider batch files a language, especially when they’re hundreds of lines long)
  • VBscript
  • HTML
  • Classic ASP
  • Topspeed Clarion
  • T-SQL
  • Java (the language that made me decide to never learn another language again)

What was your first professional programming gig?

Telman (subsequently bought by UniFocus) hired me on the basis of personal relationships and my hospitality industry knowledge, and then sent me off to Clarion training.  I really liked Clarion, and I haven’t touched it in years.  Makes me want to go play with it now, come to think of it.  Clarion was a database-independent language: in theory, you could change your database back end with a couple of mouse clicks, recompile your program, and hook it up to a different database.

The problem is that when you’re using a particular database, you want to take advantage of the database-specific features that give you better performance or more capabilities.  If your code is generic enough, though, or if you’re willing to invest the time to debug it once, it does work, and I did manage to switch a few apps between Clarion’s proprietary flat files, to Microsoft Access, to SQL Server 7.0.

Telman was also my last professional programming gig.  Clarion was a dying language, so the company had to switch to a “new”, more maintained language - either Java or .NET.  I saw the writing on the wall; .NET would have short-term staying power because it has the Microsoft marketing power behind it, but something else would come along in 5-10 years and knock it over.  I could spend a few years becoming really proficient in Java or .NET - but then have to relearn a new language within a decade.  Why bother?  The ANSI SQL language lasts forever, even across different vendors.

If you knew then what you know now, would you have started programming?

Yeah, because I think it makes me a better database administrator.  No way in hell would I go back to programming, though - I hate finding bugs in my own code.  Stored procs are easy enough to unit test and be pretty certain that they’re correct, whereas code that faces end users - that’s another problem entirely.  Users are crazy.  They click everywhere, they do things that don’t make sense, and they expect everything to work flawlessly.  That is seriously hard work.  I really respect good programmers.

If there is one thing you learned along the way that you would tell new developers, what would it be?

Languages come and go like fashion trends.  Don’t get stuck in a pair of baby blue bell bottom pants: choose a language based on how your resume will look 5 or 10 years from now, not based on what the cool kids are doing this week.

Really, really good programmers can pick up a new language in a heartbeat, but you may not be so lucky.  Spend your time focusing and getting really good at one language, and don’t listen to the siren song of whatever new language is sexy today.  Take database administration: learn to code ANSI SQL today, and you’ll still be using that same syntax in 20 years.  Learn the trendy new LINQ, and you may be relearning something else in a few years.  (I’m not sayin’, I’m just sayin’.)

What’s the most fun you’ve ever had … programming?

Embedding sounds in other people’s help desk tickets with hidden HTML code in the ticket notes.  I embedded the A-Team theme song in a note on somebody’s ticket, so whenever he pulled up his list of tickets, the A-Team song started playing, and he didn’t know why.  When I finally let out the secret, hoo boy, that triggered a flood of embedded sounds and graphics.

Who are you calling out?

Bert Scalzo because he talked me into this sweet job

Brian Knight because reading his stuff is almost as good as listening to his seminars

Conor Cunningham before he goes back to work for Microsoft

Jeremey Barrett because I bet he’s better at programming than he lets on

Linchi Shea because he’s a SAN genius

Rhonda Tipton because I’m seconding Jason’s call-out

Before me, the tag order was something like this: Jason Massie > Denis Gobo > Andy Leonard > Frank La Vigne > Pete Brown > Chad Campbell > Dan Rigsby > Michael Eaton > Sarah Dutkiewicz > Jeff Blankenburg > Josh HolmesLarry Clarkin

The Cloud is Coming After Your Children!

I blogged about why database administrators should use Twitter, and I personally loved that tool.  It helped people build faster personal connections with each other, and it helped you get a picture of what the brilliant people were thinking at any given time.  Take Brian Knight, a SQL Server guru, for example - I love being able to see what he’s working on now, because he does things that shape how the DBA world works.

Unfortunately, Twitter started running into some serious infrastructure and architecture problems.  They built the whole thing on top of a couple of MySQL servers with no real Plan B - no way to scale out, no automated cluster failover, no partitioning, and basically no way to make the database fast enough to withstand the user load.

All of this was frustrating, especially since early Twitter adopters tended to be people who worked in & around information technology.  We do this stuff for a living, and it frustrates us when we depend on a tool that is constantly borked.  I got a few laughs out of Twitter’s scaling issues - not because I thought they were easy to solve, but because I’ve worked with coders who took that same approach.  How many times have we heard, “But it works fine on my desktop with ten users….it’ll be fine when we move it to a real server and we have a hundred thousand users.”

Here’s a couple of links with more information about Twitter’s struggles:

That’s right, fifteen million bucks.  A company with a newly minted $15m in the bank should be able to build in some scalability pretty quickly, right?  Buy some faster servers, build in some redundancy, off we go, right?

Not so fast.

Architecture problems take time - especially downtime - to fix, and the one thing Twitter didn’t have was time.  More users piled onto the service every day, more applications built hooks into Twitter’s open API, and the press started asking questions about this cool new service.

Smelling blood, a lot of other competitors have sprung up around Twitter.  Jaiku, Pownce, Friendfeed and many more sites launched to do similar things.

On July 2nd, a new competitor called Identi.ca launched and gathered a lot of press right away.  I’m not going to go into most of the technical differentiators here, but the bottom line is that Identi.ca picked up over 10,000 new users in its first 3 days.

Let me repeat: Identi.ca picked up over 10,000 new users in its first 3 days.

That’s a big rush, and it requires a lot of hardware.  Hardware is expensive.  Datacenter space is expensive.  These things require a lot of VC funding up front, plus planning time to buy, install and provision.  Nobody wants to pay for users they don’t have yet, especially in this tight economy, so how was Identi.ca able to scale up that fast?

Identi.ca is hosted on Amazon EC2 cloud servers.  Anybody can rent EC2 servers by the hour with no capital required, no down payment.  Simply pick what type of servers you want to spin up, and spin up as many as you want.  Pricing ranges from $.10 per hour for a 1-cpu, 1.7gb ram server to $.80 per hour for an 8-core instance with 7-15gb ram.  As load grows, spin up more servers to handle the load.  If the load doesn’t come, or if it comes and goes, just spin down the extra servers.  They’re not sitting around on your balance sheet draining your company dry.

Cloud computing lets anybody with a good idea, good architecture and good coders scale without investing in hardware ahead of time.  Suddenly, those three guys working in a garage (or a coworking space) are much more dangerous to your established - and funded - internet business.

If I worked for Twitter, I would be terrified that a startup company with no VC funding would be able to steal 10,000 of my users from my precious web application in 3 days.  I would be standing in front of my web servers with my arms outstretched, screaming, “No, don’t take my children!”

I don’t work for Twitter, though, so you can catch me on Identi.ca.

AMD triple-core servers and SQL Server 2005

If you’re using AMD Phenom 3-core processors, SQL Server 2005 won’t install without going through some hoops first.  Microsoft just published a knowledge base article on SQL 2005 errors on 3 and 6 core servers.

I don’t usually blog about knowledge base articles, but I got such a laugh out of that one that I just had to share it.

Want to know about MS KB articles as soon as they’re published?  Subscribe to the Microsoft Knowledge Base RSS feed for SQL Server.  You DO use an RSS reader, don’t you?  Hint hint….

Transparent Data Encryption in SQL Server 2008

SQLblogcasts.com has a great tutorial about SQL Server 2008 Transparent Data Encryption. If you want to learn how to encrypt your databases, read that article.

I played around with TDE a couple of weeks ago, and I was surprised by how difficult it is to implement.  I’d expected to be able to check a box, put in a password, and click OK, but it’s nowhere near that easy.  Restoring encrypted databases from one server to another can also give DBAs a nasty surprise when they least expect it.

I’ve heard several DBAs comment recently about how SQL Server Management Studio is targeted more at developers than it is database administrators, and I think 2008 will reinforce that perception.  Implementing TDE is a good example: there’s no wizard, there’s no obvious steps, etc.  Right-click on a database and try enabling encryption, and there’s no obvious reason as to why the feature is disabled - the DBA has to dig through documentation to find out that a server certificate is required first.  Ugh.  TDE is a good first step towards secure data files, but any toddler will tell you that those first steps are always the toughest.

Russia as seen from my hotel bathroom

I’m spending a few days in St. Petersburg, Russia to visit Quest’s office.  Sure, I have the regular touristy photos, like myself in front of old buildings:

But what I really like about overseas travel is that everything looks and feels different, and it forces me to look at the design and user interface for everything.

Take toilets, for example.  American toilets have the button on the side, and Russian toilets have the buttons on the top.  That’s right, buttons - more than one:

My first thought when I saw this in my hotel room was, “Ah, that’s interesting.  Now what would be the advantages or disadvantages of putting the button on the top?”  I guess you can’t leave things on top of the toilet if the button is on the top.  With American toilets, you can just pull the lid off, and the button remains on the side, so it’s probably easier to design the lid.  This particular toilet lid doesn’t come off at all, which is an advantage in hotels - tourists hide all kinds of stuff in toilets.  I know this from my years of working in hotels.

My second thought was, “Two buttons?  I’m screwed.”  I’ve been in Europe just enough to have seen bidets, and heard about toilets with built-in bidet attachments.  I half expected water to come shooting up and hit me in the eye, so I stood to the side when I tested the buttons.  Turns out the two buttons have two different amounts of water: regular, and Niagara Falls.  I think I could flush a small cat down this toilet.  Now that’s cool.

Are American toilets better than Russian toilets?  No, they’re just different.  The differences continue throughout the bathroom, like the hooks on the wall:

These have a fun, playful design language, very mid-century-modern.  These would look awesome in an American 1950’s ranch-style house, but I’ve never seen anything like them before.  I certainly wouldn’t expect to see them in a traditional hotel like ours where the paintings on the wall are framed with mock gold leaf.  The hotel is very conservative, and then these bathroom hooks say BAM!  Bathroom fixtures seem to be more modern-looking in Europe and Russia, and more traditional in America.  Dunno why.  Not better, just different.

But wait - there’s more.  Check out the toiletries in my room, but look closely and see if anything looks odd:

In the white box, there’s a nail file.  A nail file!  Wow.  I’ve never seen that before.  I’d be curious to know how often people need a nail file when they travel.  Different.

Plus, the labels are all in English.  Not Russian, just English.  Know why?  Because only Americans are dumb enough to travel without all of their stuff.  And I should know - last week I went to California without any socks, and this week I came to Russia without my electric razors.  Which brings me to this:

It takes a big man to admit he’s scared, and I’m a mighty big man, so I don’t mind telling you: razor blades scare the hell out of me.  There must have been some terrifying experience in my childhood involving razor blades, because I never touch them.  When I was old enough to shave, I bought an electric razor, and I’ve always used them ever since.  Normally, if I forget my razor when I travel, I go buy another one.  But here I am in Russia, and if I bought an electric razor, it’d have their electric plugs, and I wouldn’t be able to use it at home without an adapter.

I said to myself, “You’re so interested in things that are different - why not give it a try?”  After all, I’ve seen Queer Eye for the Straight Guy.  I know how the concept works.  So with shaking hands, I lathered up my face and used a sharp piece of metal to scrape hairs off my face.  (Can you tell this concept bothers me?)

And you know what?  I liked it.  It gives a really nice, close shave, and as long as you lather up with warm water and use a sharp razor, it doesn’t cut your face.

That’s why I like travel.  Different can be better.  I might even stick with razor blades.

Quickly replying to recruiter emails

In today’s hiring market, DBAs get a lot of emails from recruiters that say things like:

“I am curious to know if you would be interested in (or know someone who is interested in) relocating to Big Falls East for the position discussed below….”

The job description never includes the salary range, and it rarely includes information about the industry - two things that should be important to a job candidate.  (If you don’t care about the industry, you should start caring - imagine going to work for Ford Motors, an airline, or a mortgage company in today’s economy.)

To handle these emails fast, go into your email program and create a new signature.  The signature should include a default reply text that is really generic, like this:

“Thanks for the email.  If you can forward on information about the salary range and the industry, I’ll pass it along to my peers.  After being repeatedly burned (hey, buddy, check out this job - oh, sorry, I didn’t know it only paid $X) I have a policy against forwarding jobs without salary & industry information.  Thanks, and have a great day.”

Most email programs will let you quickly change your signature for one specific email.  That way, when recruiter emails come in, just hit reply, insert your recruiter signature, and hit send.

When the next email comes in (either with the right info, or without it) you can handle it the right way.  If it includes good information, then it’s a good recruiter, and you should pass it on to your buddies.  If it doesn’t include any more information than the first email, just hit delete, because the recruiting process is off to a bad start.

I know recruiters will disagree, but here’s the deal: if a recruiter is desperate enough to send a blind email to a stranger without a resume, asking that stranger to pack up their house and move to another state, then there’s a reason.  They’ve already exhausted the local candidates - either because they’re not paying enough, the company has a bad reputation locally, or they need a hard-to-find skill set.  Think about that before replying with your resume, because the first two reasons will come back to haunt you, and the third reason means you’re worth a lot of money.

Quest LiteSpeed isn’t for everybody

Sean McCown at Infoworld wrote a blog post today that said some negative things about Quest and LiteSpeed, and I had to respond. He’s switched away from using LiteSpeed, and he had an interesting set of reasons.

“I just don’t need the centralized repository in my lab, and I do enough demos that having native SQL syntax for my backups is worth something to me.”

Everybody has different needs. Sean and I would totally agree that labs have a different set of needs than a typical enterprise or datacenter.

In fact, if I was doing a lot of demos like Sean, and if I wanted native SQL syntax for my backups, I wouldn’t even use a third party product at all: I’d use SQL Server 2008’s new backup compression to make my life really easy. That way, you can stay purely native, and that makes for really easy demos.

Quest LiteSpeed, on the other hand, isn’t targeted just at labs and demos. It’s targeted at DBAs who need enterprise-ready backup solutions, complex needs like log shipping, centralized reporting, log reading with undo & redo for transactions, and so on. Not everybody needs the advanced features that LiteSpeed provides, and Sean doesn’t. We would both agree that LiteSpeed is not the right product for his lab.

“Even knowing any of the devs is hard these days, and getting anyone to make any changes is next to impossible.”

At smaller companies, especially startups with a minimum of customers, developers can work directly with end users. For better or worse, Quest has a lot of customers, and I’d like to think that’s because Quest has fantastic products. (It’s a pretty good problem to have.) Our product managers coordinate feature requests across all of our enterprise customers, large and small, which helps our developers focus on what they need to do: develop great software.

Just this week, in fact, Quest Software hosted a SQL Server Customer Advisory Board at the offices in Aliso Viejo. We flew out a dozen customers for two days of meetings about our road maps, upcoming features, and asking for their input on where we go next. I wish we could bring every Quest customer out there for the same input, but on the other hand, Steve Ballmer isn’t flying me up to Seattle anytime soon to ask me what SQL Server needs to do next.

We’re in a feature freeze for our new LiteSpeed v5.0. The management tools have been rewritten from the ground up, and I’m really impressed. But there comes a point where you have to stop adding new features and just focus on testing the code, and that’s where we’re at now. Heck, I have a long list of things I want to add, but even I can’t sneak ‘em in - and I work for Quest! I’m already excited about the features I’m trying to cram into v5.1.

“Now, with the team being more corporate, and a LOT less responsive than they used to be, Hyperbac is a good choice for me now.”

If your main criteria for a backup product is easy access to the coders, then Quest LiteSpeed is not the product for you.

Just a couple of months ago when I was a DBA instead of a Quest employee, though, that was not my main criteria for software selection. My main criteria included a global 24×7 support organization with experience across database technologies, a large QA team dedicated to testing mission-critical backup software on every possible platform and a product with an unbelievably strong track record. The fact that Quest is Microsoft’s Global ISV Partner of the Year again this year is proof that sometimes bigger is better.

US Airways Hung Up On Me. True Story.

I used to live out of planes and hotels (literally, I worked for a hotel company) but I haven’t traveled much for the past five years.  This afternoon I had an ugly experience that reminded me of the sad truth of long-distance travel: crappy customer service can wreck your trip, and great customer service is worth extra money.

I tried to check in via the web around 1pm for my 3:45pm US Airways flight back from the Quest offices in Orange County, CA.  The web site informed me that something had changed about my flight, and that I needed to call the US Airways customer service number.  I called, gave them my confirmation number, and got placed on hold.  A few minutes later, the voice returned and informed me that my flight had been cancelled.

Period.

I waited to hear the next words out of her mouth, because I’m thinking, this is typical, right?  These things happen.  It’s a big company.  They must have a system for this, a checklist that defines what step happens next.  I know things break, and I’m cool with that.  When I’m traveling and things break, I’m really cool when it happens on the ground.  It’s when things break in the air that I get fidgety.

But she doesn’t keep going.  To her, that was it.

I prompted her by asking, “Okay, what next?  Is there a later flight, and can you rebook me?”  She put me on hold for a few more minutes to go find out.

While I was on hold, I started wondering what she’d done the first time she went on hold.  Was it that hard to find out the flight was cancelled?  Why make me wait that long and not have an answer ready for what the next question would be?  Did she think I was just going to stay in Orange County for the rest of my life?  Had any customer in the entire history of US Airways ever been told their flight was cancelled and simply responded with, “Okey dokey, sounds like we’re done here!  Thanks!”

She came back and said the next flight would be the next day.  I said okay, no problem, go ahead and book me.  She asked to put me on hold again, but I asked if she could skip putting me on hold, because my cell phone battery was almost dead.  She put me on hold anyway.

When she came back, she said I was all set to fly out the next day.  I asked which hotel they were putting me up at, and that’s when things went south pretty fast. To be honest, I didn’t expect them to put me up in a hotel, but I was cheesed off enough that they’d cancelled the flight without calling me, without rebooking me on another flight, and without seeming like they had any semblance of a Plan B when I called.

She started talking to someone else in the same room, escalated it to a supervisor, and didn’t do a particularly good job of holding her finger over the microphone.  Actually, to be honest, I think she was passing the microphone around to the different people in the room, because I heard them pretty clearly.  One of them said something to the effect of, “He said his battery was almost dead, right?  Hang up on him.”

And they did.

Rather than call ‘em back, I called the Quest travel department, who performed like rock stars.  Within the first two minutes of the call, she identified that all of the flights out of Orange County, LAX and Long Beach were booked, with the exception of an LAX flight that I couldn’t make in time.  She got me booked on a 6:15 AM flight, booked me into an airport hotel, emailed the updated confirmation info to me and asked if there was anything else she could do to make it easier.

All in less time than the US Airways rep had me on hold before she told me the flight was cancelled.

The taxi dropped me off at the Hyatt, and it turned out that the cab went to the wrong Hyatt.  Jessica at the Hyatt Regency Irvine front desk said no problem, called the Hyatt Regency Newport Beach, got the reservation moved, kept the same discounted corporate rate that Quest had at the other Hyatt, and kept apologizing the whole time about my cancelled flight.

I got into my room, called down for room service, and the great service kept going.  The operator cheerfully greeted me by name, and she told me when the food would be ready - but she did it in a way that I rarely hear.  She said, “We’re running about 30 minutes right now, so your food should be there just before 4 PM.”  Usually, I hear things like, “It’ll be half an hour to an hour depending on how busy they are.”  They - what do you mean they?  It’s you.  You’re they.  At the Hyatt, there was no “they” - it was “we.”  I love that.  And you know the food showed up in 25 minutes, because that’s the kind of operation they’re running.  Wow.

So yeah, I’m stuck in California for another night, and I miss Erika and Ernie, and I wish I was home sleeping in my own bed.  But if it wasn’t for the happy folks I dealt with after the wackos at US Airways, things could be a lot worse.

Free SQL Server tools and utilities

Mark Garner came across a great list of free SQL Server tools & utilities.  I took one look at that page, bookmarked it, and realized it’s going to take me a week to go through that list!  Good stuff.

Developers: sometimes you just gotta trust the DBA

At this week’s Customer Advisory Board at Quest, I’ve had a great time sharing war stories with other database administrators.  One theme keeps popping up again and again: developers don’t trust DBAs and seem to refuse to take the DBA’s word for anything.  It seems that developers just want to keep learning things the hard way.

Case in point: a developer wants to loop through all string fields in every database in the shop looking for secure information.  They’re after things like social security numbers, credit card numbers, bank account numbers and so forth.  They want to check the contents of every char/varchar/nchar/nvarchar field in every table, look at the top X records, and check the string contents to see if it matches known text patterns.

In theory, this could be solved by a stored procedure.

In practice, a stored proc would be a bad idea.  SQL Server isn’t particularly good at string processing, and a query like this will instantly push the CPU to 100%, thereby severely slowing down other queries on the box.  Plus, large databases like SAP can have tens or hundreds of thousands of string fields, and storing the results will take gigs of memory and/or TempDB space on a large BI/BW data warehouse.

The DBA involved has repeatedly recommended against the stored proc approach, but the developer still wants to build one, saying he’s Googled for ways to do it and he’s getting some good leads.  At what point  the DBA will sit back and say, “You know what, go for it, because when your stored proc takes down a production box, I’ll be all too happy to explain whose code did it.”

Another case involved developers who want to store completely unstructured XML data inside the database as text fields, and want to parse through that XML data when searching millions of records for matching attributes.  The XML format will be extremely flexible, but when storing it as text, it’s just not going to perform.  The developers see that as a SQL Server performance problem, when in reality it’s a bad design that won’t scale.

We spent some time tonight at the CAB talking about where the DBA draws that line - where he gives up trying to protect the developer and just gives the developer enough rope to hang themselves.  It’s an interesting challenge politically, but the moral of the story is this: developers out there, if your DBA keeps recommending that you do something, and all of a sudden one day he caves in and sees things your way, be careful.  You might be in for a rocky ride.