The Cloud is Coming After Your Children!

I blogged about why database administrators should use Twitter, and I personally loved that tool.  It helped people build faster personal connections with each other, and it helped you get a picture of what the brilliant people were thinking at any given time.  Take Brian Knight, a SQL Server guru, for example - I love being able to see what he’s working on now, because he does things that shape how the DBA world works.

Unfortunately, Twitter started running into some serious infrastructure and architecture problems.  They built the whole thing on top of a couple of MySQL servers with no real Plan B - no way to scale out, no automated cluster failover, no partitioning, and basically no way to make the database fast enough to withstand the user load.

All of this was frustrating, especially since early Twitter adopters tended to be people who worked in & around information technology.  We do this stuff for a living, and it frustrates us when we depend on a tool that is constantly borked.  I got a few laughs out of Twitter’s scaling issues - not because I thought they were easy to solve, but because I’ve worked with coders who took that same approach.  How many times have we heard, “But it works fine on my desktop with ten users….it’ll be fine when we move it to a real server and we have a hundred thousand users.”

Here’s a couple of links with more information about Twitter’s struggles:

• Analysis of Twitter’s infrastructure problems
• Current Twitter Status (color coded by whether features are up or down.  Example shown at right.)
• Twitter scores its first round of VC funding in 2007
• Twitter scores a second $15m round in 2008

That’s right, fifteen million bucks.  A company with a newly minted $15m in the bank should be able to build in some scalability pretty quickly, right?  Buy some faster servers, build in some redundancy, off we go, right?

Not so fast.

Architecture problems take time - especially downtime - to fix, and the one thing Twitter didn’t have was time.  More users piled onto the service every day, more applications built hooks into Twitter’s open API, and the press started asking questions about this cool new service.

Smelling blood, a lot of other competitors have sprung up around Twitter.  Jaiku, Pownce, Friendfeed and many more sites launched to do similar things.

On July 2nd, a new competitor called Identi.ca launched and gathered a lot of press right away.  I’m not going to go into most of the technical differentiators here, but the bottom line is that Identi.ca picked up over 10,000 new users in its first 3 days.

Let me repeat: Identi.ca picked up over 10,000 new users in its first 3 days.

That’s a big rush, and it requires a lot of hardware.  Hardware is expensive.  Datacenter space is expensive.  These things require a lot of VC funding up front, plus planning time to buy, install and provision.  Nobody wants to pay for users they don’t have yet, especially in this tight economy, so how was Identi.ca able to scale up that fast?

Identi.ca is hosted on Amazon EC2 cloud servers.  Anybody can rent EC2 servers by the hour with no capital required, no down payment.  Simply pick what type of servers you want to spin up, and spin up as many as you want.  Pricing ranges from $.10 per hour for a 1-cpu, 1.7gb ram server to $.80 per hour for an 8-core instance with 7-15gb ram.  As load grows, spin up more servers to handle the load.  If the load doesn’t come, or if it comes and goes, just spin down the extra servers.  They’re not sitting around on your balance sheet draining your company dry.

Cloud computing lets anybody with a good idea, good architecture and good coders scale without investing in hardware ahead of time.  Suddenly, those three guys working in a garage (or a coworking space) are much more dangerous to your established - and funded - internet business.

If I worked for Twitter, I would be terrified that a startup company with no VC funding would be able to steal 10,000 of my users from my precious web application in 3 days.  I would be standing in front of my web servers with my arms outstretched, screaming, “No, don’t take my children!”

I don’t work for Twitter, though, so you can catch me on Identi.ca.