Let’s bust the buzzword bubble. Big Data is a sexy problem to have, so everybody’s claiming it. I’m sick of people using this phrase to gloss over their real challenges.
You have a Small Server problem. If you haven’t bought a new server in the last two years, you’re not allowed to complain about the size of your data. You can buy a 4-socket server with 1TB of memory for less than what it’ll cost to bring in a fancypants Big Data consultant for a month or two.
You have a Slow Storage problem. That massively expensive SAN you bought five years ago is eclipsed by a single consumer-grade SSD today. Run CrystalDiskMark on your SSD-equipped laptop, and then run it on your database server. If the laptop’s faster, does that mean your MP3 collection is Big Data?
You have an Awkward ETL problem. Call it “unstructured data”, but it’s really sloppy data modeling. It’s completely okay to save money on design and pass the savings on to – wait, actually, you don’t save money. If you don’t structure your data, everything else you do costs more money. Anytime you want to interact with that data, it’s going to cost extra – but it’s not the size of the data that’s the problem.
You have a Small Documentation problem. Don’t think that you can hire one 18th Level Data Scientist Mage and suddenly get amazing insights from your data. The first thing he’ll need to do is talk to the rest of your staff about the sources of the data, the timing, and the quality. They won’t remember offhand why the mainframe’s Gross Net Subnet Tax Top Line is different from the accounting system’s Net Gross Top Tax Subnet field. You can’t Hadoop your way out of that one.
You have an Small Momentum problem. Every few years, there’s a new buzzword to describe how your data is going to become magically actionable. Business Intelligence! No, wait, Self-Service Business Intelligence! No, wait, Data Visualization! No, wait, Big Data! Before you embark on the next one, take a critical look at why past data initiatives in the company have failed, and I bet it doesn’t have anything to do with the data’s size.
When someone wants to talk Big Data to you, ask what they’re trying to sell you. Odds are, they’re really trying to solve your Big Wallet problem.