Comments on: Whither Hadoop?

By: Jeremiah Peschka

Jeremiah Peschka — Wed, 19 Dec 2012 18:53:38 +0000

You’re absolutely correct!

There is a lot that could be said in a much more specific way in this post. If I were talking to a non-Microsoft audience (or assumed that they had more exposure to the ecosystem), I’d be much more specific. Inside the Microsoft community, the entire ecosystem of Hadoopery is still frequently referred to as Hadoop. Sometimes you have to start vague and bring in specifics over time.

Thanks much for contributing, though, the differentiation is key.

By: Josh Reuben

Josh Reuben — Wed, 19 Dec 2012 17:20:27 +0000

This post comes across wrong to me.
Hadoop is not an ETL tool, its a MapReduce stack for scatter-gather-aggregate scaleout of compute jobs. If you want to compare it to something on the MS stack, it is more akin to HPC than SSIS !

By: Links

Links — Wed, 19 Dec 2012 10:43:52 +0000

Huh, I’ve been trying HD-Insight the last few weeks, which is MSFT & Horton’s Hadoop distribution for Windows Server (local not cloud based), and you could submit Map/Reduce jobs using C# rather than Java (I think u could do the same with the Azure based Hadoop distribution too).

To be honest I don’t think we should limit ourselves because we haven’t played with a particular technology before, Java (or generally any popular enough programming language) is based on sound programming principles that can easily be picked up… Am having to learn Python these days for some social network data mining, and it is soo awesome, sometimes its good to step out of your comfort zone.

My only beef with the MSFT local Hadoop installation is that it does not yet support Mahout, which is only supported in the Azure distribution.

By: Jeremiah Peschka

Jeremiah Peschka — Tue, 18 Dec 2012 22:22:09 +0000

Hi Jay,

While some Hadoop development is done in Java, much of the day-to-day business data analysis happens in tools like Pig, Hive, and Impala. These tools make it possible for non-Java devs to pull meaningful data out of massive data systems and, very frequently, push it in a distilled form back into something like SQL Server or SSAS. Developers are typically only dropping down into Java when it’s absolutely necessary or to implement custom user defined functions.

If you’re interested in seeing more of the Java side of development, I suggest you check out one of the many Apache hosted mailing lists (http://hadoop.apache.org/mailing_lists.html) to see how other people are working with Hadoop.

By: Jay

Jay — Tue, 18 Dec 2012 22:04:17 +0000

Hadoop is very good product but and not useful to DBA or any other person unless he or she is TRUE JAVA developer. Hadoop is developed using 100% Java

So i would like to know & learn more about Hadoop been used currently. what i see that SQL Server will be front end for Hadoop & you need developer who can develope good code using java to develop bridge between Hadoop & SQL Server.

Thanks
Jay-

By: Joe Harris

Joe Harris — Wed, 12 Dec 2012 16:17:57 +0000

At core Hadoop is a way to execute logic in the language of your choice over raw data on multiple machines. It’s all just files – no rows, no tables, no records, no tuples – what you do from there is entirely your responsibility.

With all respect to Jeremiah, I’d suggest those interested in Hadoop should also read “What Hadoop Is. What Hadoop Isn’t” by Mark Madsen at http://www.insideanalysis.com/2012/12/what-hadoop-is-what-is-isnt/.

A quote: “The Hadoop stack is a data processing platform. It combines elements of databases, data integration tools and parallel coding environments into a new and interesting mix.” “It combines data storage, retrieval and programming into a single highly scalable package.”