About Us Icon About Us Icon Business Analyst Icon Business Analyst Icon CEO Icon CEO Icon Datameer Icon Datameer Icon Envelope Icon Envelope Icon Facebook Icon Facebook Icon Google Plus Icon Google Plus Icon Instagram Icon Instagram Icon IT Professional Icon IT Professional Icon Learn Icon Learn Icon Linkedin Icon Linkedin Icon Product Icon Product Icon Partners Icon Partners Icon Search Icon Search Icon Social Networks Icon Social Networks Icon Share Icon Share Icon Support Icon Support Icon Testimonial Icon Testimonial Icon Twitter Icon Twitter Icon

Datameer Blog

The State of Hadoop: What’s Next?

By on August 17, 2015

**This post originally published on InsideBigData**
Everybody wants a piece of the big data pie – particularly Hadoop. Startups are popping up left and right in attempt to be a part of the Hadoop action and industry watchers are fueling the buzz — and for good reason.

Hadoop has emerged as the leading software framework for the storage and analysis of big data. Early adopters such as Facebook, Twitter and Yahoo! have successfully built custom analytics using Hadoop to tackle big data analytic challenges. Given this initial success, Hadoop has become the poster child for delivering scalable analytic powers that meet today’s big data requirements, and companies are biting at the opportunity to benefit from that potential.

Yet with the growing buzz surrounding Hadoop so comes the skepticism. While it would be ludicrous to doubt the value of data and its ability to create high-resolution observations and interpretations about how businesses are performing, it’s time to ponder how to bring big data technologies, such as Hadoop, into the next phase of efficiency and utility. In order to do that, we must understand what’s driving the skepticism that’s out there, and how to address it.

Who’s Jumping on the Hadoop Bandwagon?

Looking at the big data landscape, one of the obvious observations is the increasing number of startups focused on Hadoop. When you have a unique shift in the market like the one brought on by big data, it’s inevitable that startups will want to jump on the bandwagon. If there’s a great opportunity, Silicon Valley and the world of emerging technology will always try to capitalize on it.

With all of this hype, people question whether there is a growing Hadoop bubble and ahead-of-time expectations. We’ve seen Hadoop-related companies leave the gates with initially promising growth numbers and then stagnate early on. People are starting to question if Hadoop is worth all the fuss.

Those looking at the Hadoop landscape need to recognize whether there’s value creation in the company or if it’s a matter of unlimited funds that’s being used to buy growth. There are Hadoop-related companies that create tremendous value, have solid bookings and revenue numbers — that is where the potential lies. On the flip side, there are also companies where growth is mostly bought — that is where the potential dies.

Bring Something New to the Hadoop Game

The companies that are giving rise to the doubts around the promise of Hadoop are those whose technologies are haphazardly duct taped together and offer little promise of sustainability. They are not delivering emerging, next-generation technology; rather they are selling the same thing with an added twist and a Hadoop sticker slapped on it.

It’s important that companies truly bring something new to the game. If they can pull this off, customers will pay for the product and growth will flourish.

Hadoop is an open-ended technology with lots of capabilities and ways to leverage them. However, organizations are stumbling around trying to get Hadoop projects off the ground because it lacks end user tools and standardizations and has a steep learning curve. Technology that can address these stumbling blocks and make Hadoop more accessible to users will prevail.

Hadoop-related technologies need a quality interface that is comparable to that of other business software. Just because Hadoop can do things that conventional databases and Business Intelligence (BI) can’t, doesn’t mean it gets a pass on usability or productivity. While technical users may be more accepting of these shortcomings, satisfying them is no longer good enough and even they will lose confidence in a system that has low usability for their colleagues. Hadoop and big data technology has to be user-friendly and self-service orientated.

Likewise, the Hadoop market needs more standardization. The goal should be to make Hadoop-based tools easy to use, yet the infighting among platforms is creating a muddled marketplace. As more and more Hadoop-based applications emerge, there will be an increased demand for standardization from customers and they will gravitate toward technologies that have standardization weaved into their ethos.

While there may be an air of suspicion that threatens to taint the outlook on Hadoop, now is not the time to be a big data naysayer. The value in Hadoop-based technologies is apparent and it is strong. The companies that continue to build upon that value will be the ones that thrive in tomorrow’s Hadoop ecosystem.


Want to learn more and join the conversation? My colleague Andrew Brust and I will be talking about this in a live webinar on August 20th. Register here to join us, or, if you can’t make it live, we’ll send you the recording.

Connect with Datameer

Follow us on Twitter
Connect with us on LinkedIn, Google+ and Facebook

Stefan Groschupf

Stefan Groschupf

Stefan Groschupf is a big data veteran and serial entrepreneur with strong roots in the open source community. He was one of the very few early contributors to Nutch, the open source project that spun out Hadoop, which 10 years later, is considered a 20 billion dollar business. Open source technologies designed and coded by Stefan can be found running in all 20 of the Fortune 20 companies in the world, and innovative open source technologies like Kafka, Storm, Katta and Spark, all rely on technology Stefan designed more than a half decade ago. In 2003, Groschupf was named one of the most innovative Germans under 30 by Stern Magazine. In 2013, Fast Company named Datameer, one of the most innovative companies in the world. Stefan is currently CEO and Chairman of Datameer, the company he co-founded in 2009 after several years of architecting and implementing distributed big data analytic systems for companies like Apple, EMI Music, Hoffmann La Roche, AT&T, the European Union, and others. After two years in the market, Datameer was commercially deployed in more than 30 percent of the Fortune 20. Stefan is a frequent conference speaker, contributor to industry publications and books, holds patents and is advising a set of startups on product, scale and operations. If not working, Stefan is backpacking, sea kayaking, kite boarding or mountain biking. He lives in San Francisco, California.