Is Big Data Doomed?

Or, Are Technology Best Practices Timeless?


A funny thing happened on the way to the mainstreaming of Big Data: it started to obey the laws of gravity. At first, this manifested in reports of significant project failure rates; more recently, it’s materialized in disappointing success rates.

What’s going on? Can we deconstruct this? Also, what’s not going on? Is it possible that some issues are being overstated?

Data is Still a Priority

First off, everyone needs to take a deep breath. Data is important, and always will be. Understanding data is also important, and necessary. Data, after all, are just point-in-time recordings of events or transactions in the life of a person, a group or an organization. In business, data is important because the business is important. Understanding data is important because understanding how the business has run, should be run and will run, is critical.

But if requirements around data and analytics are of such high priority, why does it seem like things are stalling? Some understanding of the “constants” in the world of software can help.

Data Questions to Ask Before Starting

Before any technology can be useful, customers need use cases, and they need to focus on the technology-agnostic aspects of those use cases. This is axiomatic, across all categories of software and technology.

Before you push any buttons, click any icons or ingest any data, think through the answers to some questions.

  • What is the area of the business whose data you wish to analyze?
  • What are the data sources that are most germane?
  • Which people or entities can help get access to those data sources?
  • Can you preview the data, in Excel or another general-purpose tool, before the project gets started and get a rough sense of its structure and quality?
  • Can you identify a subject-matter expert who can help you make that determination and give you pointers on how to mitigate and resolve data quality issues?
  • Do you know what insights you’re hoping to derive form the data?
  • Which ones would be of high value, as opposed to just academically interesting?

Plan For Your Big Data Project

As you answer these questions, two things should become apparent:

  1. There is a ton you can learn about — and plan around — your analytics needs, before you ever touch a tool.
  2. Going through this exercise should make you excited and motivated to embark on your project.

Now that we’ve outlined some steps around planning an analytics project, it’s a good time to point out that many customers don’t go through these steps before a project starts. And by trying to skip these steps, the chances of project success are significantly reduced.

All customers will have to go through these steps anyway. But by doing it in a planning phase up-front, efficiency, motivation and inspiration are maximized. And those are critical precursors to good outcomes. Conversely, doing these steps on the fly causes efforts and motivation to be fragmented and morale to be low.

How Marketing Fails Big Data

Many customers aren’t doing this. That contributes to low project success rates, which begets low adoption. But the interesting thing is this lack of a disciplined approach often isn’t the customer’s fault. It’s the market’s.

Vendors spend a lot of time talking about the ease of use of their products. They spend even more time talking up their products’ individual features and capabilities. Much of this marketing seems to convey a message that identifying use cases and planning how to attack them isn’t necessary. Phrases like “visual,” “self-service,” “point-and-click” and especially “agile” can make it seem like planning is overly formal and superfluous.

But software doesn’t run itself. And insights don’t just happen. Like physical tools, software tools work best when you know what you want to do with them and you know what you want to build. In the world of big data, planning and organization are more important than ever.

The Difficulty of Choosing Your Big Data Solution

One more problem the market has imposed on customers (and thus on itself) is the vast array of big data technologies and products, and the overlap between them. The result is a market that is customer-hostile, because of the inflated permutations of products and technologies.

Start your free trial of Datameer to learn more

That added surface area of complexity and failure is a turn-off for customers, with a risk-reward ratio that’s unattractive at best. The knock-on effect on customer adoption of the technology can be severe, and with hindsight, should be obvious. Why is the industry hamstringing itself like this?

Big Data Best Practices

Vendors (and their investors) can lose sight of the fundamentals just as easily as customers can. Their planning and priorities can get off kilter too. But vendors and customers need to work together to improve the status quo. Here are three steps to get there:

  • Customers need to have well thought-out, vetted, detailed use cases, buy-in on those use cases from stakeholders and pledged assistance from parties who can enable the plan’s success.
  • Vendors should concentrate on taming and simplifying the market for customers, by integrating and abstracting various underlying open source (or proprietary) technologies.
  • Investors should avoid funding and enabling companies and products that largely duplicate capabilities that have already been addressed. Priority should be given to companies and products that are adding value, by making big data technology less complex, more accessible and thus making customers more successful.

Is this all that’s necessary to move on? Probably not.

But if the actors in this industry heed the above prescriptive advice, significant improvement will come. Innovation is exciting, but without dedication and discipline, we’re just taking business problems and throwing tech at them. That may work to build interest, momentum and initial support, but it won’t sustain growth and it won’t bring necessary maturity.

We’re at a crossroads, and we can move on. We just need a greater focus on value and success.

Free Trial Datameer

Connect with Datameer: