About Us Icon About Us Icon Business Analyst Icon Business Analyst Icon CEO Icon CEO Icon Datameer Icon Datameer Icon Envelope Icon Envelope Icon Facebook Icon Facebook Icon Google Plus Icon Google Plus Icon Instagram Icon Instagram Icon IT Professional Icon IT Professional Icon Learn Icon Learn Icon Linkedin Icon Linkedin Icon Product Icon Product Icon Partners Icon Partners Icon Search Icon Search Icon Social Networks Icon Social Networks Icon Share Icon Share Icon Support Icon Support Icon Testimonial Icon Testimonial Icon Twitter Icon Twitter Icon

Datameer Blog

Top 3 Big Data Pain Points for Energy Companies

By on June 10, 2015

I recently came across this great article that shares how Shell is using big data analytics to improve oil well efficiency. Having worked with Energy companies, (primarily Upstream and Midstream) the last 15 years, I’m acutely aware of the pain points they face when it comes to harnessing all the data they’re creating. Speaking with other companies in the space, its clear to me there are some significant pain points all oil & gas companies have in common.  I am going to list 3 for the sake of brevity, but this could be the beginning of a very interesting discussion.

1.  “If it isn’t broken…” – With the price of WTI currently hovering just under $60, certainly capital budgets have been affected for FY15, and who knows what the future holds for next year, but budget planning is in-process.  In working with operations and IT liaisons, a common sentiment is, “We don’t want to do something too unconventional and we need to ensure we stay in line with our competition… but… we believe we need to do something with our data – we have tons!! – just not sure what to do or where to start…”

Taking on anything new can certainly be perplexing and at times intimidating, but Datameer has certainly made it significantly easier to tackle big data initiatives quickly.  My discussions usually have to do with “oceans” of sensor data (historian) or unstructured data and where to start, but, an easier initial path would be starting with some analysis around Well Spacing and Completions.  Rather than blaze a new trail right out of the gate, you can get a quick win with this use case and realize some tangible value in 4 – 6 weeks (to production, mind you), then tackle some of the challenges that might be more unique to your organization. 

2.  “Who is on first..?” – Who ultimately owns the initiative?  Big Data (Hadoop) certainly falls into the IT camp, but in terms of value and consumption, it should be accessible to a large group, namely, and ideally in our opinion, the business analyst community.  If IT has to do everything so the business can ultimately consume it, well, the Software Development Life Cycle (SDLC), is essentially a mirror image of the BI paradigm of old.  Too many “hops”, and groups, to ultimately get the information needed to make decisions quickly.

I had a conversation with a prospect (IT department) and the focus was, “… we just need to turn around BI reports faster than 90 days”. I was somewhat aghast at that predicament, but I held my tongue.

Datameer enables a clear segregation of duties that empowers the business to ask industry-specific questions, without:

• Creating a use case spec
• Taking that to an IT/Business liaison for translation
• Taking that data to an ETL team to ensure schemas, mappings, and MDM initiatives are in place to enable what is being asked
• Going to traditional BI team for translation
• Working with the DBAs on the backend to once again validate schemas, foreign key constraints, potential ripple effect, etc.

All the while the person in operations says, “All I want to know is, ‘what time is it??’ not ‘how the watch is made’”. I am not over simplifying the questions that the business is asking, but I do believe, in my experience, they are on point as many of the O&G professionals I work with still recall the Oil Bust of the 80s with a clear mind. They know what they want and need, and Datameer can empower them to get the answers rather than going through the aforementioned “checklist”.

3.  “Where do I start…?” After an hour-long discussion, 9 times out of 10 the professionals I meet with agree with the assessment of the predicament and want to “do something”.

Start with the end in mind.

This might seem silly compared to a traditional approach, but that is the beauty of Big Data.  It is easy to ingest large quantities of data into your Hadoop cluster of choice, then I encourage you to “break glass”.


You can’t hurt the data.  Ask questions and use the tool to see how quickly you can get answers you already know to be true.  If that approach works, then new questions and horizons are all fair game AND if you fail, it will be quick!  You won’t work for months, weeks or days only to find out your efforts are in vain.  Datameer in concert with Hadoop allows you to answer questions quickly and “fail fast”.  If you aren’t “skinning your knees” on a daily basis, well, you probably aren’t in the game.

The great thing is, the learning curve being what it is, you will find very quickly that you arrive at a point where those painful falls or failures are less and less frequent.

The use cases are out there just looking for a good home. So let’s go. If you want to learn more about how we’ve helped other Oil & Gas companies, be sure to read this detailed case study about how a Fortune 500 O&G company realized over $100M in incremental revenue using Datameer. 

Connect with Datameer

Follow us on Twitter
Connect with us on LinkedIn, Google+ and Facebook

Matt Boone