Today I’m excited to share how a 100+ year-old revered retail brand is reinventing itself for the newer generation of retailers (and e-tailers!) when it comes to leveraging Big Data analytics + Hadoop. You might have read about them in a great article by InformationWeek’s Doug Henschen, but today we’re very excited to share our video interview with the CTO of Sears, Phil Shelley:

We’re excited to kick off this series of blog posts where we’ll highlight what some of our customers are doing, and the benefits they are realizing. We’ll start with a short video from the guys at TasteMakerX, who talk about how they’re using Datameer Big Data analytics to change the way people discover music. Check them out:

When I first spoke to the Datameer team, I was amazed at the value Datameer was bringing to customers. You see and hear a lot of hype about big data from a lot of vendors. But Datameer already has customers in Financial Services, Retail, Internet Security, Transportation, Online Gaming, Manufacturing and other industries. Every day, they’re doing fraud detection and prevention, behavioral analytics, customer segmentation, pricing optimization and credit risk analysis. And rather than experiments, they’ve already seen amazing, hard benefits such as preventing $2B in potential fraud, increasing customer conversion by 60% and decreasing customer acquisition costs by 30%. One of the cool aspects of my job is working with our customers to quantify the value Big Data Discovery has brought to their organizations.

Another aspect of Datameer that attracted me was the team that has decades of experience in Hadoop, data management and business intelligence. Datameer was architected by those who have been working with Hadoop since Hadoop’s very beginning. Each day I work more with the team, I continue to be impressed by the people. They are driven, focused on customer success and really smart.

The other aspect for me was timing. I was at Informatica before joining Datameer. Don’t get me wrong, large companies are good, but when a market revolution like big data is happening, you want to be part of that revolution rather than simply reacting to it. Datameer is clearly a driver in this market, pioneering native-Hadoop analytics for the business users.

Cool market, cool company, cool opportunity.

Google Fusion Tables is a really cool repository of thousands of publicly available datasets. You can use the fulltext search to hunt for all kinds of data sets to use for your analytics and visualization projects.

Let me show how easy it is to work with Google Fusion data with a quick example. I’ll make a simple data comparison visualization of GDP spending in countries around the world.

First, I type in the following keywords into the Google Fusion fulltext search field.

I’m able to access and import three data sources to be used for my simple visualization.

In Datameer, I create a workbook with the three imported data sources, and do some data cleansing afterwards
(trimming country names to be joined, converting values to numbers, stripping currency signs. etc.)
and then join the the data sources through a join sheet.

Finally, using a Multi- Bar Chart, I visualize the GDP spending percentage per sector.

And there you have it! Google Fusion data, imported, cleansed and visualized in Datameer.

Second, the only constant in analytics is that data continues to increase in volume and complexity.  Analytics are changing, in fact Big Data is changing everything.  And while data volumes are certainly a factor, new data types and the velocity of new data sources are revealing new customer behaviors and preferences, operational intelligence, security threats, etc.

Typically, technology doesn’t drive change.  Rather, it’s an enabler to address change.  In the case of big data, technology is responding to the user’s need to integrate and analyze all of the new data that is being generated.  Data integration has to evolve as well, both in capabilities and approaches, if it is to keep up with the need for new analytics.

Data integration tools developed historically to meet the need to integrate multiple, structured data sources so users could analyze data around transactions and products.  Developed during the time of limited hardware resources, these tools forced software engineers to require pre-built models of the data so user queries would not overwhelm the available hardware.

Of course, the requirements of big data changes all of that.  New types of data, rapidly evolving data sources and high volumes of data make traditional data integration obsolete.  Big data discovery differs significantly from traditional BI in that it looks to iteratively reveal unknown patterns, relationships and insights across all available data rather than focus on a simple question and answer paradigm.

Big data integration must access and integrate all types of data, otherwise only partial views are possible. Today’s fast pace of business means that new data sources must be be added instantly, without having to model or remodel the data, so that analytics can be delivered quickly and iteratively.  And often,  the volume of today’s data quickly overwhelms traditional hardware architectures so big data must leverage inexpensive hardware resources that can be quickly and cost effectively scaled to meet data demands.

So, a new data integration ecosystem and approach is required to address big data.  Leveraging the memory (and analytics) of an elephant, Datameer is leading that ecosystem with comprehensive wizard-led big data integration built on top of Hadoop.  This includes dozens of connectors to enterprise data that covers all relevant structured and unstructured data sources as well as the ability to read and understand those sources so business folks don’t have to wait for scarce IT resources so sort it all out for them.  Second, Datameer leverages the linear scalability of Hadoop to avoid the need to pre-model the data for analytics.  Users simply ingest their relevant data into Datameer then perform unlimited analytics in a “model upon read” approach that enables users to iteratively discover insights, patterns and relationships without constraints.  And if they find they need additional data sources or more data, they simply add those sources and continue on.

Datameer thrives in Fortune 100 companies and data-driven startups, often with Hadoop acting as the central data hub for all data sources.  Users simply point Datameer at the data to quickly and easily perform analytics, discovery and visualizations that reveal insights across all of the data.  The results can also feed additional analytics solutions or recommendation engines. Datameer’s big data integration ecosystem approach closely parallels how business is evolving and is critical to maintaining insights in today’s competitive markets and industries.

To find out more about Datameer’s Data Integration capabilities, check out this short video.

—————

I gave a talk about getting involved with the D3 community at Trulia for the Bay Area D3 User Group, with Scott Murray and short talks by Jérôme Cukier, Santiago Ortiz, Jyri Tuulos, Alex Johnson, Amelia Greenhall, Mike Travers and Péter Boda. My slides are here and it was recorded on video. I launched my alternative D3 gallery made with D3 and based on a Google Spreadsheet.

Here is the talk in a blog post format. It is mostly a collection of links for getting started with D3 and contributing to this awesome community.

D3.js stands for “Data-Driven Documents”, as you can easily see on the official website and the academic paper.

D3.js Website
D3.js paper

It is a library to bind data to graphics. What does it means? In short: give shape to your data by using some data elements to set some attributes of your graphical elements. The long answer: read “The Grammar of Graphics” by Leland Wilkinson.

The Grammar of Graphics

D3.js is known to have a pretty steep learning curve. In fact, the syntax is very clean, expressive, and fairly easy to learn. But you need to already know Javascript, SVG, the DOM, HTML5 and other web technologies.

DOM
Javascript
CSS
SVG
HTML5
Other web technologies

But it will get you nowhere if you are not interested first in visualization.

Visualization books

D3.js is not a shim to help with cross-browser issues (like in JQuery). It is not a graphical abstraction like Raphael or Dojox.GFX. It is not a chart library like Highcharts, even if nice chart packages has been built on top of it.

NVD3
xChart
Dc.js/
Dance.js
Rickshaw

It is a set of helpers (reusable components, helper functions) to help you build visualizations using web technologies.

Where to start learning it? It depends on what’s your favorite way to learn.

Books
Murray
Dewar

Tutorials
Video
Slides
Others

Blogs
Mike Bostock
Jim Vallandingham
DashingD3Js

Personally, I prefer to look at the source code or read examples.

API on the Wiki
Example Gallery

That’s probably why I launched an alternative gallery, giving you more freedom to filter and explore more than a thousand examples.

Alternative Gallery

You can contribute to this gallery by asking for administrative rights on the spreadsheet used as a database (or by sending me a tweet at @d3visualization).

Alternative gallery spreadsheet

You can contribute to the D3 community in many ways.

Show your examples
Official Gallery

Share Code
Gist
bl.ocks (RSS)
Tributary
JsFiddle
Plunker

Submit
Bug report
Pull Request
Plugins

If you need help, we are never far away.

Google Groups
StackOverflow
IRC: #d3js

BTW, If you want to stay up to date with new D3 examples, events and releases, follow my twitter feed exclusively about D3.

@d3visualization

Thanks for being part of this awesome community!

This talk and blog post were made possible by Bay Area D3 User Group and Datameer.

Late last year at Strata New York, we announced our Datameer Analytics App Market – the first true marketplace for buying and selling pre-built big data analytic applications. We launched the marketplace with several dozen free apps, and we’ve been adding to it ever since. So its only appropriate that this week at Strata Santa Clara, we turn on that ‘paid’ feature of the app market so you can make money building apps, along with several other in-product feature enhancements in our 2.1.3 update.

Here’s the deal. We know you guys have domain expertise meaning you know your data and what kind of insights you’d want out of it. So you should get paid to create and distribute that expertise. We’re giving you the DIY tools you need to build —and now sell—your own big data analytics apps. Datameer lets you:

• Get any data in yourself (really, we have 50+ built-in connections to Twitter, Salesforce, Oracle, DB2, Google… etc.)
• Build your analysis with simple point-and-clicks (we’ve included more than 240 pre-built analytic functions, and you see the results instantly in our spreadsheet UI before running it against your full dataset)
• Create custom data visualizations (no more box-after-box dashboards, our Business Infographic designer lets you drag-and-drop different charts and graphs onto a blank canvas, so you can annotate and design as you see fit)
• Package your final product into an analytic app with a single click. It’s just a right-click ‘save as’…

From there you just fill out information about you and your new app, set your price, and you’re done. From there we’ll make sure it’s fit for the app market, and will place it in the market for you. That’s all there is to it!

There’s too much talk about the lack of Data Scientists out there. Let’s attack this problem from the other end and give business users the pre-built apps they need to simply plug in their own data and get their insights without the work. Download the free trial of Datameer to build your first app today.

First, we are extremely honored to be included in Fast Company’s Top 100 most innovative companies and Top 10 most innovative Big Data companies list. When you look at who’s included, you’ll notice the list is very use-case oriented, it’s about what you can do with big data, not the deep tech behind it. There’s no technical talk – Hadoop isn’t even mentioned once.

Second, we’re extremely proud that the editors and contributors at CIO UK have included Datameer in their list of 20 vendors to watch in 2013. What’s exciting is that it’s not just 20 Big Data vendors, it’s 20 vendors for CIOs to watch. Big Data is making its way into the general business fold. We’re included in a list with Apple and Google. Amazing.

We’re on our way to crossing the chasm. And to think… it’s only February.

“…outside of companies like Google that have long made use of rich rosters of PhDs, there are nowhere near enough “data scientists” — graduate-level candidates with backgrounds in machine learning or statistics — to analyze the massive streams of information that are being produced, and that gap is growing by the day.

Cofounder Ali Behnam of Riviera Partners, a Palo Alto-based technology recruiting firm, says he’s aware of “thousands” of data science jobs that are awaiting candidates at nearly every type of Bay Area-based tech company.”

While demand certainly outweighs supply, and while there’s certainly more work to do building curriculum for future data scientists, we also need to be focusing on empowering line of business users to address the larger issue from the other end. They know their own data better than anyone else, and with the right tools that makes getting data in, analysis, and visualizing results fast and easy, businesses can become more data-driven before they land their elusive Data Scientist.

In October, we announced the Datameer Analytic App Market, where you can download packaged analytical apps, plug in your own data and it automatically analyzes the data and populates the pre-built visualization or infographics.

Bring the two together and you get full editorial control over your final data visualizations. Here’s how to customize your visualizations in Datameer: