Design Patterns for Data Preparation using Datameer
While each use case may be unique, several common design patterns can be used for data preparation in data pipelines that curate data for analysis. Let’s explore these and see how you can apply them.
About The Design Patterns for Data Preparation Book
Each business case is a unique opportunity to harness data for analysis. These analyses take many different forms depending on the matter at hand. For example, if your function is marketing, you might be interested in analytics to understand customer sentiment, or a retail company might be interested in customer churn. Although these analytics vary, there are some common patterns and best practices that we can use across any analytics project and industry.
Data preparation is often used to merge different data sources with different structures and different levels of data quality into a consistent, reusable format. This canonical format is what makes all that data preparation, that is, the cleansing, the transformation, the blending, the enrichment, and the exploration, so powerful. This is how to find meaningful business insights in data.
What Are the Four Stages for Data Preparation?
See the four stages of data preparation, including identifying the right data, cleansing it, enriching it, and finally delivering a curated dataset.
Design Patterns for Customer Churn
Explore some common patterns to prepare data to analyze the various aspects of customer churn.
Design Patterns for Marketing
Explore some common design patterns for marketing analytics, including social media and sentiment analysis.
Using Algorithms to Find Patterns
See ways to use built-in advanced algorithms in Datameer (Smart Analytics) to find consumer and transactional data patterns.
ANALYZING THE DATA FOR CHURN
A common analysis in a retail environment is determining customer churn. We can add new features to the purchasing data we have just created to determine the customers at risk of leaving our business – all with existing functions in Datameer. In addition, once we have finalized our dataset, it can be scheduled and operationalized.
Using a simple model, we will look for patterns that are outside a customer’s purchase behavior. Again, this example uses retail data, but the pattern can be applied to similar projects; essentially, wherever behavior is analyzed using historical transactions.
We accomplish all this in two Datameer workbooks. The first workbook contains the two imported datasets, and then the joined set we will use for modeling. The second workbook is the workbook used for grouping, filtering, adding features, and holding the final data that we can operationalize for future analytics.
Grouping, Sorting, and Modeling
This is a ubiquitous and iterative process in an analysis project to find and add value to the data. In Datameer, we will add three worksheets to a new workbook. Each worksheet represents a primary stage in the preparation process – this is simple to do and provides clear steps as we create the features for our models and analysis.
We begin by adding the previously prepared dataset, in this case, customer purchases, to the first worksheet in our new workbook. This is the source data for grouping, filtering, and enriching to understand customer churn. For churn, we really want to know the period between customer activity and purchase, which will take some preparation to parse out of the data.
Grouping and Sorting – We want to structure the data for analysis. The goal at this stage in the churn analysis is to determine the number of days between purchases for each customer.
Modeling – We have the data needed to create our analytical model to determine customer shopping behavior. We begin by adding a final worksheet to our existing workbook.