Data Preparation

Data preparation is the process of cleaning, structuring, and enriching raw data, including unstructured or big data. The results are consumable data assets used for business analysis projects.

In the data science community, data preparation is often called feature engineering. Although data prep and feature engineering are used interchangeably, feature engineering relies on domain-specific knowledge compared to the standard data prep process. Feature engineering creates “features” for specific machine learning algorithms, while data prep is used to disseminate data for mass consumption.

Both data preparation and feature engineering are the most time-consuming and vital processes in data mining. Having data prepared correctly improves the accuracy of the outcomes. However, data preparation activities tend to be routine, tedious, and time-consuming.

Data transformation has historically been the “T” of the ETL process – extract, transform, and load.  ETL developers, and eventually data engineers, would transform data as part of a larger, more complex process, in order to mark the data ready for analytics.  One reason why the data transformation was the domain of these highly technical teams was that the target structures in a traditional data warehouse or mart were highly complex – e.g. star- and snowflake-schemas.

In the Hadoop and data lake era both data engineers and analysts were thrust into working with data that was much more complex in terms of diversity and format.  BI tools in this era were not equipped to handle such data.  Early data preparation tools came on to the market to make it much easier to transform complex data into analytics-ready formats consumable by BI tools.  Eventually, the BI tools started to introduce their own data preparation within their suites.

Conceptually, data preparation and data transformation are similar.  The introduction of cloud data warehouses and the new ELT model of processing introduced one major difference – data preparation tools used their own processing engines (Spark, etc.), while data transformation tools relied on the scalable modern cloud data warehouses such as Snowflake for their processing power.

Many data preparation tools were designed to be self-service for analysts and data scientists, with methods of transforming data without writing code.  Initial data transformation tools in the ELT stack such as dbt focused on using SQL coding as the primary means to transform data, pushing the domain back to more technical, programming-savvy staff.

Next-generation data transformation tools such as Datameer also facilitate data preparation by embracing:

• The self-service principles of self-service for non-technical team members, first introduced in data preparation tools,
• The need to support multiple personas (technical and non-technical) and collaboration among these personas, and
• Integration with cloud data warehouses such as Snowflake for their processing power.

Datameer is a powerful SaaS data transformation platform that runs in Snowflake – your modern, scalable cloud data warehouse – that combines to provide a highly scalable and flexible environment to transform your data into meaningful analytics.  With Datameer, you can:

• Allow your non-technical analytics team members to work with your complex data without the need to write code using Datameer’s no-code and low-code data transformation interfaces,
• Collaborate amongst technical and non-technical team members to build data models and the data transformation flows to fulfill these models, each using their skills and knowledge
• Fully enrich analytics datasets to add even more flavor to your analysis using the diverse array of graphical formulas and functions,
• Generate rich documentation and add user-supplied attributes, comments, tags, and more to share searchable knowledge about your data across the entire analytics community,
• Use the catalog-like documentation features to crowd-source your data governance processes for greater data democratization and data literacy,
• Maintain full audit trails of how data is transformed and used by the community to further enable your governance and compliance processes,
• Deploy and execute data transformation models directly in Snowflake to gain the scalability your need over your large volumes of data while keeping compute and storage costs low.

Datameer’s self-service Excel-like interface, rich catalog-like data documentation, data profiling, and a rich array of functions available through a graphical formula builder allow your analytics teams to quickly perform data preparation.  They can also do so in collaboration with more technical data engineers in a process where data engineers build base models from raw data, then analysts shape and organize the data to their specific needs.

Datameer supports all the critical aspects of data preparation, including:

• Data cleansing – functions for the removal of bad records, replacing invalid or blank values, and de-duplicating data,
• Data blending – join and union functions to blend disparate datasets into a common, normalized view,
• Advanced transformations – pivoting, encoding, date and time, conversion, working with lists, parsing functions,
• Data enrichment – functions to create value-added columns including math, statistical, trigonometric, mining, and path construction,
• Data grouping and organization – more sophisticated ways to group, aggregate, and slide-and-dice data, including pivot tables, sessionization, custom binning, time windows, statistical grouping, and algorithmic grouping,
• Data science-specific – one-hot, date/time, and binned encoding functions for data science models.

Datameer can provide the universal tool for all your data transformation needs, whether data engineering, analytics engineering, and analyst or data scientist data preparation, and facilitate cataloging and collaboration across all these functions.