Why Every Organization Needs a Data Transformation SaaS Platform

Data transformation has been a significant component in how organizations deliver analytics-ready data for nearly 30 years.  The data transformation process for many years focused on the “T” in ETL ( extract, transform, and load ).  Inside an ETL pipeline, data transformation primarily focused on cleansing data and mapping it from a source schema to a destination one.  As organizations became more sophisticated in their destination schemas (star- and snowflake-schemas), these mapping processes became more complex.

In the 2010s through today, the role of data transformation has expanded.  New data sources and formats emerged, and new platforms such as data lakes were deployed to support this.  The role of data transformation became increasingly important to help deliver value.  New, complex data formats needed to be normalized, cleansed, integrated with traditional sources (often existing data warehouses), and then enriched.

And finally, data preparation became an integral component for self-service data transformation by analysts in the second half of the 2010s.  This allowed individual, less technical analysts to perform a wide variety of transformations on the data without relying on and waiting for IT and data teams to create data transformation pipelines for them.

In recent years, new trends have emerged that have changed the way people think about and perform data transformation, including:

• Cloud data warehouses – Many organizations deploying new or migrating existing data warehouses in the cloud to take advantage of the economics, flexibility, and modern architecture.  Cloud data warehouses also provide a unique platform to perform SQL-based data transformation.
• ELT (Extract, Load, and Transform) – The processing model for data pipelines has changed from the previous ETL one to ELT.  In the ELT model, the data transformation step is last (after loading source data) and is performed inside the cloud data warehouse either directly via SQL scripting or higher-level tools.
• Multi-persona analytics engineering – While data engineering was an inside-out task for data teams to deliver data to the business, a new analytics engineering process has emerged.  Analytics engineering involves collaboration between the data and analytics teams, uses the ELT approach, and shares the workload between these different personas to make the process more efficient.
• Cloud/SaaS – With much of your data residing in a cloud data warehouse, organizations want similar SaaS-based services for data transformation that give them the same economics, flexibility, and fit in their modern architecture.

With a Data Transformation SaaS solution, a customer can subscribe to and gain nearly immediate access to the managed service, use the data transformation tools to create models, manage and deploy data transformation models, and operate these pipelines to ensure an effective data transformation flow.

As a managed service, the customer does not need to bring or operate any cloud compute or storage services of their own.  When the SaaS platform focuses on facilitating the transformation in an ELT process, the customer simply points the system to their existing cloud data warehouse, which provides the data storage and compute horsepower.

One obvious and core benefit a Data Transformation SaaS solution offers is similar to that of cloud and SaaS services: scalability, flexibility, and modernization.  Beyond this, it brings together additional benefits that include:

• A neat fit into a modern data stack
• Direct integration with your cloud data warehouse
• More modern tools with interesting new capabilities
• Data documentation and discovery services
• Collaboration across the entire team

Let’s explore each of these in more detail.

As organizations move their analytics into the cloud, they often create a new, modern data stack around tools and processes that embrace cloud technologies and embody scalability and flexibility.  The new modern data stack often includes:

1. Data loaders that facilitate the EL of the ELT process,
2. A cloud data warehouse,
3. A data transformation tool,
4. Analytics and/or data science tools
5. DataOps/data observability tools

Of the modern data stack components, items 1, 2, 4, and 5 are all SaaS/managed services that are easy to subscribe to, use, and operate.  Hence, Data Transformation as a Service is a much more natural fit within this stack carries with it similar compute and operational benefits as the other cloud-based SaaS services, and integrates more effectively with core components such as your cloud data warehouse (see below)

As organizations create their modern data stack, the cloud data warehouse becomes the centerpiece and main workhorse.  It provides the core storage and compute (query) facilities for the architecture.  It also provides a standard interface and component to which the remaining tools in the stack can integrate.

A data transformation as a service platform embraces the cloud data warehouse as its’ lynchpin, integrating with the core services the CDW provides.  Through its integration with the CDW, the Data Transformation SaaS platform will:

• Create and maintain its data models within the CDW,
• Embrace the CDW’s SQL language for transformation execution (either thru direct SQL or SQL generated via higher-level modeling – see below), and
• Use the CDW query engine for processing and storage engine for materialized data models.

This creates a highly efficient execution model for data transformations and uses highly cost-effective CDW query/compute and storage services.

There are a large number of tools that perform data transformation on the market.  Some are legacy tools, and some are more modern.  Even though legacy tools have had multiple facelifts over the years, these products still embrace old-style data transformation, as seen here with Informatica and Talend .  Even some cloud/SaaS-based tools that support the ELT process still have old-style data transformation interfaces, as seen here with Matillion .

A good data transformation as a service tool embraces the modern fact that there are multiple personas involves in the analytics engineering process, including data engineers, analytics engineers (more technical analysts), data analysts, and data scientists.  Each of these personas often has:

• Different skill sets ranging from being highly technical to less technical,
• Different knowledge and understanding of how data is used,
• Different focal points on their role and where they can best use their time.

To this end, a SaaS Data Transformation tool will provide a much more modern tool that allows each persona to get involves in the data transformation process, best use their skills and ensure they spend the most time doing what they do best.  For example, data analysts can spend more time on analysis and less on data transformation through an easy-to-use no-code or low-code data transformation interface.

A highly modern SaaS Data Transformation platform should support three different interfaces to support multiple personas:

• No-code – for non-technical data and business analysts
• Low-code – for slightly more data-savvy data analysts or for analytics engineers that want greater productivity than coding
• Code – for data and analytics engineers that want to use SQL for the control and optimization that comes with it.

Data documentation is often spread among wiki pages, metadata management systems, or early versions of data catalogs.  Most of these sources still do not capture much of the knowledge there is about the data.  Some data transformation tools attempt to generate documentation about data, but often this is just taking comments from SQL code and generating a wiki page or adding a limited description.

A Data Transformation SaaS tool will embrace the ability to capture as much information as possible about the data it is working with, the transformations performed, and the resulting data models.  This would include auto-generated documentation and information such as schema information, transformations performed, data lineage, and audits.

A Data Transformation SaaS tool should also facilitate user-generated information about data, capturing additional information from across the data and analytics community, including:

• Descriptions, which can both explain data and how best to use it,
• Tags, which can help organize and identify data,
• Comments, which can add simple ideas around data or enable collaboration,
• Business metadata, which translates technical metadata into business terms
• Status and certification fields, which describe the state of a data object

In traditional data engineering processes, often where things break down is when the data team needs to interpret the needs of the analytics teams and mismatches arrive in the produced datasets, wasting valuable time and effort.  This happens because the data teams know a lot about the data but little about how it is used by the business, and visa-versa for the analytics teams, they know little about the data but know how the business will use it.

Besides supporting the multiple personas that are involved with a modern data stack, a data transformation platform facilitate a collaborative data lifecycle process that eliminates errors and mismatches in the analytics engineering process:

• Analytics teams can use the no-code or low-code tools to interactively create and explore base models they are looking for from the data, and also add descriptions of the resulting analytics to show how the business will use it,
• Data teams can then use the SQL or low-code tools to fully flesh out the final data pipeline and models to the “specs” provided by the analysts, then operationalize these

This collaborative process eliminates errors in data transformation and modeling, speeding the overall process by 20-30x.

While a data transformation SaaS platform can provide all the aforementioned everyday benefits to the data and analytics teams, the most important benefits it brings is to the bottom line:

• Up to a 40% reduction in data and analytics engineering costs due to greater productivity,
• Up to 300% greater ROI from data and analytics engineering due to greater productivity,
• Up to 500% greater ROI from your cloud data warehouse from greater productivity and faster availability of data to the business

Datameer is a powerful SaaS data transformation platform that runs in Snowflake – your modern, scalable cloud data warehouse – that combines to provide a highly scalable and flexible environment to transform your data into meaningful analytics.  With Datameer, you can:

• Allow your non-technical analytics team members to work with your complex data without the need to write code using Datameer’s no-code and low-code data transformation interfaces,
• Easily combine large volumes of captured data with master and other data to create context-rich, meaningful datasets for analysis,
• Fully enrich analytics datasets to add even more flavor to your analysis using the diverse array of graphical formulas and functions,
• Generate rich documentation and add user-supplied attributes, comments, tags, and more to share searchable knowledge about your data across the entire analytics community,
• Use the catalog-like documentation features to crowd-source your data governance processes for greater data democratization and data literacy,
• Maintain full audit trails of how data is transformed and used by the community to further enable your governance and compliance processes,
• Deploy and execute data transformation models directly in Snowflake to gain the scalability your need over your large volumes of data while keeping compute and storage costs low.