Matillion versus Datameer Spectrum

Matillion & Datameer

Datameer is an extremely easy-to-use tool that focuses on the T in your ELT processes for faster, more reliable data transformation.  Datameer offers a much richer transformation library and feature set to support many more use cases than Matillion and deep integration into your Snowflake cloud data warehouse for optimal performance and data security.

about informatica

What is Matillion?

about matillion

Matillion is one of the younger, cloud-based ETL solutions on the market.  The platform covers all three aspects of data integration – extract, transformation, and loading.  The graphical user interface allows users to orchestrate and run data pipelines without coding.  The platform is flexible, offering an extensive array of cloud connectors, as is the cloud-based pricing model.

How Does Matillion Work?

How does Matillion Work

Matillion consists of three components: the underlying platform, a graphical data orchestration tool, and a management tool.  The three combine together to enable the definition and operation of ETL data pipelines.

The graphical orchestration tool allows users to string together “components” into a data flow.  Components can be data source connections and extractions, data staging and definition, flow control, transformations, messaging, and loading into destinations.  Matillion offers 105 connectors and 75 components, of which 25 are for transformations.

The management tool allows admins to set up users and security, run jobs, configure the system, and perform other administrative tasks.  For security, Matillion offers user- and role-based security and access controls, LDAP integration, and Single Sign-On (SSO) integration.

The platform is the execution part of the system.  It is essential to understand that Matillion does NOT have a storage and execution engine.  All data processed in a data flow is stored in its intermediate form in your cloud data warehouse tables.  All data management and transformation operations are pushed down into the cloud data warehouse.  The scalability and performance of Matillion are dependent upon that of your cloud data warehouse.

Spotlight icon

What is Datameer?

Datameer is a powerful SaaS data transformation platform that runs in Snowflake – your modern, scalable cloud data warehouse – that combines to provide a highly scalable and flexible environment to transform your data into meaningful analytics.  With Datameer, you can:

  • Allow your non-technical analytics team members to work with your complex data without the need to write code using Datameer’s no-code and low-code data transformation interfaces,
  • Collaborate amongst technical and non-technical team members to build data models and the data transformation flows to fulfill these models, each using their skills and knowledge
  • Fully enrich analytics datasets to add even more flavor to your analysis using the diverse array of graphical formulas and functions,
  • Generate rich documentation and add user-supplied attributes, comments, tags, and more to share searchable knowledge about your data across the entire analytics community,
  • Use the catalog-like documentation features to crowd-source your data governance processes for greater data democratization and data literacy,
  • Maintain full audit trails of how data is transformed and used by the community to further enable your governance and compliance processes,
  • Deploy and execute data transformation models directly in Snowflake to gain the scalability your need over your large volumes of data while keeping compute and storage costs low.

Quick Comparison

At a high level, Datameer focuses on data transformation and is the T in your ELT, while Matillion is an end-to-end ETL data integration tool.  You can do data transformation in Matillion and that is what we will focus on here.  From a data transformation standpoint, several areas differentiate Datameer from Matillion, including:

  • Tools that work for all your personas: no-code, low-code, and code.
  • Discover, share, and reuse data transformation components.
  • Fully enrich your data with an Excel-like formula builder and a large library of functions,
  • Let non-technical analysts model and transform data without any coding or schema knowledge.
  • Rich, searchable, catalog-like data documentation, attributes, comments, tags, and more.
  • Deep integration with Snowflake for cost-effective scalability.

Let’s explore each of these in more detail.

Multi-Persona Tools

Matillion’s data orchestration tool makes it harder to put together data flows than the company lets on.  This is because:

  • Components are fine-grained, performing highly specific tasks, forcing users to string together many components in even a simple data flow
  • Each component requires extensive configuration, requiring a good amount of time by the user
  • Data flows require you to define and manage intermediate forms of data adding to the complexity of the flow
  • Each transformation is applied as its own component, forcing users to put together a large number

This combination of factors can lead to very complex data flows with many components.

Datameer wants to be attractive to a wide audience with many different skills across the entire data and analytics teams – data engineers, analytics engineers, data analysts, and more.  Some personas may be more technical and coding-centric while others may be less technical but more data-savvy.

To meet this goal, Datameer offers three different user experiences:

  • No-code – Easy drag-and-drop modeling that lets non-technical analysts create base models specific to their needs.
  • Low-code – A wizard-driven Excel-like interface that allows non-technical analysts to enrich data and dramatically improve data engineer productivity.
  • Code – Full SQL coding and scripting that gives your data engineering teams the control to create and deploy highly optimized data models.

Each allows team members to participate in the data transformation process while using their various skills.  Models created in each of the three UXs can be mixed and matched together.

Discoverability, Sharing, and Reuse

faster pipeline definition

In Matillion you can create reusable components of functions and apply them to different data pipelines.  But as you you reuse these, you are making copies.  In addition, you need to recreate schemas within your data pipelines – you are not reusing existing schemas in your cloud data warehouse.

In Datameer, shared workspaces are a foundational component.  Data engineers, data analysts, business analysts, and data scientists can work together in shared workspaces creating and reusing models using the different tools – code, low-code, no-code.  Beyond working together on modeling, the broader team can share knowledge about the workspace and models, including:

  • Adding descriptions, which can both explain data and how best to use it,
  • Applying tags, which can help organize and identify data,
  • Supplying comments, which can add simple ideas around data or enable collaboration,
  • Adding business metadata, which translates technical metadata into business terms
  • Setting properties or status and certification fields, which describe the state of a data object

In addition, all information documented about models is fully discoverable via Datameer’s Google-like faceted search.

 

Full Data Enrichment

data preparation

The main design point for Matillion is to allow users to extract, load, and do simple data transformations in the form of data mapping.  It contains a limited set of functions to create new fields within the data.

Datameer offers a graphical formula builder and a deep array of functions to enrich your datasets with calculated columns.  Any function that is available in SnowSQL (Snowflake’s SQL version) is available to users in Datameer, and Datameer maps these functions one-to-one to SnowSQL for optimal performance.

people icon

Get the Non-Technical Analysts Involved

Hybrid Cloud

Matillion requires ETL developers with strong Python and SQL coding skills. It is difficult for non-technical, non-coding team members to use.

Datameer has different interfaces for different personas – no-code, low-code, and code.  Technical coders will like to use the SQL-style code interface.  Less technical staff members can use the no-code completely graphical interface or low-code Excel-style formula interface to define data models and transformations.  Regardless of how they are built, data models are translated into SQL and put inside your Snowflake instance. Models build in any of the interfaces can be mix-and-matched inside of larger dataflows.

Rich, Catalog-like Data Documentation

security and governance

Matillion offers very little means to document your data models and transformations.  In addition, it offers no search to discover existing data models and flows.

Datameer facilitates capturing as much information as possible about the data it is working with, the transformations performed, and the resulting data models.  This includes:

  • Auto-generated documentation and information such as schema information, transformations performed, data lineage, audits, and certain system-generated properties
  • User-supplied documentation such as descriptions, tags, comments, and business metadata.

In addition, all information documented about models is fully discoverable via Datameer’s Google-like faceted search.

snowflake-logo

Deep integration with Snowflake

snowflake-logo

Matillion does integrate with Snowflake to perform data transformations inside of Snowflake.  But the integration is high-level and requires users to perform the complex process of defining and mapping schemas.

Datameer builds models directly into Snowflake using data already extracted and loaded to maintain extremely high data security and easy governance.  Transformations are executed using the Snowflake engine to take full advantage of its power and scalability.  Datameer also inventories and documents the loaded raw data to add further knowledge and discoverability.

Conclusion

Datameer Spectrum Versus Matillion

Matillion is very much a one-trick pony for performing integration of cloud and SaaS data sources into a cloud data warehouse – more for the EL rather than the T.  The user experience makes it unsuitable for self-service data pipelines and still makes it difficult and complicated for data-savvy users to create data pipelines successfully.  Its restricted set of transformation functions and capabilities severely limits where and how you can use Matillion.

Datameer focuses on the T in your ELT.  It is a robust data transformation tool and platform that gets all your personas involved with three different style interfaces, lets them share and collaborate around data modeling, and has deep integration with Snowflake.

See How Quickly Datameer Can Transform Your Data in Snowflake.

Learn More