Talend versus Datameer Spectrum

Talend offers a comprehensive integration platform covering a full range of integration scenarios, including data integration, real-time data pipelines, application and API integration, master data management, and data quality and governance.  Talend wants to be your soup-to-nuts provider of an enterprise data fabric.

In our examination of Talend, we will focus on the data integration solution.  This includes a core data integration platform (on-premises or in the cloud), a specialized data replication product for the cloud – Stitch – and related products for data cataloging, data preparation, and data stewardship.

Although the complete platform has multiple offerings, customers mainly use Talend for data integration.  In the early days of the product, it was used mostly to take data from on-premises operational systems, integrate it, and load it into on-premises data warehouses or data marts – a very typical ETL use case.  The product suite has expanded in recent years to cover cloud data integration via expansion of the main product and through the acquisition of Stitch.

The platform consists of five key components:

• The Data Integration platform provides the core services, including connectors, transformation components, execution engines, and security.
• The Studio, Pipeline Designer, and Management Console are the tools to define, manage, and run data integration pipelines.
• The Data Preparation tool is used to define data transformation components to be inserted into the data pipelines.
• The Data Catalog provides an inventory of all datasets and pipelines managed within Talend and information about these.
• The Data Stewardship module is a tool to run processes for data stewardship and governance.

Also, there is Stitch, a platform for cloud “data loading.” Stitch is a SaaS platform that allows customers to easily “replicate” their data from many cloud data sources, applications, and services into a cloud data warehouse.

As you can probably tell by the above description, the Talend platform is extremely complex, with a variety of different platform services and user interfaces/tools.  Also, the services you would use are different based on what you are trying to achieve.  For example:

• On-premises data integration – you would use the on-premises data integration platform, which has 75+ data source connectors and additional data cataloging, data preparation, and data stewardship tools.
• Extending data integration into the cloud – if you are trying to integrate on-premises data into a cloud destination, such as a cloud data warehouse, you need to use the on-premises data integration platform as the cloud platform only has connectors for cloud data sources (see below).
• Pure cloud data integration – if you are trying to integrate data from the cloud and send it to a cloud destination (data warehouse or data lake), then you would use the cloud data integration platform, which has a limited set of connectors reaching only cloud data sources, and cloud-based versions of the studio, pipeline designer, data catalog, data preparation, and data stewardship tools.
• SaaS and Cloud services data integration – unfortunately, the cloud data integration platform has only a small number of connectors for SaaS application sources (5) and none for cloud services such as Google Analytics or Adobe Analytics.  In this case, one would need a disjointed ELT approach using Stitch to replicate the data into a cloud data warehouse and the cloud data integration platform and tools for transforming the data.

Datameer Spectrum is a fully-featured ETL++ data integration platform with a broad range of capabilities for extracting, exploring, integrating, preparing, delivering, and governing data for scalable, secure data pipelines.    Spectrum supports analyst self-service data preparation and data engineering use cases, enabling a single hub for all data preparation across an enterprise.  Pipelines can span across various approaches and needs, including ETL, ELT, data preparation, and data science.

The Spectrum user experience offers wizard-driven simplicity and visual transformations for fast, easy data preparation by your analyst community without writing one line of code.  Simultaneously, the deep suite of functions and governance features enables data engineering pipelines of any level of sophistication.  The result is a cooperative environment where analysts and data engineers can get your data analytics-ready 10 to 20 times faster at a fraction of the cost.

Once integration dataflows are ready, Spectrum’s enterprise-grade operationalization, security, and governance features enable reliable, automated, and secure data pipelines to ensure a consistent data flow.  Spectrum has extensive features to support your hybrid-cloud data landscape. It is cloud-native on all three major cloud platforms (AWS, Azure, GCP) and carries the elasticity and cost economics you would expect in the cloud.  Spectrum can bring together any data sources you have regardless of type, format, and location (cloud or on-premises).

At a high level, both Talend and Datameer Spectrum are enterprise-grade, scalable data integration platforms with deep capabilities and rich tools to facilitate the creation and deployment of data pipelines.  But several areas differentiate Spectrum from Talend, including:

• Providing a single integrated platform and toolset
• Offering a smooth, easy user experience from one tool
• Supporting a large, consistent set of data sources
• Enabling data integration in the cloud with on-premises sources
• Offering data preparation as a first-class tool
• Delivering direct integration with BI tool destinations
• Having a simple, lower-cost pricing model

Let’s explore each of these in more detail.

As we saw in the “How do Organizations Use Talend” section above, the Talend platform consists of a complex set of tools and services designed for specific purposes but is not well integrated.  We also saw the Talend platform services available are different based on the data integration scenario.  A quote from the 2020 Gartner Magic Quadrant for Data Integration Tools confirms this:

“Reference customers and Gartner clients currently exhibit a limited understanding of which tool to use and how to deliver integrated views of data using Talend’s data integration tools portfolio.”

Spectrum offers a single integrated platform and toolset that provides a unified data integration hub for any of your use cases.  All the Datameer services, including tools, connectors, execution engines, security, and more, are consistent regardless of how you need to use the platform.  This spans on-premises and the cloud, and self-service analyst pipelines, and transformation-rich data engineering ones.

Working with Talend requires using multiple tools, each with specific functionality to design a piece of your data pipeline or manage the pipeline. Talend’s original design point was for typical data integration from operational sources to a data warehouse, and the product makes those straight-forward for data-savvy people.

Creating and executing transformation-rich, more sophisticated data pipelines is much more difficult and complicated.  It requires moving around between multiple tools.  Datasets are discovered via the data catalog.  The pipeline designer is where components are linked together in a pipeline.  Data preparation is performed in a separate tool Analyst self-service is not achievable with Talend.  To quote the Gartner Data Integration MQ once again:

“Some reference customers reported challenges with the overall orchestration and operationalization of complex data pipelines in Talend’s data integration tools.”

This problem further exasperates if you need to work with SaaS and cloud data sources using Stitch.  Stitch can get the raw data into a cloud data warehouse but has no transformation capabilities.  You need to go into the cloud data integration platform and write ELT transformations, creating multiple disjointed pipelines and processes to manage.

Spectrum provides a single, seamless user experience that supports all aspects of data pipeline creation and management, including discovery and exploration, design, testing, and deployment – without any coding.  Spectrum supports analyst self-service data preparation and pipelines with an easy-to-use spreadsheet-like UI.  Spectrum supports a deep library of close to 300 functions applicable graphically to tame even the most complex data for more sophisticated data engineering tasks.

Talend claims to have over 1,000 connectors AND components.  This number is misleading as each connector has anywhere from 10 to 15 components for specific tasks.  In reality, the number of real connectors is lower and varies when you move between different tools and platforms:

• The on-premises platform supports 75 or so connectors on-premises databases and data warehouses, on-premises applications, and a limited set of cloud warehouses.
• The cloud platform only supports 27 connectors to cloud data sources only, supports only 5 SaaS applications, and no cloud services.
• Stitch offers many connectors but has no data transformation capabilities, requiring loading into a CDW before transformation in a separate tool.

Spectrum has an extensive array of over 80 connectors designed to work with different sources – databases, data warehouses, files, SaaS applications, and cloud services – in various formats – structured, semi-structured, and unstructured.  Spectrum does not require multiple components in a pipeline to work with a source or destination – it is a single component easily defined via a wizard-driven interface.  Spectrum has automated capabilities to parse and extract data from complex formats such as JSON and contains an extensive suite of functions to mine insights from semi-structured and unstructured data.

As you may have surmised by earlier information, getting Talend to work in a hybrid environment is difficult at best, and in many cases, not practical.  This is due to the mismatches in functionality between Talend’s on-premises and cloud offerings, particularly in the connectors supported.

Trying to integrate on-premises and cloud data in the same pipeline with Talend requires you to use the on-premises product and provision new compute and storage resources to support it.  You can then have a destination that is a cloud data warehouse.  The alternative is to replicate and load data into the cloud and transform it there with either Talend’s cloud platform or manually in the CDW.

Spectrum’s platform has the same functionality regardless of whether running it on-premises or in the cloud.  If the pipelines are in the cloud, secure protocols, enterprise security controls, and encryption can access and bridge on-premises sources to the cloud securely and have robust data retention policies that ensure extra copies of data are not left around.  Spectrum also lets you seamlessly burst or migrate pipeline workloads into the cloud.

With Talend, data preparation is an after-thought.  Talend data preparation is designed to help cleanse and transform data and has a limited set of functions.  Data preparation is a separate tool and process from the pipeline designer, and “preparations,” which Talend calls components that prepare data, must be integrated and run as part of a data pipeline in the pipeline designer.  This limits the ability to have analysts create self-service data pipelines.

With Spectrum, data preparation is a central part of the platform and a critical piece of the user experience.  The easy, spreadsheet-style user interface allows an analyst to create data pipelines rapidly regardless of technical skills.  For more sophisticated data pipelines, Spectrum provides over 300 functions, all of which can be applied graphically.  The user experience is also interactive, allowing users to immediately see the impact as they apply functions or make changes to speed the process.

Spectrum facilitates self-service data preparation for both analytics and data science, and robust data engineered pipelines.  It provides a shared hub where analysts, data scientists, and data engineers can collaborate and a central repository for regulatory compliance.

With Talend, your data destination is always a database or data warehouse.  This creates an extra hop for BI tools to use the data, requires extra compute and storage costs in the destination, making end-to-end operationalization with BI processes difficult.

Spectrum supports the ability to directly send data to BI platform servers in their native format, including Tableau, PowerBI, Qlik, Looker, and ThoughtSpot.  For operationalized BI use cases, this simplifies the end-to-end process, lowers your costs, and speeds up data delivery.

Talend has a per-user pricing model, which in itself is not bad.  However, it is costly per-user and requires you to purchase add-ons for additional items such as working with big data, data governance, and using their APIs.  The high per-user price is also a disincentive to add more users, particularly analysts and data scientists, often keeping the Talend data integration platform in the domain of a small set of data engineers.

Spectrum offers a simple, cost-effective pricing model based on the number of users and compute resources required for data pipelines.  This pricing model keeps costs low if data volumes are low, effectively scales with an organization’s needs without breaking the bank, and incentivizes getting the broader analytics community on-board.

Talend offers a very comprehensive data integration platform that has been on the market for many years.  But the underlying complexity of the platform is its undoing, especially relative to Datameer Spectrum.  This complexity will drive data engineering costs up and make data pipelines challenging to deliver.  And the Talend pricing model will limit the platform to the domain of the data engineering teams.

Datameer Spectrum simplifies how you manage your data for analytics and makes it faster and easier to create data pipelines ranging from simple to complex.  Like Talend, Spectrum offers a comprehensive set of capabilities but provides it in a well-integrated platform with a seamless, easy user experience.  Spectrum offers something for everyone – data engineers, analysts, and data scientists – and provides a shared hub for this entire community to collaborate with a pricing model that fosters more extensive team usage.

Want to learn more?  Please visit our Datameer Spectrum microsite.  Or experience Spectrum first hand by requesting a personalized demo or our free trial.