What is Data Observability?

Data volumes continue to grow exponentially due to digital transformation and our modern digital economy.  According to IDC’s Global Datasphere, more than 59 zettabytes (ZB) of data were created, captured, copied, and consumed in the world in 2020.  Strong data growth is forecast to continue through 2024 with a five-year compound annual growth rate (CAGR) of 26%.

A major contributor to data growth is the replication of data across an organization, oftentimes for analytics.  IDC estimates the ratio of unique data (created and captured) to replicated data (copied and consumed) to be around 1:9.  By 2024, IDC expects this ratio to grow to 1:10, with even more data being replicated.

With all this data flowing in an organization and business recipients becoming increasingly reliant on data delivery, DataOps and data observability are playing a highly critical role in everyday business operations.   Disruptions in the flow of effective data reduce the business team’s ability to make important decisions and take actions in a timely manner.

In an earlier article, we defined and explained DataOps – an operational process and set of practices around automating data pipelines and flows.  Data Observability is a set of tools to track and manage the health of your data to ensure proper flow and usage.  It is highly related to DataOps.

System, network, and application monitoring is not a new subject.  But an important lesson learned from these tools also applies to data observability – the need to get a holistic and impactful view across your entire data stack.

Data observability is managing (not just monitoring) the health of your data.  Just like many companies strive for a Customer 360, organizations require complete visibility into data pipelines to gain full context into their health, the ability to drill into any problems and determine how to alleviate issues.

Data observability helps improve your DataOps processes by:

• Ensuring data is properly delivered in a timely manner for faster decisions
• Increasing the usefulness, completeness, and quality of data for more accurate decisions with full context
• Delivering greater trust in data so the business can make more confident data-driven actions
• Improving the responsiveness of the DataOps team to the business and meeting promised SLAs

When it comes to the health of your data, the problems go beyond questions such as “did a data pipeline run and deliver its payload.”  Data observability incorporates additional questions such as:

• Did the data arrive on time?
• Did all the data arrive?
• Where was the data delivered to?
• Was the data in the right format?
• How did the data come into the final format?
• Is the data at risk in any way?
• What is the degree of data quality?
• How useful and complete is the data?

Answering these questions provides a complete view of the health of your data and data pipelines.  It also allows your organization to measure the effectiveness and operative use of your data.  Let’s explore each of these in more detail.

Timeliness

Delivering data on a timely basis ensures that analysts and business teams are working from fresh data to make their decisions and see trends as near to real-time as possible.  To ensure timeliness, DataOps teams need to automate and run data pipelines as often as the infrastructure allows and monitor for their clean execution.

Volume

Erratic data volume production in data pipelines can be an indicator that the pipelines are broken and can create unforeseen holes in the resulting analytics.  Not only do DataOps teams need to monitor overall data volume, but they also need checkpoints at different points within the pipeline in order to drill down and identify where data pipelines are broken.

Delivery

Data pipelines can have multiple delivery points for both the finished and intermediate datasets, and data pipelines can also be extended by analysts to produce derivative datasets.  DataOps teams need to monitor if datasets are being properly delivered to their destinations and what those destinations are to ensure proper use of the data.

Formats

A data pipeline with multiple sources and destinations will work with and deliver data in different formats.  DataOps teams need to monitor for format and schema changes, keep them from breaking pipelines, and adjust the pipeline logic as needed.

Data Lineage

The end-to-end lineage of a data pipeline is important for many reasons, including data governance, regulatory compliance, and building trust in the data.  DataOps teams need to have and publish a complete, detailed data lineage that tracks every source, transformation, and destination.

Data Risk

Data risk takes into account the risk of exposing data from a security, privacy, and regulatory control.  While data privacy teams may manage this overall process, DataOps teams should continuously monitor, assess, and govern the risk within their data pipelines.

Data Quality & Consistency

Incomplete and inconsistent data creates potential holes in the end analytics leading to less than optimal decisions and low trust in the data by the business.  DataOps teams need to constantly measure and monitor data quality and completeness, and be able to drill down, identify, and fix problems.

Data Completeness

In the same way poor data quality can hinder data use and trust, data completeness can improve accuracy and context of decisions.  DataOps teams need to monitor the completeness of the data and collaborate with analytics and business to maximize usefulness and completeness.

A logical question one might ask is: shouldn’t data observability be a part of data governance?  Based on our previous exploration of data governance essentials, you can certainly see some convergence, but the two circles do not completely overlap.  Data observability and data governance need to work harmoniously, but each has a slightly different focus and may often be operated by different teams.

There are independent data observability tools emerging on the market, many being from early stage startups.  So this begs the question: do I need a separate data observability platform?

Some organizations build their data stacks with a complex set of multiple platforms and tools, perhaps with open source ones, and perform data movement in one tool and data transformation in other places.  Often the tools in this stack do not have integrated data observation capabilities, forcing an organization to explore independent data observability tools to navigate the complexity.

Deep and well-rounded ETL and data integration platforms such as Datameer have a complete suite of data observability tools that cover all the aspects we have outlined here in addition to the other data integration, DataOps, and governance features.  The integrated data observability capabilities are closely linked to the rest of the platform, ensuring seamless monitoring, measurement, and drill down.

Datameer provides all the key data observability capabilities discussed here, including:

• Complete monitoring and auditing of all statistics and details of data pipeline execution, including timeliness and volume
• Full visibility and drill-down in the formats of data in a pipeline from sources, to intermediate, to destination
• Data lineage with drill-down all the way into each source, transformation, and destination
• A complete view on the data security and privacy aspects of the data within each pipeline for data risk assessment and observation
• A rich and detailed set of data profiling at every point in the pipeline for data quality monitoring
• The largest suite of data transformation, enrichment, organization, and aggregation functions of any data integration tool for data completeness

Read more about these capabilities in the following white papers:

• Data Governance with Datameer technical brief
• Five Ways DataOps Increases Analytics ROI
• Maximizing Data Completeness with Datameer

Or, see Datameer first-hand by scheduling a personalized demo.