Collaborative Data Transformation

The word collaboration refers to the action of working with someone to produce or create something. Collaboration is fundamental to teams working together successfully and delivering the best possible result in our output. Collaboration also leads to a much better work environment and culture within an organization.

Spotlight for Tableau Server - Comparing Spotlight with Tableau Data Management - true collaboration
Data modeling The Cloud Data Warehouse Era icon

What is Collaborative Data Transformation?

collaboration trend

The above definition accurately captures what we mean when we talk of collaborative data transformation workflows – namely, the act of bringing together diverse people in an organization, each with their own unique knowledge, to produce more comprehensive data transformation models.

Ventana Research, and leading research firm on BI and analytics, has devoted research on the use of collaboration in data and analytics this year. As organizations evolve their use of data for more in-depth insights that answer increasingly complex questions, they recognize that collaboration can help them get there faster and eliminate errors.

The core need for collaborative data transformation is to (a) help share the workload across the broader team, (b) allow different team members to bring their unique skills and knowledge, and (c) increase the analytics team’s knowledge of data. A greater understanding of the data transformation assets facilitates greater trust, brings new ideas on using the analytics, and generally produces faster insights with greater detail and accuracy.

A recent survey by the Datameer team showed that knowledge around analytics assets is dispersed in many locations across an organization. Collaborative data transformation can foster a shared understanding of an organization’s analytic assets.

people icon

Personas

Implicit Collaboration people

A modern data transformation workflow embraces the fact that there are multiple personas involves in the analytics engineering process loosely grouped into two buckets: data engineers and data analysts/scientists.  Each of these personas often has:

  • Different skill sets ranging from being highly technical to less technical,
  • Different knowledge and understanding of how data is used,
  • Different focal points on their role and where they can best use their time.

Data engineers tend to know more about the data itself – where it resides, how it is structured and formatted, and how to get it – and less about how the business uses the data.  Their role is highly focused on managing data.

Data analysts and scientists know less about the data itself but have a complete understanding of how the business would use the data and how it would be incorporated into analytics.  They may have varying technical skills but would prefer to spend more time on what they are good at – analysis – and less on coding data transformation.

virtualization

Forms of Collaborative Data Transformation

Collaborative Analytics hands

A collaborative data transformation process for modern analytics engineering might look like:

  • Data analysts/scientists explore and discover the data assets available for their analytics to identify the ones best suited to the problem at hand,
  • Data analysts/scientists iteratively model and transform the data assets they choose to answer the analytics question(s) best and explore the results,
  • The business stakeholders buy off on the model and analytics results created by the data analysts/scientists,
  • The data analysts/scientists pass over the final model to the data/analytics engineering team,
  • The data/analytics engineering team optimizes, tests, secures, and puts into production the data transformation models into the CDW,
  • The data/analytics team puts governance around the new, usable, modeled asset(s) and makes it discoverable for other data analysts/scientists to use.

This process is efficient and relatively self-service.

no code icon

Multi-Persona UIs

Implicit Collaboration people

The different personas may have varying technical skills.  Data engineers will be highly proficient in SQL and using data tools.  Data analysts may have various degrees of SQL knowledge, but in general, maybe less technical and proficient in SQL.  A data scientist may be technical because they need to program data science models but might not be efficient in programming data transformations in SQL.

To this end, Datameer DTaaS supports three different interfaces to support these multiple personas:

  • No-code – for non-technical data and business analysts. This interface is entirely graphical and drag-and-drop.
  • Low-code – for slightly more data-savvy data analysts or for analytics engineers that want greater productivity than coding. This interface is Excel-like, with a wide range of functions and a formula builder wizard.
  • Code – for data and analytics engineers that want to use SQL for the control and optimization that comes with it.

In addition, the different types of models (code, low-code, no-code) can be connected in a common data flow.  This allows the different personas to contribute to the same workflow, using their best-suited modeling skills.  Data engineers can create base models using SQL, then have analysts create their own tailored models using a no-code interface to graphically combine or reshape data or a low-code Excel-like interface where they can further refine the data or enrich it.

Shared Workspaces

Collaborative Analytics people

Data transformation collaboration begins and ends with teams working together in a common workspace to create the best models for their analytics.  In shared workspaces, they can collaborate, share their knowledge, and share the modeling workload.

In Datameer DTaaS, shared workspaces are a foundational component.  Data engineers, data analysts, business analysts, and data scientists can work together in shared workspaces creating and reusing models using the different tools – code, low-code, no-code.  Beyond working together on modeling, the broader team can share knowledge about the workspace and models, including:

  • Adding descriptions, which can both explain data and how best to use it,
  • Applying tags, which can help organize and identify data,
  • Supplying comments, which can add simple ideas around data or enable collaboration,
  • Adding business metadata, which translates technical metadata into business terms
  • Setting properties or status and certification fields, which describe the state of a data model

 

checklist-icon

Benefits

At the highest level, collaborative data transformation facilitates faster, analytics projects with fewer errors and re-work. At the next level, it delivers six key benefits:

  • A more efficient, shared workload among team members
  • Increased knowledge sharing around data models
  • Greater trust in the analytics assets and the results they produce
  • Better leverage of different skills in the data and analytics community
  • A broader inventory of and information about data models

Collaborative data transformation will increase the value the analytics community offers to the organization and bring more significant ROI to analytics programs.

Datameer Dot Green and Navy Blue

Datameer DTaaS

Datameer

Datameer DTaaS creates a highly communicative and collaborative workflow across the entire team of data engineers, analysts, and data scientists.  Through multi-persona tools, shared workspaces, rich data documentation, tagging, commenting, custom properties, business metadata, and search and discovery, Datameer DTaaS gets the entire team working together, communicating, and sharing the workload.  This enables faster, more agile data transformation processes and eliminates errors due to misinterpretation to ensure data transformation models get done right the first time.

No-Code Analytics Built for Snowflake

Try Free Now