A Better Way To Organize Your Data Models

A recent Medium article outlined a methodology to organize your Dbt models.  The methodology called for a complex folder structure and file naming convention.  While the structured approach was sound, users could easily stray from the methodology and/or get confused with the cryptic names.

In addition, the complex folder and file structure would make it easy for the average data analyst to get lost, not be able to find the models they are looking for, or worse yet, work from the wrong models.  The lack of a search and discovery capability and very limited data documentation makes Dbt difficult to use for the average data analyst.

This shed light on the true nature of Dbt – it is really an integrated development environment for SQL and Jinja, focusing on data transformation.  It is a tool for developers, not your everyday data analyst.

What Should Your Data Transformation Tool Have?

While Dbt copies the older IDE-style model for users, more modern tools focus on four things: low/no-code, collaboration, a rich set of information about objects, and discoverability.  Rather than complex folder structures and file naming conventions, organizing your data models should focus on:

• Shared workspaces where users can find related data models and collaborate on them,
• Catalog-like user-added data documentation and knowledge sharing, and
• Google-like search and discovery capabilities to find just the right data models.

Datameer SaaS data transformation embraces the aforementioned principles of modern tools to get non-programmers involved in data modeling and transformation, deliver a much easier user experience, make the entire team more productive, and get data modeling done faster and with fewer errors. Let’s explore four key ways Datameer does this.

Low-/No-code

Datameer offers a unique hybrid code/no-code toolset that allows the entire data and analytics teams to get involved regardless of their programming skills.  Those who like to code can create models in SQL.  Non-programmers can create models using the low-/no-code UI.  Models defined in SQL and the no-low-code UI can mix and match the same project, letting team members with varying skill sets work together.

The No-/Low-code interface is not just about creating models faster.  It also is less error-prone, and you don’t have to repeat code or schema definitions – everything is derived.  Making your team more productive isn’t just about speed.  It also means getting it done right the first time.

Shared Workspaces

A more modern approach to organization is with shared workspaces.  Shared workspaces in Datameer SaaS allow teams to group data models in a specific project, for specific purposes, or that are related, say, within a use case.  Datameer allows teams to see what the models are and how they are linked together visually within a workspace.  Users can then drill down on the specific models of interest and add new ones to the chain.

Shared workspaces also foster collaboration.  Teams can work together to share models and build them together, and QA them.  Permissions can be set on the workspaces and models for security.  And, as previously mentioned, SQL and no-low-code models can be mix-and-matched in the same project, letting team members with varying skill sets work together.

Rich, User-assisted Data Documentation

Information about data is often sparse or non-existent.  The information is usually spread among wiki pages, metadata management systems, or early versions of data catalogs.  Some data transformation tools such as Dbt claim to generate data documentation, but this is often just taking comments from SQL code and generating a wiki page.

Datameer SaaS facilitates capturing as much information as possible about the data it is working with, the transformations performed, and the resulting data models.  This includes:

• Auto-generated documentation and information such as schema information, transformations performed, data lineage, audits, and certain system-generated properties
• User-supplied documentation such as descriptions, tags, comments, and business metadata.

Discoverability

A final piece in the collaborative workflow story is search and discovery.  Assets, and the information about them, are only valuable if the broader community of analysts and data scientists can find these assets.  Having a complex, cryptic folder structure and naming convention makes it practically impossible to find models.

Datameer SaaS provides a rich faceted search capability, allowing users to easily search and drill down to potential assets they could use.  The search indexes all of the available information discussed above and provides facets users can drill into, such as system and custom properties, tags, and more.

Wrap Up

How you organize your data models is very important.  Over time you will likely build hundreds or even more than a thousand models.  But you should not need a complex methodology that is difficult to use and navigate to organize your data models.  Your data transformation tool should provide an easy way to organize, discover, and see your data models.

Datameer SaaS offers the easiest user experience for any data modeling and transformation tool, helping create a highly communicative and collaborative workflow across the entire team of data engineers, analysts, and data scientists.  Through multi-persona tools, shared workspaces, rich data documentation, tagging, commenting, custom properties, business metadata, and search and discovery, Datameer SaaS gets the entire team working together, communicating, and sharing the workload.  This enables faster, more agile data transformation processes and eliminates errors to ensure data transformation models get done right the first time.

Do you want to see Datameer in action, explore these unique data modeling and transformation capabilities, and discover how Datameer can make your teams far more efficient? Sign up for your free trial today!