Is Analytics Engineering a Necessary Evil?
- John Morrell
- February 10, 2022
And Why Analytics Engineering Should be Self-Service
Analytics engineer is a new role in the analytics lifecycle being discussed by many. This new role is emerging due to the data engineering talent shortage, which is causing data engineering resources to be stretched thin and projects to get delayed.
Until now, data and analytics teams had three main roles: data engineer, data analyst, and data scientist. These personas might not reside in the same department – many analysts and data scientists are dispersed out in the business – but would be a virtual team for projects.
To date, analytics projects have a strong dependency on data engineering resources. A recent Cloud Data and Analytics Engineering survey showed that 76% of organizations relied completely or partially on data engineering resources for data modeling and transformation.
Once the data is loaded into the cloud data warehouse, who in your organization models and transforms the data for your analytics?
Source: Datameer Data and Analytics Engineering Survey, 2022
What is an Analytic Engineer?
With such a strong dependency on limited data engineering resources, analytics teams are seeking ways to take charge and eliminate this bottleneck. Creating a specialized role of analytics engineer is one approach. This Medium article suggests such a new role should be created.
An analytics engineer would bridge the data producers (data engineers) and data consumers (data analysts and data scientists in the business). In this role, they would transform data provided by the data engineers into usable datasets for the analysts. In addition, they would be responsible for best practices around the data – testing, validation, documentation, etc.
An analytics engineer would ideally be one of the following:
- An analytics team member with greater technical skills and knowledge of how to work with data, or
- A former data team member that has a good grasp of how the business wants to use various data assets
Why Do We Need a New Role?
With analytics cycles long and complex as it is, the question we need to ask ourselves is: would adding a new role in the producer to consumer chain make our analytics cycles faster and more effective? If the following are true, you may not need additional analytics engineering resources:
- Your data engineering is doing a good job continuously getting raw data into your cloud data warehouse and performing basic transformations.
- Your data analysts have a solid grasp of what data is there, how they want to use it and have the right tools to transform data themselves.
Analytics Engineering Should be Self-Service
In a recent blog post, How a Major Manufacturer Creates Responsible Self-Service Data with Datameer , we outlined how one major manufacturer implemented data self-service responsibly and effectively. This manufacturer did not want to hire more engineers. They wanted to make their team more effective as a unit and allow data engineers and analytics team members to use their knowledge and skills.
Using a data modeling and transformation platform with a multi-persona (SQL, Low-code, No-code) user experience and services that facilitate engineering best practices eliminates the need to hire specialized analytics engineering roles. Data engineers and analytics team members can collaborate around their data and data transformation processes to best use their skills and knowledge to facilitate a smooth and efficient flow of data between producers and consumers.
Using a collaborative, multi-persona data modeling and transformation process will enable:
- Data engineers to be more effective at delivering basic consumable datasets for analytics team members to use, keeping data engineering costs in line,
- Analytics team members to have data self-service to model and transform data on their own, lowering their dependency on data engineering and speeding analytics cycles,
- Engineering best practices to be in place for proper promotion and use of data models and transformations to ensure proper, cost-effective use of the cloud data warehouse.
The last point is very important. Simply giving analytics team members free access and use of the cloud data warehouse can waste resources and drive up costs. Your data modeling and transformation tool must have close integration with your cloud data warehouse to support the right processes for best practices.
Datameer
Datameer’s SaaS data transformation platform focuses on the T – transformation – in your ELT or ETLT stack. Datameer is the industry’s first collaborative, multi-persona data transformation platform integrated into Snowflake. The multi-persona UI, with no-code, low-code, and code (SQL) tools, brings together your entire team – data engineers, analytics engineers, analysts, and data scientists – on a single platform to collaboratively transform and model data. Catalog-like data documentation and knowledge sharing facilitate trust in the data and crowd-sourced data governance. Direct integration into Snowflake keeps data secure and lowers costs by leveraging Snowflake’s scalable compute and storage.
Are you interested in learning more about Datameer and how it can deliver agility and collaboration for the “T” in your modern ELT data stack without requiring you to add additional resources? Please visit our website or Sign up for your free trial today!