Snowflake Integration To Connect On-Prem Data Silos To Cloud

  • Adam Wealand
  • June 11, 2020

Datameer Spotlight Introduces Native Snowflake Integration to Bridge Organization’s On-Premise Data Silos With Snowflake Data Cloud

We are excited to formally announce that Datameer Spotlight provides native integration with Snowflake data warehouses!

Datameer Spotlight’s native Snowflake integration performs advanced queries and searches directly in Snowflake. This saves analytics practitioners valuable time and reduces network data costs compared to generic integration. Besides, since Datameer Spotlight provides virtualized access to data, it eliminates the need to move data (which keeps it secure) and reduces the need for complex, expensive upfront data engineering projects. Datameer Spotlight’s native integration and optimization with Snowflake eliminates the bottlenecks in distributed query providing faster response times to and from Snowflake virtual warehouses.

What is Native Snowflake Integration for Datameer Spotlight Customers?

Datameer Spotlight enables scalable and governed self-service analytics, utilizing native security and performance functionality, without the need to move data. Datameer Spotlight’s native integration and optimization with Snowflake eliminates the bottlenecks in distributed query providing faster response times to and from Snowflake virtual warehouses.

Due to its inherent difficulty, the complexity of problems related to distributed query optimization continues to expand. To solve this issue, Datameer Spotlight provides intelligent, native integration with Snowflake in two key areas – push down processing and smart caching.

Push Down Processing

With data storage continually growing, the cost of computing is also increasing. It becomes vital to look for alternate methods to improve query speed and efficiency. With Datameer Spotlight’s pushdown query processing, SQL queries and transformation logic can be “pushed” to data residing in Snowflake in the form of generated SQL statements. So, rather than bringing the data to processing logic, Datameer Spotlight takes the logic to where data resides in Snowflake.

Push down processing can be thought of as the best-place processing for your data. The principle of pushdown (queries) is to process an operation that reduces the number of rows earlier in the process, so later processes – and computation resources – have to perform work on fewer rows. For example, Datameer Spotlight will push SQL statements (i.e., filter, join, etc.) deeper in the query order to inside a Snowflake warehouse to read all rows, remove rows that do not match the statement, and then only perform the computation for the remaining rows. This saves the work from doing computation for the rows that do not match.

Smart Caching Option For Snowflake

Datameer Spotlight instances can be configured to cache datasets using existing Snowflake data warehouses instead of using Datameer Spotlight-native cloud storage. Datameer Spotlight users have the option to run queries on uncached (direct connect) versions of their Snowflake tables and run queries on the cache.

Benefits of Datameer Spotlight’s Smart Caching

Datameer Spotlight’s cache infrastructure offers the following benefits over a direct connection to Snowflake:

  • Compatibility with multiple database types
  • Reduction in load time on Snowflake databases
  • Leverage Materialized Views to improve query performance further
  • Store large amounts of data with relatively low compute demand
  • Lower variability in Snowflake compute demand

With data caching, Datameer Spotlight provides unmatched performance tuning to Snowflake query performance.  That allows users to optimize their Snowflake’s compute resources fully.

Using Datameer Spotlight For Smart Cloud Analytics With Snowflake Data Cloud

Last week, Snowflake unveiled their Data Cloud – a one-stop-shop where organizations can execute a full range of data-oriented tasks. Data Cloud is essentially an ecosystem of partners, customers, data providers, and data service providers meant to bridge data silos. In theory, Data Cloud should make it easier to migrate and provision data in Snowflake and remove the complexity of dealing with multiple cloud providers and regions. With Data Cloud, Snowflake hopes to become the hub of an organization’s data activities as they migrate their data to the cloud.

Public clouds’ benefits are generally understood, and cloud migration from on-premise has accelerated in recent years. That said, on-premises data transfers can be complex and prone to failure. Cloud migration requires strategy and time. Most data in large organizations will be in a hybrid environment, with some data in public clouds and some data remaining on-prem.

With Datameer Spotlight, the analytics community can discover, share, blend, and perform analytics and ML across cloud and on-premise data sets. Datameer Spotlight is a virtual layer that bridges the two worlds. With Datameer Spotlight, analysts can work on datasets spread across Snowflake Data Cloud and their organization’s legacy on-prem systems. Using Datameer Spotlight, organizations take advantage of the instant, elastic, and unified data platform that Snowflake Data Cloud plans to offer while reducing the risks, time, and resources associated with migrating massive on-premise data workloads in the cloud.

Summary

Now you can harness Datameer Spotlight to push down data wherever it resides directly into Snowflake for queries and processing. This new capability enables data and analytics teams to have easy access to large volumes of dispersed data. Data consumers directly query and configure Snowflake data via Datameer Spotlight for their specific business needs. Then, they directly connect to or cache that data and analyze it in the business intelligence (BI) or machine learning (ML) tools of their choice.

See Native Snowflake Integration In Action

You can view a recording of our Snowflake webinar where we demonstrated our native Snowflake integration here. We also have live, instructor-led classes where you can try Datameer Spotlight and Snowflake in a sandbox environment.

Or test-drive Datameer Spotlight on your own to see how it can quickly answer ad-hoc analytics questions here.

Subscribe for the Latest Posts

Search

Discover the Top ETL and Data Integration Platforms

Comparison_of_Leading_ETL_And_Data_Integration_Platforms

Featured Blog Posts

The Role of Chief Data Officers (CDOs) in 2020
The Role of Chief Data Officers (CDOs) in 2021

The Chief Data Officers role (CDOs) in 2021 is evolving as CDOs are having quite possibly their m...

  • John Morrell
  • April 3, 2021
Spectrum ETL
Disrupting the no-code cloud ELT market: Datame...

More than just loading Data: Datameer launches Datameer Spectrum ETL++ to disrupt the no-code clo...

  • Press Release
  • February 9, 2021
Google Partners with Datameer
Datameer Partners with Google Cloud to Deliver ...

Datameer is now a Google Cloud migration partner The partnership will help customers build secure...

  • Press Release
  • December 2, 2020
Datameer Spotlight - Disrupting the traditional central data warehouse model
Disrupting the traditional central data warehou...

The new flagship product from Datameer upends a three-decade-old approach to data analytics ̵...

  • Press Release
  • December 1, 2020
READ ALL

More from Our Blog

Top 5 Fivetran competitors

Top 5 Fivetran Competitors and Alternatives

What is Fivetran?  Fivetran is a cloud-based ELT integration tool that teams can use to synchroni...

  • Justin Reynolds
  • June 15, 2021
The Simplest Road to a Modern Data Stack with Snowflake

The Simplest Road to a Modern Data Stack with S...

The first building block of a cloud data stack starts with Snowflake.  Your analytics engine and/...

  • John Morrell
  • June 14, 2021
Top 5 Matillion Competitors

Top 5 Matillion Competitors and Alternatives

Matillion ETL Review Matillion is a cloud-based ETL tool that enables teams to create and orchest...

  • Justin Reynolds
  • June 10, 2021

Updating your ETL? Your guide to the 10 things to consider when modernizing your ETL.