Snowflake Integration To Connect On-Prem Data Silos To Cloud
- Adam Wealand
- June 11, 2020
Datameer Spotlight Introduces Native Snowflake Integration to Bridge Organization’s On-Premise Data Silos With Snowflake Data Cloud
We are excited to formally announce that Datameer Spotlight provides native integration with Snowflake data warehouses!
Datameer Spotlight’s native Snowflake integration performs advanced queries and searches directly in Snowflake. This saves analytics practitioners valuable time and reduces network data costs compared to generic integration. Besides, since Datameer Spotlight provides virtualized access to data, it eliminates the need to move data (which keeps it secure) and reduces the need for complex, expensive upfront data engineering projects. Datameer Spotlight’s native integration and optimization with Snowflake eliminates the bottlenecks in distributed query providing faster response times to and from Snowflake virtual warehouses.
What is Native Snowflake Integration for Datameer Spotlight Customers?
Datameer Spotlight enables scalable and governed self-service analytics, utilizing native security and performance functionality, without the need to move data. Datameer Spotlight’s native integration and optimization with Snowflake eliminates the bottlenecks in distributed query providing faster response times to and from Snowflake virtual warehouses.
Due to its inherent difficulty, the complexity of problems related to distributed query optimization continues to expand. To solve this issue, Datameer Spotlight provides intelligent, native integration with Snowflake in two key areas – push down processing and smart caching.
Push Down Processing
With data storage continually growing, the cost of computing is also increasing. It becomes vital to look for alternate methods to improve query speed and efficiency. With Datameer Spotlight’s pushdown query processing, SQL queries and transformation logic can be “pushed” to data residing in Snowflake in the form of generated SQL statements. So, rather than bringing the data to processing logic, Datameer Spotlight takes the logic to where data resides in Snowflake.
Push down processing can be thought of as the best-place processing for your data. The principle of pushdown (queries) is to process an operation that reduces the number of rows earlier in the process, so later processes – and computation resources – have to perform work on fewer rows. For example, Datameer Spotlight will push SQL statements (i.e., filter, join, etc.) deeper in the query order to inside a Snowflake warehouse to read all rows, remove rows that do not match the statement, and then only perform the computation for the remaining rows. This saves the work from doing computation for the rows that do not match.
Smart Caching Option For Snowflake
Datameer Spotlight instances can be configured to cache datasets using existing Snowflake data warehouses instead of using Datameer Spotlight-native cloud storage. Datameer Spotlight users have the option to run queries on uncached (direct connect) versions of their Snowflake tables and run queries on the cache.
Benefits of Datameer Spotlight’s Smart Caching
Datameer Spotlight’s cache infrastructure offers the following benefits over a direct connection to Snowflake:
- Compatibility with multiple database types
- Reduction in load time on Snowflake databases
- Leverage Materialized Views to improve query performance further
- Store large amounts of data with relatively low compute demand
- Lower variability in Snowflake compute demand
With data caching, Datameer Spotlight provides unmatched performance tuning to Snowflake query performance. That allows users to optimize their Snowflake’s compute resources fully.
Using Datameer Spotlight For Smart Cloud Analytics With Snowflake Data Cloud
Last week, Snowflake unveiled their Data Cloud – a one-stop-shop where organizations can execute a full range of data-oriented tasks. Data Cloud is essentially an ecosystem of partners, customers, data providers, and data service providers meant to bridge data silos. In theory, Data Cloud should make it easier to migrate and provision data in Snowflake and remove the complexity of dealing with multiple cloud providers and regions. With Data Cloud, Snowflake hopes to become the hub of an organization’s data activities as they migrate their data to the cloud.
Public clouds’ benefits are generally understood, and cloud migration from on-premise has accelerated in recent years. That said, on-premises data transfers can be complex and prone to failure. Cloud migration requires strategy and time. Most data in large organizations will be in a hybrid environment, with some data in public clouds and some data remaining on-prem.
With Datameer Spotlight, the analytics community can discover, share, blend, and perform analytics and ML across cloud and on-premise data sets. Datameer Spotlight is a virtual layer that bridges the two worlds. With Datameer Spotlight, analysts can work on datasets spread across Snowflake Data Cloud and their organization’s legacy on-prem systems. Using Datameer Spotlight, organizations take advantage of the instant, elastic, and unified data platform that Snowflake Data Cloud plans to offer while reducing the risks, time, and resources associated with migrating massive on-premise data workloads in the cloud.
Now you can harness Datameer Spotlight to push down data wherever it resides directly into Snowflake for queries and processing. This new capability enables data and analytics teams to have easy access to large volumes of dispersed data. Data consumers directly query and configure Snowflake data via Datameer Spotlight for their specific business needs. Then, they directly connect to or cache that data and analyze it in the business intelligence (BI) or machine learning (ML) tools of their choice.
See Native Snowflake Integration In Action
You can view a recording of our Snowflake webinar where we demonstrated our native Snowflake integration here. We also have live, instructor-led classes where you can try Datameer Spotlight and Snowflake in a sandbox environment.
Or test-drive Datameer Spotlight on your own to see how it can quickly answer ad-hoc analytics questions here.