Modern architecture for the data and analytics driven enterprise
Centralized vs. Decentralized
Data centralization used to be the end goal—the pinnacle of data excellence. But more often than not, it turns into a game of whack-a-mole as data engineers and architects can’t possibly keep up with an ever-growing volume of data sources. Datameer Spotlight complements an organization’s data centralization efforts by providing a unified layer to bridge data lakes, data warehouses, data marts, and SaaS applications (see all integrations here) across the enterprise.
Distributed Data Model and the Virtual Data Lake
Datameer Spotlight acts as a “virtual” data lake. With Datameer Spotlight, data does not need to be physically in the same data warehouse or lake for your team to discover, query, or manipulate it. Datameer Spotlight lets you search and perform standard SQL operations across data sources as if they were in the same physical warehouse—making rogue and shadow data marts a thing of the past. Datameer Spotlight lets your business build as many virtual data marts as it wants, with zero data replication. Data remains at the source, and Datameer Spotlight only stores the query logic.
Eliminate Costly & Risky Data Replication
Datameer Spotlight reduces data engineering costs for complicated ETL processes, the cost of moving data and the risk of trying to govern multiple copies of data. Spotlight lets your business build as many virtual data marts they need without complicated data engineering and with zero data replication. Data remains at the source where data teams can make sure it is secure and governed.
Unified Access Layer & Decoupling
Datameer Spotlight acts as the gateway between the organization’s data sources and the business users who consume the data. Datameer Spotlight provides a single access layer (or semantic layer) for all data consumption applications (e.g., BI and data science tools) on one end and bridge all data silos on the other. This unified access layer, in addition, enables decoupling of data sources from the tools that consume them.
1 Unified Data Inventory
Datameer Spotlight automatically builds and keeps an up-to-date and searchable inventory of technical metadata AND lets data consumers create a business level semantic layer making it easy for anyone to discover the data they are allowed to see.
2 Data Profiling & Data Lineage
Datameer Spotlight provides a clear, business understandable representation of datasets, including a complete profile of the data and full lineage. This transparency helps promote trust and data literacy.
3 Unified Data Models & Decoupling
Datameer Spotlight allows for decoupling of your BI and data science tools from their underlying data assets. For example, each BI vendor has its own proprietary flavor of SQL, and subtle variations in query results could possibly occur across identical datasets. With a common layer for data modeling in place, analysts can now use their preferred BI tool and trust that they’ll get the same results as they would using other tools.
Self Service and Collaborative Analytics
Domain experts know their data better than anyone else. Datameer Spotlight enables these business domain experts to collaborate in the data engineering curation effort.
Ultra Fast Queries
Datameer Spotlight uses Spark as its query engine. The platform also leverages AI to distribute queries across systems and prioritizes them using “best place processing” to optimize for speed and reduce data transfers over the wire
Keep Data Secure and Governed
DAMA International defines data governance as “the exercise of authority, control, and shared decision-making over the management of data assets.” Datameer Spotlight is uniquely positioned to achieve the optimal balance between IT and data engineering when it comes to control over data quality, privacy, and security along with the business user’s need for collaboration, flexibility, productivity, and autonomy in their analytics work. The virtual access layer lets data remain in place and works with in-place security and governance, eliminating the need to replicate access controls.
1 Datameer Spotlight provides 3 layers of governance
First, Datameer Spotlight preserves any data security, privacy, access, and policies that the data engineering team set at the data source level. A business user can’t access or perform operations on any assets if they don’t have permission to access the underlying data sources.
Second, Datameer Spotlight is adding a wiki-style approach—i.e., leveraging the wisdom of the crowd—to manage the quality, trust, and usability of datasets. Spotlight scales the data governance efforts in an enterprise by letting the business teams contribute while continuing to let IT teams manage overall governance.
2 Regulatory Compliance
Datameer Spotlight provides full auditing of data usage and complete lineage, enabling teams to report on and maintain regulatory compliance.
All data transmitted and stored by Datameer Spotlight, both at rest and in motion, is encrypted using industry-standard TLS (Transport Level Security).