churn_features.ipynb
scheduled
import spark
features = spark.table(
"analytics.orders")
df = features.select("customer_id")
The notebook someone
scheduled two years ago.
It reads a table by name. It is on nobody’s list.
Don’t migrate an estate
you haven’t mapped.
Callisto maps your estate across platforms, column by column, and flags the systems it can’t read yet.
Callisto
Reach
analytics.orders · customer_id
Warehouse & transform
Pipelines & syncs
Consumers
analytics.ordersraw.ordersstg_ordersfct_revenueload_orders
app.ordersdaily_rollup
orders_to_crmrevenue_daily
churn_features.ipynbUnmapped namespace· not yet mappedGitHubSnowflakeOpenLineagedbtAirflow
One map across all of them.
01 · The problem
A migration plan is only as
good as the map behind it.
Warehouse lineage covers the warehouse. Plans get scoped from it, and what sits around the warehouse is found late.
The notebook someone
orders_to_crmReverse-ETL sync · undocumented
Field map
customer_idAccount ID
amountLifetime value
The reverse-ETL sync
nobody documented.
It copies columns into another system.
Change the source and it loses its input.
revenue_dailyTableau extract · dashboard with readers
customer_id
order_date
amount
field
The Tableau extract
three systems away.
A dashboard with readers, well past the table
you are moving.
02 · Who it’s for
Who needs the whole map
The architect asks what the estate contains. The data lead asks whether the plan covers all of it.
Data architect
“What does this estate
actually contain?”
What keeps them up
A migration plan built on an incomplete map.
What they get
The estate mapped across platforms, with the gaps labelled.
data.architect
What reads analytics.orders outside dbt?
callistoapp
These systems read analytics.orders outside dbt:
Airflow ·
daily_rollupTableau · revenue_dailyReverse-ETL ·
orders_to_crmNotebook ·
churn_features (inferred)1 namespace not yet mapped
Enterprise data lead
“What if a system isn’t covered?”
What keeps them up
Warehouse lineage covers the warehouse. Plans get scoped from it, and what sits around the warehouse is found late.
What they get
Callisto maps your estate across platforms, column by column, and flags the systems it can’t read yet.
data.lead
What is in the estate?
callistoapp
Systems touching analytics.orders
Snowflake ·
raw.ordersdbt · stg_ordersAirflow ·
load_ordersTableau · revenue_dailyUnmapped namespace · not yet mapped1 namespace not yet mapped
03 · How the map is built
The map a data migration
risk assessment depends on.
Across platforms, down to the column, with the gaps labelled.
dbt
Snowflake
Avro/S3
Airflow
Catalog
raw.ordersSnowflake
orders.avroAvro/S3
load_ordersAirflow
One catalog across platforms.
OpenLineage, Snowflake, Avro/S3 and systems that aren’t dbt, in one cross-platform catalog.
stg_orders.sql
select customer_id, amount as order_total from raw.orders
raw.orders
stg_orders
customer_id
amount
customer_id
order_total
parsed
parsed
Column by column.
Column-level lineage is derived by parsing each job’s SQL, with no manual curation.
analytics.ordersSnowflake
churn_features.ipynbNotebook
inferredmatched by name
Unmapped namespacenot yet mapped
Gaps are labelled, not hidden.
Edges matched by name are marked inferred. Systems Callisto can’t read yet are shown as not yet mapped.
stg_ordersdbtanalytics.ordershighlightedorders_to_crmReverse-ETL
The map, as of a point in time.
Metadata history is versioned, so you can read the map as it stood at an earlier point.
load_ordersV-oldV-new
Duration
Rows
Bytes
Scanned bytes
Design time and runtime.
Lineage shows what depends on what. Runtime history shows how the jobs ran: duration, rows, bytes and scanned bytes across job versions.
04 · How it works
From estate
to pull request
Once the work reaches a pull request, Callisto checks it against the same map.
Read-only, except for the comments it posts on pull requests.
Estate
analytics.orders · customer_id
Sources read
- SnowflakeQuery history and view SQL
- OpenLineageRun events from any emitter
- dbtManifest and compiled SQL
- AirflowDAG code and task SQL
- GitHubPull request diffs
- Avro/S3Schemas of files on S3
Systems touching analytics.orders
- raw.ordersSnowflake
- stg_ordersdbt
- load_ordersAirflow
- orders_to_crmReverse-ETL
- revenue_dailyTableau
- churn_features.ipynbinferred
These systems read analytics.orders outside dbt
- daily_rollupAirflow
- revenue_dailyTableau
- orders_to_crmReverse-ETL
- churn_featuresinferred
- Unmapped namespacenot yet mapped
Illustrative sampleRead-only
05 · Is this you?
Where it fits, and where it doesn’t.
Fits
- You are planning or running a migration, consolidation or AI-enablement programme.
- Snowflake plus at least two other platforms.
- Something consumes data outside the warehouse: BI, reverse-ETL, notebooks or applications.
- Airflow, Dagster or similar, and more than one team touching the same tables.
Not a fit
- Single-platform shops.
- Small teams where one person holds the whole picture.
- Anyone looking for a data catalog, a glossary or data quality monitoring. Those are different products.
Callisto does not move data or run the migration. It is read-only outside the comments it posts.
For consultancies and SIs →Faster estate discovery, and a tool the client keeps using after handover.
Each tool is right about its own platform. The breaks happen in the edges between them.
06 · FAQ
FAQs
It is the work of finding out what depends on what before you move it. Callisto supplies the map that work relies on: lineage across platforms, with the gaps labelled.
Join waitlist
Map the estate before you move it.
Join the waitlist to see the map.
Single-tenant deployment. Callisto reads metadata, lineage and run history. It is read-only outside the comments it posts.
Estate map
app.ordersPostgres
raw.ordersSnowflake
analytics.ordersSnowflake
revenue_dailyTableau
churn_featuresinferred
Unmappednot yet mapped