pii_registry
One row per model and PII field: classification, handling, confidence, and whether it was declared or inferred. The live inventory an auditor asks for on day one.
chameleon_pii is an open-source dbt package that builds a PII registry, a field-level lineage map, and a shred-readiness verdict for every sensitive column in your project. It reads the dbt graph, not your data — the core models bill zero bytes. The first time we ran it on our own warehouse, it flagged a hashed email surrogate that had quietly propagated into a mart table nobody had declared.
Our staging layer carries an email_token column — an HMAC surrogate of the user's email, used as a join key. Somewhere along the way a dbt model selected it into dim_users, a mart-layer table, and from there it was one SELECT away from every dashboard and export in the company. No scanner complained, because scanners look at data values, and a hex digest looks like nothing. No review caught it, because the column was never declared anywhere a review would look.
The package's discovery model flagged it on the first run: a column matching a PII naming pattern, present in a mart table, declared in no schema.yml. The query behind that finding read column names from INFORMATION_SCHEMA — metadata, not rows. The registry and lineage models that put the finding in context billed exactly zero bytes, because they never touch the warehouse at all: they read the dbt graph.
That is the design thesis of chameleon_pii: most PII governance work is a metadata problem, and metadata is free. You should not need a data-scanning platform — or a data-scanning bill — to know where your PII is and where it flows.
Every capability in the package is assigned to a cost plane, and the expensive one is off by default. This is what makes it safe to run on every single dbt build.
Columns are tagged with meta.pii in schema.yml (classification and handling), and anything untagged is picked up by configurable column-name inference. Reads the dbt graph only; zero warehouse queries.
A breadth-first walk of the dbt DAG builds the used-where map: every downstream model each PII field reaches, with hop counts. This is how a staging column gets caught surfacing in a mart. Still zero warehouse queries.
Column names are read from INFORMATION_SCHEMA across your configured datasets and matched against PII naming patterns. Columns that look like PII but were never declared become findings. Names, not values — metadata pricing.
Content scanning inspects sampled column values for email, phone, SSN, credit-card, and IP patterns — catching PII inside free-text fields that no name reveals. TABLESAMPLE plus a maximum_bytes_billed cap keep it bounded, and it is off by default.
Everything materializes as ordinary dbt models in your project's schema. Nothing leaves your warehouse; there is no service to sign up for.
pii_registry
One row per model and PII field: classification, handling, confidence, and whether it was declared or inferred. The live inventory an auditor asks for on day one.
pii_field_lineage
The used-where map: origin model, downstream model, and hop count for every PII field. Answers 'if we delete this user's email, where else does it live?'
pii_discovery
Undeclared PII candidates found by name across your datasets — including raw sources dbt never documented and leaks into mart layers.
pii_shred_readiness
A per-field verdict — READY, AT_RISK, NOT_SHREDDABLE, or UNREGISTERED — for whether that field could actually be erased on request, and where it escapes to.
pii_content_findings
The opt-in data plane: PII patterns found in sampled column values, for the emails and phone numbers hiding inside notes fields.
no_undeclared_pii (CI test)
A dbt test that warns — or fails the build, in CI — when discovery finds PII nobody declared. Shift-left enforcement: undeclared PII becomes a failing pipeline, not a quarterly surprise.
There is no separate catalog to fill in. PII is declared where dbt users already document columns — meta.pii on the column, with a classification (DIRECT_IDENTIFIER, CONTACT, SENSITIVE, QUASI_IDENTIFIER, BEHAVIORAL, SYSTEM_IDENTIFIER) and an optional handling that defaults sensibly from it.
Anything you forget is the point of the rest of the package: name inference records it with INFERRED confidence so you can review and promote it, discovery flags it if it lives in a table dbt never documented, and the CI test makes the omission loud instead of silent.
It is BigQuery-only for now — the discovery and content models use BigQuery-specific SQL. Cross-warehouse support via adapter dispatch is the most-requested next step, and Snowflake would come first.
Graph-based models only see columns documented in schema.yml; that blind spot is exactly what the INFORMATION_SCHEMA discovery plane exists to cover. And name matching is deliberately high-recall: a column called subscription_name will be flagged even though it isn't a person's name. Findings are candidates to review, not verdicts — tune the patterns, promote the real ones, allowlist the rest.
Finding PII is also only half the problem. Once the registry tells you a user's data reaches nine downstream tables, you still have to erase it from all nine — and prove you did. That deletion-and-proof layer is what Chameleon itself does, with crypto-shredding and signed deletion certificates; the package is how your dbt project describes what needs protecting.
Does chameleon_pii read or export my data?
The core models never read row data at all — they read the dbt graph and INFORMATION_SCHEMA column names, and they bill zero to near-zero bytes. The optional content-scanning model samples column values but is off by default, capped by maximum_bytes_billed, and everything it produces stays in your own warehouse. Nothing is sent anywhere.
Which warehouses does the package support?
BigQuery only, currently. The registry and lineage models are graph-based and mostly portable, but discovery and content scanning use BigQuery-specific SQL (INFORMATION_SCHEMA, TABLESAMPLE, regexp_contains). Cross-warehouse support via dbt's adapter.dispatch is on the roadmap, with Snowflake first.
How does it detect PII without scanning data?
Three metadata signals: explicit meta.pii declarations in schema.yml, configurable column-name inference across the dbt graph, and an INFORMATION_SCHEMA scan that flags PII-looking column names in tables dbt never documented. Field-level lineage then propagates every declared or inferred field through the DAG, so a staging column that resurfaces in a mart is caught by graph traversal, not by reading rows.
Can it fail CI when someone adds undeclared PII?
Yes. The package ships a no_undeclared_pii dbt test that runs against the discovery findings. It defaults to warn locally; set its severity to error in CI and a pull request that introduces an undeclared PII column fails the build. An allowlist variable handles accepted false positives.
The category of problem discovery findings represent: personal data that landed in your warehouse outside any governance.
What READY in pii_shred_readiness is measuring against: erasure by key destruction, the strategy that survives copies.
Why row deletion fails in BigQuery and what a provable deletion workflow looks like end to end.