A few weeks ago, tens of thousands of people packed at Moscone for the Data + AI Summit, and a large share of the agenda touches governance. I spend my year in the rooms where governance stops being a keynote slide and becomes someone’s audit finding.
The question used to be: How do we govern Databricks?
The question now is: How do we stay consistent across Databricks and everything connected to it?
The short version
- Unity Catalog is the right governance authority for Databricks. It’s the wrong anchor for an enterprise that also runs Snowflake, Lake Formation, BigQuery, and Iceberg.
- The fix isn’t ripping out native catalogs. It’s separating where policy is defined from where it’s enforced: author intent once, compile it to each engine’s native controls, keep one reconciled audit.
Give Databricks its due
Unity Catalog is one of the most important governance innovations in the modern stack: unified metadata, fine-grained access control, lineage, auditing, one permissions model. Inside Databricks, it’s the right answer. And Databricks is pushing outward: Unity Catalog Open APIs now extend row- and column-level controls to external engines.
So why not just wait and let it become the enterprise control point? Three reasons:
- Single-vendor concentration. Every extension is anchored to the Databricks roadmap which is the exact dependency multi-engine enterprises are trying to escape.
- The unprioritized long tail. Platforms extend to engines they have commercial reasons to prioritize. Your operational databases are someone else’s problem.
- Reading isn’t owning. “Can read external tables” is not “system of record for who-can-do-what across the estate.”
The enterprise is not a platform. It’s a portfolio, and a portfolio needs a coordinator that isn’t one of the holdings.
Reality of Large Enterprises: Heterogenity
No large enterprise runs one engine. It’s Databricks and Snowflake and Lake Formation and Fabric, SQL Server still running the business, and the same Iceberg table read by four engines that each think they own the governance model.
Each platform enforces its own policies. The governance team is still on the hook for one outcome: consistent access and one answer to “who accessed what.” Configure each platform independently and hope they converge (they won’t). Policies drift, audits don’t reconcile, and the same person gets different entitlements depending on which door they walked through.
Define intent once. Enforce natively everywhere.
Separate the Policy Administration Point from the Policy Enforcement Point.
Centrally define what should never differ by engine: who gets access, what’s sensitive, which purposes and regulations apply. Then each platform enforces its own native machinery: Unity Catalog grants and masks, Lake Formation tag policies, Snowflake row-access policies.
If you own a platform, your next question is the right one: does this add something to my query path? No. The control plane compiles intent into native constructs and pushes them out-of-band; the engine keeps enforcing in its own query path. A reconciliation loop flags drift. Where an engine lacks a native equivalent, the honest model degrades to the nearest control and surfaces the gap, rather than pretending it’s covered.
In production, not pilots
Real deployments, anonymized; numbers are customer-reported and directional.
Delegated enforcement at scale. A Fortune 500 financial-software company runs Lake Formation and Unity Catalog side by side: one administration point, enforcement delegated to each native catalog. Fine-grained access control at that scale isn’t tractable without a layer above the catalogs.
Killing policy explosion. A global advertising group replaced ~2,000 static Unity Catalog policies with ~20 attribute-based ones. When they onboarded Snowflake, the same policies cascaded on day zero. A Fortune 50 telecom reports a similar ~100x reduction. This is the practical answer to the RBAC-at-scale wall. It’s where Databricks itself is heading, which validates the model.
Governance that travels with Iceberg. An enterprise-application provider built an AI lake on Iceberg with five engines on top, and hit the hard truth: access governance doesn’t propagate to every engine automatically. Close that gap or every engine becomes its own policy island.
Nobody ripped out a native catalog. They put a coordination layer above it.
What you’re signing up for
A control plane is another product to license, run, and integrate, and someone owns the policy translations. The question every security lead should ask any vendor is what happens when the control plane is unreachable.
The honest answer: enforcement doesn’t depend on it being online. Compiled policy lives in each engine, so an outage stops new changes from propagating. It doesn’t open access. Fail-static, not fail-open. The cost is change latency, not exposure. If a vendor can’t answer that crisply, that’s your answer.
Questions for the hallway
- CDO / CISO: Where does governance intent live, where is it enforced, and are those honestly the same system?
- Data owners: Does an access grant carry a purpose and an expiry, or is it standing access nobody revisits?
- Platform owners: When central policy changes, how does it reach each engine, and what keeps enforcing if the central layer goes down?
The half this post didn’t cover
Everything above is the data half which is what governance programs were built for. The half they weren’t built for is already running in your environment: AI agents reaching across platforms faster than any human, most with no governance over what they can see or do. Same principle, harder problem, real regulatory clock. That’s Part 2.
Let’s chat further here: https://trust3.ai/demo/