Databricks Integration Guide

This guide covers how to connect a Databricks SQL warehouse to BurnLedger, configure query templates, set up a read-only service principal that BurnLedger's write-access gate will accept, and choose the hash scope for your compliance needs. The connector is implemented and registered as connector type databricks.

Databricks integrations fail in two characteristic places, and neither failure message points at the actual cause. First, the read-only gate: on Free Edition and standard workspaces, a built-in workspace-users group carries CREATE-class grants on the default schema, so a service principal you granted nothing but SELECT is still classified write-capable and rejected with WRITE_ACCESS_DETECTED — the fix is a dedicated schema, not more revoking. Second, token lifetime: OAuth service-principal access tokens expire after about an hour, and a registered system's credential cannot be updated in place, so a system that was healthy at registration quietly starts failing every health check an hour later. Read Read-Only Principal Setup and Tokens and their lifetime before you register a system; those two sections cover the failures that account for nearly every broken Databricks registration.

Connection Configuration

BurnLedger connects to a Databricks SQL warehouse over HTTPS, configured by discrete keys rather than a DSN.

Connection config keys

There is no dsn convenience form for Databricks and no read_only flag: privileges are always introspected on the workspace (see What BurnLedger checks), and a stray read_only key is ignored, never honored. The keys are exactly these snake_case strings in both SDKs — do not write httpPath.

The host must be a bare hostname. The Databricks driver derives the protocol from the host string itself: a leading http/https is stripped by plain prefix match and adopted as the protocol, and the bare host localhost is mapped to plaintext HTTP. So https://dbc-….cloud.databricks.com would connect somewhere you did not write, and a real hostname that merely starts with those four letters — httpbin.example.com becomes plaintext to bin.example.com — would be silently rewritten. BurnLedger refuses all of these at construction (CONNECTION_FAILED, "Databricks host must be a bare hostname") and classifies the config's transport as unknown, which no deployment admits. Paste the hostname only.

Set catalog and schema. They do two things at once: unqualified table names in your query template resolve against them, and the read-only gate scans exactly the securables you configure — the metastore always, the catalog when catalog is set, the schema when both are set. Point them at the schema that actually holds the subject data.

When registering the system via the SDK:

TypeScript

const system = await bl.registerSystem({
  name: "lakehouse-users",
  connectorType: "databricks",
  connectionConfig: {
    host: "dbc-a1b2c3d4-e5f6.cloud.databricks.com",
    http_path: "/sql/1.0/warehouses/1234567890abcdef",
    token: process.env.DATABRICKS_TOKEN!,
    catalog: "workspace",
    schema: "blcert",
  },
  subjectQuery: "SELECT * FROM workspace.blcert.users WHERE email = ?",
});

Python

system = bl.register_system(
    name="lakehouse-users",
    connector_type="databricks",
    connection_config={
        "host": "dbc-a1b2c3d4-e5f6.cloud.databricks.com",
        "http_path": "/sql/1.0/warehouses/1234567890abcdef",
        "token": os.environ["DATABRICKS_TOKEN"],
        "catalog": "workspace",
        "schema": "blcert",
    },
    subject_query="SELECT * FROM workspace.blcert.users WHERE email = ?",
)

Connection requirements:

Every value except port must be a JSON string, and port must be a JSON number — a mistyped value (say "port": "443") fails the config decode with CONNECTION_FAILED ("failed to parse Databricks config") before anything is dialed.

Tokens and their lifetime

token takes any bearer token the workspace accepts for SQL warehouse access. That flexibility hides a trap on each path:

• A workspace admin's personal access token will be rejected. Not because PATs are unsupported — because the gate judges the principal, and an admin inherits CREATE-class grants (for example CREATE SHARE through the workspace admins group) that classify it write-capable. WRITE_ACCESS_DETECTED, correctly. Do not try to onboard with the credential you happen to have in your shell.
• OAuth M2M service-principal tokens expire after about an hour. The token is stored (encrypted) with the system at registration, and there is no API to update a system's connection config in place — the systems API is register / get / list / health-check / deregister. An hour after registering with an OAuth token, health checks and attestations start failing with CONNECTION_FAILED, and the only remedy is registering a fresh system with a fresh token.

So: prefer a long-lived token for a dedicated read-only principal where your workspace can issue one. Where OAuth M2M is the only option (it is what BurnLedger's own daily Databricks re-certification uses), mint the token immediately before registering and run the whole attest → delete → verify flow within the token's lifetime — or mint-and-register per run, deregistering afterwards:

# Mint a fresh access token for the service principal (client-credentials grant)
curl -sS -u "$SP_CLIENT_ID:$SP_OAUTH_SECRET" \
  -d "grant_type=client_credentials&scope=all-apis" \
  "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com/oidc/v1/token" \
  | jq -r .access_token

Pass the result as token. The client ID and OAuth secret come from the service principal you create in Read-Only Principal Setup.

Network Access

The connector dials two kinds of endpoint, both over HTTPS on BurnLedger's egress route:

1. The warehouse endpoint — https://<host>:443<http_path>, which carries the SQL statements.
2. Cloud-storage hosts — when the warehouse returns large result sets as CloudFetch links, the row data behind the record hashes is downloaded directly from the workspace's cloud storage (an S3/Azure/GCS host that is not the Databricks host). This leg is only exercised when rows are actually fetched (Merkle proof mode); the count query returns a single row and never rides it. Both legs are resolved and SSRF-validated by the same egress policy.

What that means for you:

• On BurnLedger Cloud all traffic originates from BurnLedger; there is nothing to install in your VPC. If your workspace restricts inbound access by IP, the BurnLedger egress address must be allowed — otherwise every connection fails as a generic CONNECTION_FAILED.
• On a self-hosted deployment, allow outbound TCP 443 from the BurnLedger host to the workspace hostname and to the workspace's cloud-storage endpoints. A host that resolves to a private range additionally needs DP_ALLOW_PRIVATE_NETWORKS=true; on BurnLedger Cloud private addresses are always blocked.
• localhost is refused outright (see the bare-hostname warning above) — there is no loopback testing path for this connector.

Query Template Format

A query template is a SELECT statement with ? as the placeholder for the data subject identifier. BurnLedger binds the subject value as a query parameter — never string interpolation.

? is the only placeholder this connector substitutes, and the subject is bound as the single query parameter, so use ? exactly once. There is no universal placeholder across BurnLedger connectors: $1 (PostgreSQL, Redshift, Teradata), @subject (BigQuery), $IDENTIFIER (MongoDB, Elasticsearch, DynamoDB) and {identifier} (S3, GCS, Redis, HBase) belong to other connectors and mean nothing here. A Databricks template without ? is rejected at registration with INVALID_QUERY_TEMPLATE. That rejection is deliberate: a template with no placeholder would run as a fixed query, match nothing for the subject, and let BurnLedger issue a valid, signed verification record asserting "0 records" — a cryptographically signed false negative telling a compliance team there was nothing to delete while the subject's data sits untouched.

Rules (all enforced at registration):

• Must be a single statement beginning with SELECT or WITH … SELECT.
• Must contain ?.
• Must contain a WHERE clause (a template without one would full-table-scan, and is rejected).
• No embedded ; — a single trailing ; is tolerated by the validator but will break execution, because the template is wrapped as a subquery. Leave the semicolon off.
• No data-modifying or DDL keywords (INSERT, UPDATE, DELETE, MERGE, TRUNCATE, CREATE, DROP, ALTER, REFRESH, SET, CALL, …), including inside CTEs.
• No dollar-quoted strings ($$ … $$).
• No FOR UPDATE / FOR SHARE locking clauses.

SELECT * is allowed but produces a warning: adding or dropping a column changes the row hash. Listing columns explicitly is more stable.

Use fully-qualified catalog.schema.table names, or set catalog and schema in the connection config and keep names unqualified — one or the other, consistently. A name that resolves at registration on your session defaults but not on the connector's configured namespace fails at attestation time, not registration time.

Examples:

-- Single table, subject identified by email
SELECT * FROM workspace.blcert.users WHERE email = ?

-- Explicit columns (recommended)
SELECT user_id, email, plan, created_at
FROM workspace.blcert.users
WHERE email = ?

-- Needing the subject in two predicates: bind it once in a CTE
WITH subject AS (SELECT ? AS email)
SELECT u.user_id, u.email, o.order_id
FROM workspace.blcert.users u
JOIN workspace.blcert.orders o ON o.user_email = u.email
JOIN subject s ON u.email = s.email
WHERE u.email = s.email

At attestation time BurnLedger wraps your template as a derived table — SELECT COUNT(*) FROM (SELECT 1 FROM (<your template>) AS _dp_inner LIMIT <max_records+1>) AS _dp_count in count mode, SELECT * FROM (<your template>) AS _dp_inner LIMIT <max_records+1> in Merkle mode — so the template must be valid as a subquery: no trailing semicolon, no statement-level keywords. Databricks SQL warehouses expose no per-statement read-only transaction mode, so the no-write guarantee at runtime rests on the two gates that both run before any query: the read-only credential enforced at construction, and the template validator above. That is exactly why both are strict.

Read-Only Principal Setup

BurnLedger requires a read-only principal and actively verifies it at connection time. This is a security property of the product, not a suggestion: BurnLedger must not be able to modify the data it attests to.

What BurnLedger checks

Immediately after connecting, the connector resolves the principal and enumerates its Unity Catalog grants on the securables you configured — without writing anything:

SELECT current_user();
SHOW GRANTS `<principal>` ON METASTORE;
SHOW GRANTS `<principal>` ON CATALOG `workspace`;           -- when catalog is set
SHOW GRANTS `<principal>` ON SCHEMA `workspace`.`blcert`;   -- when catalog AND schema are set

The credential is rejected with WRITE_ACCESS_DETECTED if any returned action_type is one of:

MODIFY, ALL PRIVILEGES, CREATE, CREATE TABLE, CREATE SCHEMA, CREATE FUNCTION, CREATE MODEL, CREATE VOLUME, CREATE MATERIALIZED VIEW, WRITE_VOLUME, WRITE_FILES, REFRESH, APPLY TAG — or any privilege whose name starts with CREATE (a prefix match, so CREATE SHARE, CREATE EXTERNAL TABLE and any securable type Databricks adds later are caught rather than slipping through).

Read privileges — SELECT, USE CATALOG, USE SCHEMA — pass. Two properties of the check to understand before you debug it:

• Inherited grants count. SHOW GRANTS reports what the principal holds through group membership as well as directly, and the gate judges everything it reports. A principal you granted nothing but SELECT is still write-capable if any group it belongs to holds CREATE on a configured securable. This is the Free Edition gotcha below.
• The gate scans the configured securables only — metastore, then catalog, then schema, as configured. Table-level grants are not enumerated (probing every table is unbounded). Do not grant the BurnLedger principal MODIFY on individual tables just because the check would not catch it.
• If SHOW GRANTS itself errors, BurnLedger fails closed: it cannot prove the credential is read-only, so it refuses with CONNECTION_FAILED ("failed to verify Databricks credential is read-only; ensure the principal can run SHOW GRANTS on the catalog/schema or supply a principal whose grants can be read").

Gotcha: the workspace-users group holds CREATE grants on the default schema

This is the single most common Databricks rejection, and it is the Databricks analogue of Redshift's CREATE ON SCHEMA public TO PUBLIC. On Free Edition (and standard) workspaces, a built-in, non-removable _workspace_users_… group carries CREATE MODEL, CREATE TABLE, CREATE VOLUME, CREATE FUNCTION and CREATE MATERIALIZED VIEW on the default schema (workspace.default). Every principal in the workspace inherits those grants there — so any system configured with schema: "default" is rejected with WRITE_ACCESS_DETECTED, no matter how minimal the principal's own grants are.

The remedy is a dedicated schema the group holds no grants on — you cannot remove the group or usefully strip its baseline. Create a schema such as workspace.blcert, put (or view) the subject tables there, and point both the connection config (catalog/schema) and the query template at it. The gate scans the securables you configure, so with the catalog and the dedicated schema clean, the same principal that was rejected against default is admitted.

Create the read-only principal

1. Create a service principal in the workspace admin settings, and generate an OAuth secret for it (this yields the client ID and secret used to mint tokens — see Tokens and their lifetime). Do not make it a workspace admin, and do not add it to groups that hold CREATE-class grants on your configured securables.
2. Create the dedicated schema and grant read access (as a privileged user, in a SQL editor on the warehouse):

-- 1. The dedicated schema (see the gotcha above)
CREATE SCHEMA IF NOT EXISTS workspace.blcert;

-- 2. Read access for the service principal, by its application ID
GRANT USE CATALOG ON CATALOG workspace TO `<sp-application-id>`;
GRANT USE SCHEMA  ON SCHEMA workspace.blcert TO `<sp-application-id>`;
GRANT SELECT ON TABLE workspace.blcert.users TO `<sp-application-id>`;

1. Give it access to the SQL warehouse ("can use" on the warehouse the http_path names).

Do not grant MODIFY, ALL PRIVILEGES, or any CREATE <object> privilege anywhere on the metastore, the catalog, or the schema you will configure.

Verify before you register

Run exactly what BurnLedger runs, substituting the principal. No returned row may carry a write-flagged action_type:

SHOW GRANTS `<sp-application-id>` ON METASTORE;
SHOW GRANTS `<sp-application-id>` ON CATALOG `workspace`;
SHOW GRANTS `<sp-application-id>` ON SCHEMA `workspace`.`blcert`;

Expect only USE CATALOG, USE SCHEMA and SELECT (on any securable). If a CREATE … row appears against the schema and you did not grant it, look at the principal column: it is almost certainly the workspace-users group baseline, which means you are looking at default (or another schema the group touches) rather than a dedicated one.

Then mint a token for the principal and confirm it can read the subject table — SELECT * FROM workspace.blcert.users LIMIT 1 — before registering. Registration itself re-runs the full gate: a write-capable credential rejects the registration outright with WRITE_ACCESS_DETECTED (HTTP 422); other connection problems store the system unhealthy for you to fix, and healthCheck(systemId) / health_check(system_id) re-runs the whole check on demand.

TLS

There is nothing to configure, which is itself worth stating precisely: the connector builds the driver from discrete options with the protocol fixed to HTTPS, and no config field can reach the scheme. A complete config (host, http_path, token) therefore classifies as verified transport under the connector transport security policy, with two ways to fall to unknown — a missing required field, or a host the driver would rewrite (leading http/https, or localhost; see the bare-hostname warning). unknown is refused on every deployment, including dev deployments running the DP_ALLOW_UNVERIFIED_TRANSPORT opt-out — that switch admits weak links, never unclassifiable ones, so it buys a Databricks config nothing.

There is no ca_cert field for this connector: workspace endpoints present publicly trusted certificates. A TLS-intercepting proxy that re-signs traffic with a private CA will fail the handshake and cannot be trust-rooted here — the connector must reach the workspace's real certificate chain.

The CloudFetch download leg (see Network Access) rides the same injected transport as the warehouse connection, so result-row downloads get the same egress routing and validation as the SQL traffic.

Hash Scope Options

The hash_scope (hashScope) parameter declares what an attestation is meant to cover. If you omit it, it defaults to existence.

What actually runs is chosen by proof_mode, not by hash_scope. On Databricks, hash_scope is recorded on the system and echoed on every attestation, but the Databricks connector does not branch on it — only the object-store connectors vary their behavior by hash_scope. The proof_mode you pass to attest() is what selects the work: proof_mode: "count" (the SDK default) runs the bounded COUNT(*) wrap and records a record count only; proof_mode: "merkle" fetches the rows and builds a Merkle tree. Set hash_scope to state your intent, set proof_mode to get the behavior, and keep the two consistent.

full

Declares that the attestation covers the complete content of every row returned. Pair it with proofMode: "merkle", which is what actually fetches and hashes the rows. Each row is canonicalized from the driver's string representation of every column value, with fields sorted by column name (so SELECT a, b and SELECT b, a hash identically) and NULL as a distinct sentinel — so any change to any column value produces a different hash.

const system = await bl.registerSystem({
  // ...
});

Use when: you need to prove the exact data that existed, or prove data was deleted rather than merely nulled out (nullifying PII fields is not the same as deletion under some interpretations of GDPR).

existence

Declares that the attestation covers only whether rows exist and how many, not their content. Pair it with proofMode: "count" (the SDK default), which is materially cheaper: a single bounded COUNT(*) on the warehouse instead of fetching and canonicalizing every row (large row fetches are also what triggers CloudFetch downloads).

const system = await bl.registerSystem({
  // ...
  hashScope: "existence",
});

Use when: you only need to prove records existed and were later removed, the result set is large, or you would rather not read PII content at all.

Comparison

"Detects" here is bounded by what a merkle certificate actually carries: a signed Merkle root over the row hashes and nothing else — BurnLedger serves no per-record inclusion proof, so a changed value is demonstrable only to someone who already holds every matching row and rebuilds the tree, and from format 7.0 the leaves are keyed inside the enclave so no party, BurnLedger included, can open the root at all (see Certificate Scope §4).

Each column describes the scope paired with its matching proof_mode (full + merkle, existence + count). Because the Databricks connector ignores hash_scope, a mismatched pair gives you the behavior of the proof_mode: registering hash_scope: "existence" and then attesting with proof_mode: "merkle" still reads and hashes every row.

Either way, a result set larger than max_records fails with RECORD_LIMIT_EXCEEDED rather than being silently truncated. Every attestation and verification runs one query on your SQL warehouse, so warehouse compute pricing applies per run.

End-to-End Example

1. Workspace setup (run once)

CREATE SCHEMA IF NOT EXISTS workspace.blcert;

GRANT USE CATALOG ON CATALOG workspace TO `<sp-application-id>`;
GRANT USE SCHEMA  ON SCHEMA workspace.blcert TO `<sp-application-id>`;
GRANT SELECT ON TABLE workspace.blcert.users TO `<sp-application-id>`;

-- Confirm: no CREATE/MODIFY rows on any of the three
SHOW GRANTS `<sp-application-id>` ON METASTORE;
SHOW GRANTS `<sp-application-id>` ON CATALOG `workspace`;
SHOW GRANTS `<sp-application-id>` ON SCHEMA `workspace`.`blcert`;

Mint a token for the service principal (see Tokens and their lifetime) and export it as DATABRICKS_TOKEN. If it is an OAuth M2M token, plan to complete the flow below within the token's roughly one-hour lifetime.

2. Application code

TypeScript

import { BurnLedger } from "burnledger";
import fs from "node:fs/promises";

const bl = new BurnLedger({ apiKey: process.env.BURNLEDGER_API_KEY! });

// Register the system (run once, store the system ID)
const system = await bl.registerSystem({
  name: "lakehouse-users",
  connectorType: "databricks",
  connectionConfig: {
    host: "dbc-a1b2c3d4-e5f6.cloud.databricks.com",
    http_path: "/sql/1.0/warehouses/1234567890abcdef",
    token: process.env.DATABRICKS_TOKEN!,
    catalog: "workspace",
    schema: "blcert",
  },
  subjectQuery:
    "SELECT user_id, email, plan, created_at " +
    "FROM workspace.blcert.users WHERE email = ?",
});

// --- When a deletion request comes in ---

const subjectEmail = "jane.doe@example.com";

// Step 1: attest that the data currently exists
const attestation = await bl.attest(subjectEmail, {
  systemIds: [system.id],
  proofMode: "merkle",
});

for (const s of attestation.systems) {
  console.log(`${s.systemName}: ${s.recordCount} records found`);
}

// Step 2: delete the data with your own tools, e.g. in a Databricks SQL editor:
//   DELETE FROM workspace.blcert.users WHERE email = 'jane.doe@example.com';
// (BurnLedger's credential cannot do this, by design.)

// Step 3: verify the deletion. If any record still matches, no certificate is
// issued — the verification reports the remaining count instead.
const result = await bl.verify(attestation.id, subjectEmail, { timeout: 60 });

for (const s of result.systems) {
  console.log(`${s.systemName}: now ${s.recordCount} records`);
}

// Step 4: download the certificate
if (result.certificate) {
  const pdf = await bl.downloadPdf(result.certificate.id);
  await fs.writeFile(`erasure-cert-${subjectEmail}.pdf`, pdf);
  console.log(`Certificate ${result.certificate.id} (${result.certificate.transparencyStatus})`);
}

Python

import os
from burnledger import BurnLedger

bl = BurnLedger(api_key=os.environ["BURNLEDGER_API_KEY"])

# Register the system (run once, store the system ID)
system = bl.register_system(
    name="lakehouse-users",
    connector_type="databricks",
    connection_config={
        "host": "dbc-a1b2c3d4-e5f6.cloud.databricks.com",
        "http_path": "/sql/1.0/warehouses/1234567890abcdef",
        "token": os.environ["DATABRICKS_TOKEN"],
        "catalog": "workspace",
        "schema": "blcert",
    },
    subject_query=(
        "SELECT user_id, email, plan, created_at "
        "FROM workspace.blcert.users WHERE email = ?"
    ),
)

subject_email = "jane.doe@example.com"

# Step 1: attest that the data currently exists
attestation = bl.attest(
    subject_email,
    system_ids=[system.id],
    proof_mode="merkle",
)
for s in attestation.systems:
    print(f"{s.system_name}: {s.record_count} records found")

# Step 2: delete the data with your own tools, e.g. in a Databricks SQL editor:
#   DELETE FROM workspace.blcert.users WHERE email = 'jane.doe@example.com';

# Step 3: verify the deletion (refused, with the remaining count, if any
# record still matches)
result = bl.verify(attestation.id, subject_email, timeout=60)
for s in result.systems:
    print(f"{s.system_name}: now {s.record_count} records")

# Step 4: download the certificate
if result.certificate:
    bl.save_pdf(result.certificate.id, f"erasure-cert-{subject_email}.pdf")
    print(f"Certificate issued: {result.certificate.id}")

If you registered with a short-lived OAuth token, note that Step 3 opens a fresh connection to the warehouse: the token must still be valid at verify time, not just at attest time.

What a deletion leaves behind

Two things survive a DELETE against a Delta table, and both are visible to anyone holding SELECT on it. Neither is a BurnLedger behaviour — they are how Delta works — but a verification record says the records were removed, and it is worth knowing exactly what that does and does not mean.

The rows remain restorable until VACUUM. Delta time travel keeps the files the delete removed, bounded by delta.deletedFileRetentionDuration (default 7 days). RESTORE TABLE … VERSION AS OF brings them back, using the customer's own credential and no operator. BurnLedger does not currently probe for this — see ADR-024, which records why the probe cannot be built soundly today — so a Databricks certification asserts the rows are absent from the current table, not that no restorable copy exists. Run VACUUM with a retention shorter than your window, and confirm with DESCRIBE HISTORY, if the deletion has to be irreversible.

The subject's identifier remains in the table history. DESCRIBE HISTORY records each operation with its predicate, verbatim:

DELETE | {"predicate":"[\"(email#14739 = alice@example.com)\"]"} | numDeletedRows: 1

So a deletion performed by matching an email leaves that email in the history for delta.logRetentionDuration (default 30 days), readable by every principal that can read the table — including the read-only principal BurnLedger uses. The row is gone; the identifier that named it is not.

What makes this worth calling out is the privilege it sits behind, not the retention. Most engines keep the statement text somewhere; on Snowflake, the comparable record is the query history, and a role holding only SELECT on the table cannot read another principal's statements — INFORMATION_SCHEMA.QUERY_HISTORY() returns just the caller's own, and SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY is refused outright (checked under BURNLEDGER_RO_ROLE, 2026-09-06). Delta's history is table metadata: every principal that can read the table can read it, including the read-only principal BurnLedger itself uses. The residue is the same; the audience is much wider.

If that matters for your obligations, the options are to delete by a surrogate key rather than by the identifier, to shorten delta.logRetentionDuration, or to accept it and record the decision. Observed on Databricks Runtime 19.2 on 2026-09-06.

Troubleshooting