Elasticsearch Integration Guide

This guide covers how to connect an Elasticsearch cluster to BurnLedger, configure query templates, and set up read-only credentials.

Connection Configuration

BurnLedger connects to Elasticsearch using the HTTP API with basic authentication.

Configuration fields:

These are the exact JSON keys the connector reads. read_only and ca_cert are snake_case in both SDKs — do not write readOnly or caCert.

TypeScript

const system = await bl.registerSystem({
  name: "user-events",
  connectorType: "elasticsearch",
  connectionConfig: {
    addresses: ["https://es.example.com:9200"],
    username: "burnledger_ro",
    password: process.env.ES_PASSWORD,
    index: "user-events",
  },
  subjectQuery: '{"query": {"term": {"user_email": "$IDENTIFIER"}}}',
});

Python

system = bl.register_system(
    name="user-events",
    connector_type="elasticsearch",
    connection_config={
        "addresses": ["https://es.example.com:9200"],
        "username": "burnledger_ro",
        "password": os.environ["ES_PASSWORD"],
        "index": "user-events",
    },
    subject_query='{"query": {"term": {"user_email": "$IDENTIFIER"}}}',
)

Connection requirements:

Query Template Format

For Elasticsearch, the query template is a JSON object that forms the body of a _count or _search request. Use "$IDENTIFIER" as a string value placeholder for the data subject.

Rules:

• Must be valid JSON. Enforced at registration and again before every query.
• Must contain at least one "$IDENTIFIER" string value. This is enforced: a template without it is rejected at registration and at query time, because it would match the entire index and could certify a false "0 records" result.
• "$IDENTIFIER" is replaced with the actual subject value as a string (not interpolated — the JSON tree is walked and string values matching "$IDENTIFIER" are swapped).
• Do not use script queries. The body is sent verbatim to _count/_search, so a script would execute server-side; BurnLedger does not parse it out. Scope the credential so scripts cannot do damage.

Examples:

// Simple term match
{"query": {"term": {"user_email": "$IDENTIFIER"}}}

// Match on a keyword field
{"query": {"term": {"user_id.keyword": "$IDENTIFIER"}}}

// Bool query across multiple fields
{"query": {"bool": {"should": [
  {"term": {"email": "$IDENTIFIER"}},
  {"term": {"secondary_email": "$IDENTIFIER"}}
]}}}

// Nested field
{"query": {"nested": {
  "path": "contacts",
  "query": {"term": {"contacts.email": "$IDENTIFIER"}}
}}}

How it works during attestation:

1. BurnLedger parses the JSON template, walks the tree, and replaces "$IDENTIFIER" string values with the actual subject.
2. For count mode: calls POST /{index}/_count with the query body.
3. For Merkle mode: calls POST /{index}/_search with search_after pagination, sorting by _doc for stable ordering.
4. Only _source content is hashed. Metadata (_id, _score, _index) is excluded.

Read-Only User Setup

BurnLedger requires credentials that cannot modify data. It detects write access during connection validation and rejects credentials that have it — without ever writing. It calls _security/user/_has_privileges to ask the cluster whether the user holds index write privileges; if so, construction is refused.

This works automatically whenever the cluster has security enabled. If the cluster has security disabled (the _has_privileges API is unavailable), set "read_only": true in the connector config to assert read-only; otherwise construction is refused (fails closed).

Elasticsearch Security (X-Pack / built-in)

Create a role with read-only permissions:

PUT _security/role/burnledger_readonly
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["user-events", "user-profiles"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

Create the user:

PUT _security/user/burnledger_ro
{
  "password": "your-secure-password",
  "roles": ["burnledger_readonly"],
  "full_name": "BurnLedger Read-Only"
}

Elastic Cloud

In the Elastic Cloud console:

1. Go to Deployments > Security > Users.
2. Create a new user.
3. Assign a custom role with read and view_index_metadata on the target indices only.

OpenSearch

PUT _plugins/_security/api/roles/burnledger_readonly
{
  "cluster_permissions": ["cluster_monitor"],
  "index_permissions": [
    {
      "index_patterns": ["user-events*"],
      "allowed_actions": ["read", "indices_monitor"]
    }
  ]
}

PUT _plugins/_security/api/rolesmapping/burnledger_readonly
{
  "users": ["burnledger_ro"]
}

Verify permissions

# These should succeed
curl -u burnledger_ro:password "https://es.example.com:9200/user-events/_count"
curl -u burnledger_ro:password "https://es.example.com:9200/user-events/_search?size=1"

# These should fail with 403
curl -u burnledger_ro:password -X PUT "https://es.example.com:9200/test-index"
curl -u burnledger_ro:password -X POST "https://es.example.com:9200/user-events/_doc" -H 'Content-Type: application/json' -d '{"test": true}'

Endpoint role

POST /v1/systems/test-connection reports replication.role: no_primary for Elasticsearch, and the reason is worth reading rather than skipping: Elasticsearch 7.0 removed the _primary search preference. A search is answered by whichever copy of each shard the cluster picks, and there is no request BurnLedger could send that would pin it to the primaries. A preference value is a routing hint, not a consistency guarantee, so none is sent — a replica shard that has not caught up returns fewer hits than the primary holds.

What BurnLedger checks instead is the _shards block on every response: a search whose shards did not all report is refused, not counted, because Elasticsearch answers 200 with a reduced count rather than an error. See endpoint role.

Hash Scope

hash_scope is accepted on the system record for every connector, but the Elasticsearch connector does not read it — it only changes behavior for the object/document-store connectors (S3, GCS, Azure Blob, MarkLogic). Setting hashScope: "existence" on an Elasticsearch system does not switch to a metadata-only hash; it has no effect at all.

What actually varies for Elasticsearch is the proof mode of the attestation:

const system = await bl.registerSystem({
  // ...
});

Important: _source must be enabled on the index for Merkle mode. If _source is disabled, BurnLedger returns SOURCE_DISABLED at connection time.

End-to-End Example

TypeScript

import { BurnLedger } from "burnledger";
import fs from "node:fs/promises";

const bl = new BurnLedger({
  apiKey: process.env.BURNLEDGER_API_KEY,
});

const system = await bl.registerSystem({
  name: "user-events",
  connectorType: "elasticsearch",
  connectionConfig: {
    addresses: ["https://es.example.com:9200"],
    username: "burnledger_ro",
    password: process.env.ES_PASSWORD,
    index: "user-events",
  },
  subjectQuery: '{"query": {"term": {"user_email": "$IDENTIFIER"}}}',
});

const attestation = await bl.attest("jane.doe@example.com", {
  systemIds: [system.id],
  proofMode: "merkle",
});

for (const s of attestation.systems) {
  console.log(`Found ${s.recordCount} documents`);
}

// Delete in your application
// await esClient.deleteByQuery({ index: "user-events", body: { query: { term: { user_email: "jane.doe@example.com" } } } });

// The certificate is ISSUED BY verify() once the data is gone -- there is
// no separate "create certificate" step.
const result = await bl.verify(attestation.id, "jane.doe@example.com", { timeout: 60 });

if (result.certificate) {
  await bl.savePdf(result.certificate.id, `es-deletion-cert-${result.certificate.id}.pdf`);
  console.log(`Certificate issued: ${result.certificate.id}`);
} else {
  console.log(`Not certified: status ${result.status}`);
}

Python

import os
from burnledger import BurnLedger

bl = BurnLedger(api_key=os.environ["BURNLEDGER_API_KEY"])

system = bl.register_system(
    name="user-events",
    connector_type="elasticsearch",
    connection_config={
        "addresses": ["https://es.example.com:9200"],
        "username": "burnledger_ro",
        "password": os.environ["ES_PASSWORD"],
        "index": "user-events",
    },
    subject_query='{"query": {"term": {"user_email": "$IDENTIFIER"}}}',
)

attestation = bl.attest(
    "jane.doe@example.com",
    system_ids=[system.id],
    proof_mode="merkle",
)

for s in attestation.systems:
    print(f"Found {s.record_count} documents")

# Delete in your application
# es_client.delete_by_query(index="user-events", body={"query": {"term": {"user_email": "jane.doe@example.com"}}})

# The certificate is ISSUED BY verify() once the data is gone -- there is
# no separate "create certificate" step.
result = bl.verify(attestation.id, "jane.doe@example.com", timeout=60)

if result.certificate:
    bl.save_pdf(result.certificate.id, f"es-deletion-cert-{result.certificate.id}.pdf")
    print(f"Certificate issued: {result.certificate.id}")
else:
    print(f"Not certified: status {result.status.value}")

Troubleshooting