MarkLogic Integration Guide

This guide covers how to connect a MarkLogic database to BurnLedger over its REST API, write structured-query templates, set up a genuinely read-only user, and choose the right hash scope for your compliance needs.

BurnLedger talks to MarkLogic through a REST API app server (the /v1/search and /v1/documents endpoints). It never uses XCC, XDBC, or xdmp:eval, and it never writes to your database.

Connection Configuration

MarkLogic is configured with discrete fields, not a connection URI. The field names below are the exact JSON keys the connector reads.

Configuration fields:

Passwords are passed as a discrete JSON field, never inside a URI, so no escaping is needed in the password field itself — supply the literal password. Where you do build a URI by hand (the curl commands below, or any tooling of your own), percent-encode @ : / ? # & in the password (@ → %40, # → %23) or the URI is parsed incorrectly. If you control the credential, the simplest option is a password limited to letters, digits, - and _.

Authentication: MarkLogic REST app servers challenge with HTTP Digest authentication; plain Basic credentials are rejected with 401. BurnLedger answers the digest challenge automatically — you do not configure anything. App servers set to digest, basic, or digest-basic all work. Verification Record, Kerberos and SAML authentication are not supported: the connector only accepts a username and password.

TypeScript

const system = await bl.registerSystem({
  name: "docs-marklogic",
  connectorType: "marklogic",
  connectionConfig: {
    host: "ml.example.com",
    port: 8000,
    username: "burnledger_ro",
    password: process.env.MARKLOGIC_PASSWORD,
    database: "Documents",
    tls: true,
    read_only: true,
  },
  subjectQuery: '{"query":{"queries":[{"collection-query":{"uri":["{identifier}"]}}]}}',
  hashScope: "full",
});

Python

system = bl.register_system(
    name="docs-marklogic",
    connector_type="marklogic",
    connection_config={
        "host": "ml.example.com",
        "port": 8000,
        "username": "burnledger_ro",
        "password": os.environ["MARKLOGIC_PASSWORD"],
        "database": "Documents",
        "tls": True,
        "read_only": True,
    },
    subject_query='{"query":{"queries":[{"collection-query":{"uri":["{identifier}"]}}]}}',
    hash_scope="full",
)

Connection requirements:

Choosing between port and database: pointing port at a REST app server whose content database is already the one you want is the most reliable setup. The database parameter is passed straight through to MarkLogic on each request; whether your credential is permitted to target another database that way depends on your MarkLogic security configuration, which the connector does not inspect.

Query Template Format

A MarkLogic query template is a MarkLogic structured query, as a JSON document — exactly the body you would POST to /v1/search. The placeholder for the data subject identifier is {identifier}.

Rules:

• The template must be valid JSON. A template that does not parse is rejected with INVALID_QUERY_TEMPLATE.
• The template must contain {identifier} at least once, inside a JSON string value. A template without it is rejected with INVALID_QUERY_TEMPLATE.
• {identifier} is the only placeholder this connector substitutes. ?, $1, {{subject}} and friends are not recognised — they are inert text.
• The identifier is substituted into the parsed JSON tree and the query is then re-serialized, so an identifier containing ", }, \ or any other JSON metacharacter is encoded as a plain string value and cannot alter the shape of the query.

Why a missing placeholder is a hard error. A template with no {identifier} matches whatever it matches regardless of the subject — usually nothing, sometimes everything. The "nothing" case is the dangerous one: BurnLedger would count 0 documents and issue a valid, signed verification record stating the subject had no data, while their documents sit untouched. The connector therefore refuses to run such a template at all.

Examples:

// Documents in a per-subject collection (collection named after the subject)
{"query":{"queries":[{"collection-query":{"uri":["{identifier}"]}}]}}

// Documents whose JSON property "email" holds the subject identifier
{"query":{"queries":[{"value-query":{"json-property":["email"],"text":["{identifier}"]}}]}}

// Documents under a per-subject URI prefix
{"query":{"queries":[{"directory-query":{"uri":["/users/{identifier}/"],"infinite":true}}]}}

// Combined: a subject value inside a specific collection
{"query":{"queries":[{"and-query":{"queries":[
  {"collection-query":{"uri":["profiles"]}},
  {"value-query":{"json-property":["user_id"],"text":["{identifier}"]}}
]}}]}}

The placeholder may appear more than once, and may be embedded in a larger string ("/users/{identifier}/" above) — every occurrence in every string leaf is substituted.

Important: design the query to match all documents associated with the data subject that you need to prove exist (or have been deleted). If a subject's documents span several collections or databases, use an or-query, or register multiple systems.

Schema drift

A zero is only certified if the paths the template names still lead to documents. MarkLogic has no schema to refuse a query against: a collection-query on a collection nothing is assigned to, a directory-query under a directory nothing is written to, and a value-query on a JSON property no document carries are all answered with a clean total of 0. A template that was right at registration and silently stopped being right — the application renamed email to email_address, moved from /users/ to /members/, renamed the profiles collection — would otherwise produce a signed record saying the subject's documents are gone while they sit untouched.

So when a search returns no documents, BurnLedger asks the same REST endpoint, under the same credential, whether each path the template names still yields any document — one bounded search per path, pageLength=0:

Every branch of an or-query is checked; not-query and the negative side of and-not-query are not, because a negation that stopped matching can only widen the result. A term that names the subject directly with nothing above it — {"collection-query":{"uri":["{identifier}"]}} — has no convention to check and costs nothing.

Outcomes:

• Every path still yields documents, or the database holds no readable document at all → the zero is reported and can be certified.
• The database holds documents but one of the template's paths yields none → refused as UNDETERMINED, with the path named. This is also what a collection, directory or property that is real but now completely empty looks like; the two are indistinguishable at a point in time, and refusing is the direction that cannot certify a lie.
• The template scopes a term with a field, a path-index or an element attribute → refused on every zero, because nothing cheap reads through a field definition to the property it covers. Name the json-property or element directly instead. (A field that is not defined at all fails earlier and louder: MarkLogic answers XDMP-NOFIELD and the count itself errors.)

What this does not catch, stated plainly:

• A collection name that carries the identifier under a literal prefix ("users/{identifier}") is not checked. Collection names cannot be prefix-matched without the collection lexicon, which your database may not have. Prefer a directory convention or a literal collection combined with a property term if you want the convention covered.
• Document permissions. These probes see exactly what the credential's searches see. If burnledger_ro can reach the database but lacks read permission on the documents themselves, every search — the count and every probe — returns 0, and the zero is certified as an empty database. No call the credential can make reveals documents it may not read. This is why the verification step below insists on a nonzero total for a subject you know has documents, not merely an HTTP 200.

Nothing is asked when the search finds documents — a nonzero count fails the deletion claim on its own.

Read-Only User Setup

BurnLedger requires a dedicated read-only MarkLogic user, and it requires you to declare it with "read_only": true.

What the product checks, plainly: for MarkLogic it does not verify read-only access itself. Proving that a credential cannot write would require the Management REST API (GET /manage/v2/users/<user> → roles → privileges) on the management port — a route the query credential pointed at the REST app server normally can neither reach nor read. The alternative, probing with an actual write, would violate BurnLedger's guarantee that it never writes to a customer datastore. So the connector uses the documented fail-closed fallback: it honours an explicit read_only: true assertion on your authority as the operator, and refuses to construct the connector at all when the assertion is absent (CONNECTION_FAILED, "cannot verify the credential is read-only"). It never guesses. A corollary: because nothing is inspected or probed, MarkLogic can never raise WRITE_ACCESS_DETECTED — a write-capable credential paired with read_only: true is accepted silently, which is why the verification steps below matter.

What it does verify is connectivity and authentication, with a harmless read (GET /v1/config/resources) at connection time.

Because the assertion is taken at your word, create the user correctly.

Create the role and user

Run these against the Management REST API (port 8002) as an admin, with your admin password in ML_ADMIN_PASSWORD. --anyauth lets curl perform the digest handshake. REPLACE-WITH-YOUR-PASSWORD is a placeholder, not a suggestion: choose your own password when you create the user below, and substitute it wherever the placeholder appears in this guide.

# 1. A role that inherits rest-reader and nothing that can write.
curl --anyauth --user admin:"$ML_ADMIN_PASSWORD" \
  -X POST -H "Content-Type: application/json" \
  -d '{
        "role-name": "burnledger-reader",
        "description": "Read-only access for BurnLedger attestations",
        "role": ["rest-reader"]
      }' \
  https://ml.example.com:8002/manage/v2/roles

# 2. The user, holding only that role.
curl --anyauth --user admin:"$ML_ADMIN_PASSWORD" \
  -X POST -H "Content-Type: application/json" \
  -d '{
        "user-name": "burnledger_ro",
        "password": "REPLACE-WITH-YOUR-PASSWORD",
        "description": "BurnLedger read-only query user",
        "role": ["burnledger-reader"]
      }' \
  https://ml.example.com:8002/manage/v2/users

Do not grant rest-writer, rest-admin, admin, or any custom role carrying xdmp:eval, xdbc:insert, or document update privileges. Note that rest-reader alone is not enough to read documents protected by document-level permissions — grant the burnledger-reader role read permission on those documents rather than widening the role.

Verify the permissions

Confirm the credential can read and cannot write, before you assert read_only: true. Put the password you chose for burnledger_ro in BURNLEDGER_RO_PASSWORD. You are running these writes yourself, against a scratch URI — BurnLedger never issues one.

These use https://, the scheme BurnLedger will use. If the app server has no SSL verification record the curl fails here, which is the same reason registration would fail later — fix it now rather than after the system is registered.

# This should succeed (HTTP 200) and return a JSON search response whose
# "total" is NONZERO for a subject you know has documents. A 200 with
# "total":0 here means the user can reach the database but cannot read the
# documents in it (document-level permissions), and every attestation would
# then see an empty database — see "Schema drift" above.
curl --anyauth --user burnledger_ro:"$BURNLEDGER_RO_PASSWORD" \
  -X POST -H "Content-Type: application/json" -H "Accept: application/json" \
  -d '{"query":{"queries":[{"collection-query":{"uri":["alice"]}}]}}' \
  'https://ml.example.com:8000/v1/search?format=json&pageLength=0'

# This should FAIL with HTTP 403 (SEC-PRIV / permission denied).
curl --anyauth --user burnledger_ro:"$BURNLEDGER_RO_PASSWORD" \
  -X PUT -H "Content-Type: application/json" \
  -d '{"probe":true}' \
  'https://ml.example.com:8000/v1/documents?uri=/burnledger/write-probe.json'

# This should also FAIL with HTTP 403.
curl --anyauth --user burnledger_ro:"$BURNLEDGER_RO_PASSWORD" \
  -X DELETE \
  'https://ml.example.com:8000/v1/documents?uri=/burnledger/write-probe.json'

If the PUT returns 201 or 204, the credential can write. Fix the roles before continuing — asserting read_only: true for a write-capable credential defeats the guarantee the verification record rests on.

Hash Scope Options

The hashScope parameter controls what BurnLedger hashes for each matching document.

full (recommended for most cases)

Fetches every matching document (GET /v1/documents?uri=…) and hashes its complete content. Any change to any part of a document produces a different hash, giving the strongest proof that data has or has not been modified.

const system = await bl.registerSystem({
  // ...
  hashScope: "full",
});

Use when: you need to prove the exact content that existed, or prove that a document was deleted rather than merely emptied (nullifying PII fields is not the same as deletion under some interpretations of GDPR).

A document larger than the attestation's max_bytes limit aborts the run with BYTE_LIMIT_EXCEEDED, naming the offending URI.

existence

Hashes only the document URI of each match — not its content. The hash set changes when documents appear, disappear, or are renamed, but not when a document's content changes in place. This is the default when no scope is given.

const system = await bl.registerSystem({
  // ...
  hashScope: "existence",
});

Use when: you only need to prove that documents existed and were later removed, and you do not need to prove what was in them. It avoids fetching document bodies entirely, which is much faster for large result sets and keeps sensitive content out of the hashing path.

Comparison

Matches above the attestation's max_records limit abort the run with RECORD_LIMIT_EXCEEDED under either scope.

End-to-End Example

1. MarkLogic setup (run once)

curl --anyauth --user admin:"$ML_ADMIN_PASSWORD" \
  -X POST -H "Content-Type: application/json" \
  -d '{"role-name":"burnledger-reader","description":"BurnLedger read-only","role":["rest-reader"]}' \
  https://ml.example.com:8002/manage/v2/roles

curl --anyauth --user admin:"$ML_ADMIN_PASSWORD" \
  -X POST -H "Content-Type: application/json" \
  -d '{"user-name":"burnledger_ro","password":"REPLACE-WITH-YOUR-PASSWORD","role":["burnledger-reader"]}' \
  https://ml.example.com:8002/manage/v2/users

2. Application code

TypeScript

import { BurnLedger } from "burnledger";
import fs from "node:fs/promises";

const bl = new BurnLedger({
  apiKey: process.env.BURNLEDGER_API_KEY,
});

// Register the system (run once, store the system ID)
const system = await bl.registerSystem({
  name: "myapp-marklogic",
  connectorType: "marklogic",
  connectionConfig: {
    host: "ml.example.com",
    port: 8000,
    username: "burnledger_ro",
    password: process.env.MARKLOGIC_PASSWORD,
    database: "Documents",
    tls: true,
    read_only: true,
  },
  subjectQuery: '{"query":{"queries":[{"collection-query":{"uri":["{identifier}"]}}]}}',
  hashScope: "full",
});

// --- When a deletion request comes in ---

const subjectId = "alice";

// Step 1: Attest that the documents currently exist
const attestation = await bl.attest(subjectId, {
  systemIds: [system.id],
  proofMode: "merkle",
});

for (const s of attestation.systems) {
  console.log(`${s.systemName}: ${s.recordCount} documents found`);
}

// Step 2: Delete the documents in your application
// (BurnLedger never writes to MarkLogic -- your own writer credential does this)

// Step 3: Verify the deletion
// The certificate is ISSUED BY verify() once the data is gone -- there is
// no separate "create certificate" step.
const result = await bl.verify(attestation.id, subjectId, { timeout: 60 });

for (const s of result.systems) {
  console.log(`${s.systemName}: now ${s.recordCount} documents`);
}

if (result.certificate) {
  await bl.savePdf(result.certificate.id, `erasure-cert-${subjectId}.pdf`);
  console.log(`Certificate issued: ${result.certificate.id}`);
  console.log(`Transparency status: ${result.certificate.transparencyStatus}`);
} else {
  console.log(`Not certified: status ${result.status}`);
}

Python

import os
from burnledger import BurnLedger

bl = BurnLedger(api_key=os.environ["BURNLEDGER_API_KEY"])

# Register the system (run once, store the system ID)
system = bl.register_system(
    name="myapp-marklogic",
    connector_type="marklogic",
    connection_config={
        "host": "ml.example.com",
        "port": 8000,
        "username": "burnledger_ro",
        "password": os.environ["MARKLOGIC_PASSWORD"],
        "database": "Documents",
        "tls": True,
        "read_only": True,
    },
    subject_query='{"query":{"queries":[{"collection-query":{"uri":["{identifier}"]}}]}}',
    hash_scope="full",
)

# --- When a deletion request comes in ---

subject_id = "alice"

# Step 1: Attest that the documents currently exist
attestation = bl.attest(
    subject_id,
    system_ids=[system.id],
    proof_mode="merkle",
)

for s in attestation.systems:
    print(f"{s.system_name}: {s.record_count} documents found")

# Step 2: Delete the documents in your application
# (BurnLedger never writes to MarkLogic -- your own writer credential does this)

# Step 3: Verify the deletion
# The certificate is ISSUED BY verify() once the data is gone -- there is
# no separate "create certificate" step.
result = bl.verify(attestation.id, subject_id, timeout=60)

for s in result.systems:
    print(f"{s.system_name}: now {s.record_count} documents")

if result.certificate:
    bl.save_pdf(result.certificate.id, f"erasure-cert-{subject_id}.pdf")
    print(f"Certificate issued: {result.certificate.id}")
    print(f"Transparency status: {result.certificate.transparency_status.value}")
else:
    print(f"Not certified: status {result.status.value}")

Troubleshooting