Amazon S3 Integration Guide

This guide covers how to connect an S3 bucket to BurnLedger, configure key prefix patterns, set up a read-only IAM policy, and understand how hash scope works for object storage.

Connection Configuration

BurnLedger connects to S3 using an IAM access key with read-only permissions.

Configuration fields:

These are the exact JSON keys the connector reads. They are snake_case in both SDKs — do not write accessKeyId, secretAccessKey, or readOnly; the connection object is sent to the API verbatim and unknown keys are ignored, so a camelCase key reads as a missing credential.

TypeScript

const system = await bl.registerSystem({
  name: "user-uploads",
  connectorType: "s3",
  connectionConfig: {
    bucket: "myapp-user-uploads",
    region: "eu-west-1",
    access_key_id: process.env.DP_S3_ACCESS_KEY_ID,
    secret_access_key: process.env.DP_S3_SECRET_ACCESS_KEY,
  },
  subjectQuery: "users/{identifier}/",
  hashScope: "existence",
});

Python

system = bl.register_system(
    name="user-uploads",
    connector_type="s3",
    connection_config={
        "bucket": "myapp-user-uploads",
        "region": "eu-west-1",
        "access_key_id": os.environ["DP_S3_ACCESS_KEY_ID"],
        "secret_access_key": os.environ["DP_S3_SECRET_ACCESS_KEY"],
    },
    subject_query="users/{identifier}/",
    hash_scope="existence",
)

Key Prefix Pattern

For S3 systems, the query template is a key prefix pattern rather than a SQL query. It uses {identifier} as the placeholder for the data subject identifier.

Pattern syntax:

• {identifier} is replaced with the subject value at attestation time. The template must contain {identifier}; a template without it is rejected with INVALID_QUERY_TEMPLATE rather than silently matching nothing.
• The expanded string is used verbatim as the S3 key prefix. There are no wildcards.
• In particular * is not a wildcard: it is sent to S3 as a literal character. users/{identifier}/* expands to the prefix users/user-4821/*, which matches no real key. Write users/{identifier}/ instead — prefix matching already covers every key underneath it.

Examples:

# All objects under a user-specific prefix
users/{identifier}/

# Objects in a flat namespace with the subject as part of the key
uploads/{identifier}-

# Subject as a directory at the top level
{identifier}/

# A single object per subject (an exact key is just a fully specified prefix)
data/exports/{identifier}/export.csv

How it works during attestation:

1. BurnLedger substitutes the subject value into the pattern to form the key prefix.
2. It issues paginated ListObjectsV2 calls with that prefix (ListObjectVersions on versioned buckets).
3. Object metadata (size, ETag, last modified) comes from the list response itself — BurnLedger does not issue a per-object HeadObject.
4. It computes the attestation hash based on the hash scope.

IAM Policy

Create a dedicated IAM user (or role) for BurnLedger with strictly read-only permissions. BurnLedger validates that the credentials cannot write to the bucket and rejects connections with write access — without ever writing. It asks IAM whether the principal's policies would allow s3:PutObject, s3:DeleteObject, or s3:PutObjectTagging — on both arn:aws:s3:::<bucket> and arn:aws:s3:::<bucket>/* — via iam:SimulatePrincipalPolicy (a policy evaluation, not an action).

This means the credential needs the iam:SimulatePrincipalPolicy permission for the check to run. If you cannot grant it (or you use a non-AWS S3-compatible endpoint, which has no IAM), set "read_only": true in the connector config to assert the credential is read-only; BurnLedger only ever reads regardless. Without either, the connection is refused (fails closed) rather than risk connecting with an unverified credential.

Minimal policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BurnLedgerReadOnly",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketVersions",
        "s3:GetBucketVersioning"
      ],
      "Resource": [
        "arn:aws:s3:::myapp-user-uploads",
        "arn:aws:s3:::myapp-user-uploads/*"
      ]
    }
  ]
}

Actions explained:

To let BurnLedger verify read-only access automatically, also allow iam:SimulatePrincipalPolicy (resource *). Otherwise, omit it and set "read_only": true in the connector config.

Create the IAM user

# Create the user
aws iam create-user --user-name burnledger-ro

# Attach the policy (save the policy JSON above as burnledger-policy.json)
aws iam put-user-policy \
  --user-name burnledger-ro \
  --policy-name BurnLedgerReadOnly \
  --policy-document file://burnledger-policy.json

# Create access keys
aws iam create-access-key --user-name burnledger-ro

Store the AccessKeyId and SecretAccessKey from the output in your secrets manager.

Verify the permissions

# These should succeed
aws s3 ls s3://myapp-user-uploads/users/ --profile burnledger-ro
aws s3api head-object --bucket myapp-user-uploads --key users/test/file.txt --profile burnledger-ro

# These should fail with "Access Denied"
aws s3 cp /dev/null s3://myapp-user-uploads/test-write --profile burnledger-ro
aws s3 rm s3://myapp-user-uploads/users/test/file.txt --profile burnledger-ro

Hash Scope

existence (default for S3)

Hashes the list of matching object keys and their metadata (size, ETag, last modified timestamp). Does not download object content. This is the default for S3 because downloading large files for hashing is expensive and usually unnecessary -- you typically need to prove that objects existed and were later removed, not what was inside them.

const system = await bl.registerSystem({
  // ...
  hashScope: "existence",
});

What is hashed (five canonical fields, sorted and joined):

• bucket — the bucket name
• content_length — object size in bytes
• etag — object ETag, surrounding quotes stripped
• key — object key (full path)
• last_modified — last modified timestamp, UTC, microsecond precision

Detects:

• Object creation (new key appears)
• Object deletion (key disappears)
• Object replacement (ETag or size changes)

Does not detect:

• Content changes that preserve size and ETag (extremely unlikely with S3's ETag algorithm, but theoretically possible with custom implementations)

full

Downloads every matching object and hashes its content. Use this only when you need to prove the exact bytes that were stored, not just that objects existed.

const system = await bl.registerSystem({
  // ...
  hashScope: "full",
});

Warning: Full scope on S3 downloads every matching object during every attestation and verification. For buckets with large files, this can be slow and expensive. Use it only when content-level proof is a compliance requirement.

Versioned Buckets

BurnLedger detects bucket versioning at connect time (GetBucketVersioning) and changes strategy when it is enabled:

• Listing uses ListObjectVersions instead of ListObjectsV2.
• If any non-delete-marker object version is present under the prefix, both attestation and verification fail with S3_VERSIONING_CONFLICT rather than reporting a count. A "deleted" key whose prior versions still exist is not deleted, and BurnLedger will not certify it as such.
• Once only delete markers remain, they are counted and hashed from three fields — bucket, key, last_modified.
• hashScope is ignored on versioned buckets. Delete markers have no content and no ETag, so full scope silently behaves like the delete-marker metadata hash above.

To attest a versioned bucket, expire the noncurrent versions with a lifecycle rule before verifying, and confirm they are gone.

Suspending versioning is not sufficient and is no longer accepted. Suspension stops new versions being created; every version already written stays recoverable. A bucket with suspended versioning and surviving noncurrent versions still holds the subject's data, so BurnLedger refuses rather than certifying zero over it.

S3-Compatible Services

BurnLedger works with any S3-compatible object storage by specifying a custom endpoint.

MinIO

const system = await bl.registerSystem({
  name: "minio-uploads",
  connectorType: "s3",
  connectionConfig: {
    bucket: "user-data",
    region: "us-east-1",
    access_key_id: process.env.MINIO_ACCESS_KEY,
    secret_access_key: process.env.MINIO_SECRET_KEY,
    endpoint: "https://minio.example.com",
    // Required: a custom endpoint has no IAM, so BurnLedger cannot verify
    // read-only access and refuses to construct without this assertion.
    read_only: true,
  },
  subjectQuery: "{identifier}/",
  hashScope: "existence",
});

DigitalOcean Spaces

const system = await bl.registerSystem({
  name: "do-spaces-uploads",
  connectorType: "s3",
  connectionConfig: {
    bucket: "myapp-uploads",
    region: "nyc3",
    access_key_id: process.env.DO_SPACES_KEY,
    secret_access_key: process.env.DO_SPACES_SECRET,
    endpoint: "https://nyc3.digitaloceanspaces.com",
    read_only: true,
  },
  subjectQuery: "users/{identifier}/",
  hashScope: "existence",
});

End-to-End Example

TypeScript

import { BurnLedger } from "burnledger";
import fs from "node:fs/promises";

const bl = new BurnLedger({
  apiKey: process.env.BURNLEDGER_API_KEY,
});

// Register the S3 bucket
const system = await bl.registerSystem({
  name: "user-uploads",
  connectorType: "s3",
  connectionConfig: {
    bucket: "myapp-user-uploads",
    region: "eu-west-1",
    access_key_id: process.env.DP_S3_ACCESS_KEY_ID,
    secret_access_key: process.env.DP_S3_SECRET_ACCESS_KEY,
  },
  subjectQuery: "users/{identifier}/",
  hashScope: "existence",
});

// Attest that files exist for this user
const attestation = await bl.attest("user-4821", {
  systemIds: [system.id],
  proofMode: "merkle",
});

for (const s of attestation.systems) {
  console.log(`Found ${s.recordCount} objects for user-4821`);
}

// Delete the user's files in your application
// await s3Client.send(new DeleteObjectsCommand({ ... }));

// Verify deletion
// The certificate is ISSUED BY verify() once the data is gone -- there is
// no separate "create certificate" step.
const result = await bl.verify(attestation.id, "user-4821", { timeout: 60 });

// The "before" count comes from the attestation; verify() reports what is
// there NOW. There is no combined before/after object.
for (const s of attestation.systems) {
  console.log(`Objects before: ${s.recordCount}`);
}
for (const s of result.systems) {
  console.log(`Objects after: ${s.recordCount}`);
}

if (result.certificate) {
  await bl.savePdf(result.certificate.id, `s3-deletion-cert-${result.certificate.id}.pdf`);
  console.log(`Certificate issued: ${result.certificate.id}`);
} else {
  console.log(`Not certified: status ${result.status}`);
}

Python

import os
from burnledger import BurnLedger

bl = BurnLedger(api_key=os.environ["BURNLEDGER_API_KEY"])

# Register the S3 bucket
system = bl.register_system(
    name="user-uploads",
    connector_type="s3",
    connection_config={
        "bucket": "myapp-user-uploads",
        "region": "eu-west-1",
        "access_key_id": os.environ["DP_S3_ACCESS_KEY_ID"],
        "secret_access_key": os.environ["DP_S3_SECRET_ACCESS_KEY"],
    },
    subject_query="users/{identifier}/",
    hash_scope="existence",
)

# Attest that files exist for this user
attestation = bl.attest(
    "user-4821",
    system_ids=[system.id],
    proof_mode="merkle",
)

for s in attestation.systems:
    print(f"Found {s.record_count} objects for user-4821")

# Delete the user's files in your application
# s3_client.delete_objects(...)

# Verify deletion
# The certificate is ISSUED BY verify() once the data is gone -- there is
# no separate "create certificate" step.
result = bl.verify(attestation.id, "user-4821", timeout=60)

# The "before" count comes from the attestation; verify() reports what is
# there NOW. There is no combined before/after object.
for s in attestation.systems:
    print(f"Objects before: {s.record_count}")
for s in result.systems:
    print(f"Objects after: {s.record_count}")

if result.certificate:
    bl.save_pdf(result.certificate.id, f"s3-deletion-cert-{result.certificate.id}.pdf")
    print(f"Certificate issued: {result.certificate.id}")
else:
    print(f"Not certified: status {result.status.value}")

Troubleshooting