Storage Read Contract

External contract

This page documents the bucket layout as a versioned contract for independent readers — clients that talk to the bucket directly instead of importing this repo's Go code (e.g. tfks/dropbear-android). It covers reads only: listing devices, fetching a manifest, fetching a blob. Uploads, tombstone emission, conflict resolution, and GC-journal semantics are internal write-side mechanics and out of scope here.

This repo (the Go writer) is the source of truth. Golden fixtures generated from the real encoders and committed at internal/contract/testdata/contract/ are the enforceable part of this contract; this page is the navigational description. If the two disagree, the fixtures and the code win.

Connection

A reader needs, per root:

Endpoint URL, region, bucket name — the S3-compatible backend (Cloudflare R2, AWS S3, MinIO, etc).
Prefix — the top-level key prefix under which the root lives (defaults to dropbear; empty prefix is valid and omits the segment entirely).
root_id — the sync root's identifier.
Read-only credentials scoped to the bucket — the recommended shape is a per-device, whole-bucket Object Read only token.

Key layout

Under <prefix>/roots/<root_id>/:

devices/
  registry.json
  <device_id>/
    manifest.json
    tombstones/<zero-padded-seq>.json
    tombstones/pending/<zero-padded-seq>.json
blobs/<ab>/<cd>/<sha256>

devices/registry.json — the device registry (below).
devices/<device_id>/manifest.json — one object per device, overwritten in place on every publish. There is no head object and no timestamped manifest history; the current object is the device's current state.
devices/<device_id>/tombstones/<seq>.json — a committed tombstone, <seq> zero-padded to 16 digits so lexicographic listing equals sequence order.
devices/<device_id>/tombstones/pending/<seq>.json — an in-flight tombstone. Readers SHALL ignore everything under tombstones/pending/ when building a device's applied-tombstone set; a listing under tombstones/ includes this nested subdirectory, so filter on "no / after the device-tombstones prefix."
blobs/<ab>/<cd>/<sha256> — content-addressed blob data, sharded by the first two hex byte-pairs of the SHA-256 to avoid hot-prefix behavior. <sha256> is 64 lowercase hex characters.

<device_id> and <root_id> both match the identifier pattern ^[a-z0-9]([a-z0-9-]{0,62}[a-z0-9])?$.

Device enumeration

A device is active — should be shown to the user, its manifest fetched — when its latest entry (highest registered_at) in devices/registry.json has retired_at: null. A device may appear multiple times in devices, once per register/retire cycle; only the latest entry per device_id matters. Retiring a device never deletes its bucket objects, so a reader MUST consult the registry rather than inferring liveness from what exists under devices/.

Schemas

All three record types share the same canonical encoding: UTF-8, no BOM, two-space indent, LF line endings, single trailing LF, object keys sorted lexicographically at every level, arrays of entries sorted lexicographically by path (or device_id for the registry), integers emitted without a decimal point, non-ASCII strings emitted as raw UTF-8 (no \uXXXX escapes). Two conformant encoders given the same logical value produce byte-identical output — this is what makes the golden fixtures meaningful as a conformance target.

Device registry (`devices/registry.json`)

Field	Type	Required	Nullable	Notes
`schema_version`	integer	yes	no	`1`.
`root_id`	string	yes	no	Identifier pattern.
`devices`	array	yes	no	Sorted by `device_id`. May be empty.

Each devices[] element:

Field	Type	Required	Nullable	Notes
`device_id`	string	yes	no	Identifier pattern.
`registered_at`	string	yes	no	RFC 3339 UTC, `Z` suffix, no fractional seconds.
`retired_at`	string	yes	yes	`null` while active.

No other top-level or per-device fields are permitted; readers should reject records containing unrecognized fields rather than ignoring them (see Stability guarantee).

Manifest (`devices/<device_id>/manifest.json`)

Field	Type	Required	Nullable	Notes
`schema_version`	integer	yes	no	`1`.
`root_id`	string	yes	no	Identifier pattern.
`device_id`	string	yes	no	Identifier pattern.
`created_at`	string	yes	no	Canonical timestamp.
`parent_manifest`	string	yes	yes	Legacy chain-linkage field; current writers always emit `null`.
`applied_tombstones`	object	yes	no	`{device_id: contiguous_seq}` — this device's contiguous-applied tombstone watermark per publishing device, including itself. May be empty.
`entries`	array	yes	no	Sorted by `path`. May be empty.

Each entries[] element has a type discriminator ("file" | "dir" | "symlink") with per-type required fields:

Field	file	dir	symlink
`path`	required	required	required
`type`	required	required	required
`mode`	optional	optional	optional
`size`	required	absent	absent
`mtime`	required (canonical timestamp)	absent	absent
`sha256`	required, 64 lowercase hex	absent	absent
`target`	absent	absent	required, non-empty

Fields marked "absent" MUST NOT appear in the JSON for that entry type. path is relative, never absolute, never containing a . or .. segment, and never under .dropbear/.

Tombstone (`devices/<device_id>/tombstones/<seq>.json`)

Field	Type	Required	Nullable	Notes
`schema_version`	integer	yes	no	`1`.
`root_id`	string	yes	no	Identifier pattern.
`device_id`	string	yes	no	Identifier pattern — the publishing device.
`seq`	integer	yes	no	Monotonic per publishing device; matches the zero-padded key suffix.
`created_at`	string	yes	no	Canonical timestamp.
`entries`	array	yes	no	Sorted by `path`. MUST NOT be empty.

Each entries[] element:

Field	Type	Required	Nullable	Notes
`path`	string	yes	no	Same constraints as manifest entry paths.
`deleted_at`	string	yes	no	Canonical timestamp.
`previous_type`	string	yes	no	`"file"` \| `"dir"` \| `"symlink"`.
`previous_sha256`	string	yes	yes	64 lowercase hex when `previous_type = "file"`; `null` otherwise.

Bytes at a tombstones/pending/<seq>.json key are byte-identical to the eventual bytes at tombstones/<seq>.json — but per Key layout, readers ignore the pending/ subtree entirely when building the applied set.

Canonical timestamp and identifier patterns

Timestamp (created_at, mtime, deleted_at): RFC 3339, UTC, Z suffix, fixed six-digit microsecond fraction — 2006-01-02T15:04:05.000000Z. Registry timestamps (registered_at, retired_at) are the same but with no fractional seconds (second precision only).
Identifier (root_id, device_id): ^[a-z0-9]([a-z0-9-]{0,62}[a-z0-9])?$ — lowercase, DNS-label-ish, 1–63 characters.

Blob addressing and verification

A blob's key is derived purely from its content hash: blobs/<sha256[0:2]>/<sha256[2:4]>/<sha256>. A reader fetching a blob referenced by a manifest entry's sha256 MUST recompute the SHA-256 of the downloaded bytes and verify it matches before treating the content as valid — the object store gives no other integrity guarantee.

Stability guarantee and version policy

schema_version is a single integer shared across the registry, manifest, and tombstone record types. Within a schema_version, the field set, types, and per-type rules above are frozen: writers will not add, remove, rename, or change the nullability of a field without incrementing schema_version, and this repo's schema-versioning rules (unknown-field rejection, schema_version-first parsing) apply identically to external readers.

A backwards-incompatible change — anything in the tables above changing shape — bumps schema_version and ships alongside regenerated golden fixtures (internal/contract/testdata/contract/, guarded by TestFixturesMatchCommitted). A reader that encounters a schema_version higher than the maximum it was built against SHOULD treat this as a distinct "too new, please update" condition — not as a malformed record — the same distinction this repo's own decoders make (SchemaVersionError vs. generic validation errors).

Source of truth

The Go types, encoders, and decoders in internal/manifest, internal/deviceregistry, and internal/objectstore are authoritative; this page and the fixtures in internal/contract/testdata/contract/ are generated from and checked against them. tfks/dropbear-android's DROPBEAR-API.md mirrors this page for convenience — if the two disagree, this page (and the fixtures) win.