marsh

Publishing native Kits

scripts/prepare-kit-inputs.py and scripts/publish-kits.py prepare and publish native Kit v3 sources. They do not create sandboxes or configure credentials. Publication requires an already authorized Buildx builder and registry.

Default and opt-in registries

packaging/commands.json selects the four public-source Kits. Other Kits (bring your own agent Kit) use their own registry file.

python3 -m venv "$HOME/.cache/marsh-publish-venv"
. "$HOME/.cache/marsh-publish-venv/bin/activate"
python3 -m pip install --require-hashes --no-deps --only-binary=:all: -r scripts/requirements-publish.txt
make kits
make kit-publish KIT_REPOSITORY_PREFIX=registry.example/team/marsh

Registries such as Docker Hub accept a namespace and a single repository name. To give images distinct names within one namespace, use --repository-prefix REGISTRY/NAMESPACE --repository-name-prefix marsh- with scripts/publish-kits.py. For example, release CI publishes the fixture as docker.io/mcavage/marsh-fixture (.github/workflows/release.yml). Create private repositories before publication when the payload must stay private.

Preparation/publishing output directories must be owner-private (0700) and outside every selected Kit source. New directories are created at 0700. For an existing target/kit-release, review its contents and set its mode to 0700 first. Files are written at 0600, and an existing output directory with unsafe permissions is rejected. make kits and --validate-only run full native cache-only builds without pushing. Those builds can execute Dockerfile instructions, pull images, and write builder cache. Every selected build must succeed before any push starts.

One canonical input set

Every default Kit and the acceptance fixture receives:

  • packaging/dhi-notices/collected as dhi-notices/;
  • packaging/image-base as image-base/ (the package assembler that builds each image on its DHI runtime base, and the smoke test).

The npm-based Kits receive scripts/npm-notices.mjs as collect-notices.mjs. Provider legal texts are indexed in the canonical DHI bundle. There are no independent Kit-local assembler, collector, or Claude/Codex notice copies to keep synchronized.

The publisher expands these sources once by their bundled source identity. It reserves their entire destination namespaces, including case/Unicode variants: source files and extra mappings cannot replace them or add new files underneath. Canonical additions require a new preparation, just like modifications/deletions. Custom Kits use their explicitly declared inputs. A fixture --shell-image override requires both the exact original exported image file and an explicit host-private full observed build receipt, not merely adjacent helper JSON:

python3 scripts/prepare-kit-inputs.py \
  --source-tree /canonical/current/candidate \
  --commands /canonical/current/candidate/tests/acceptance/fixture/commands.json \
  --shell-image /private/observed-export/shell-image \
  --shell-build-receipt /private/host-control/build.json \
  --output /private/kit-preparation/fixture-inputs.json

--source-tree defaults to this checkout and selects the candidate/Kit worktree. Image, receipt and explicit source-tree paths must be canonical absolute paths, not symlink aliases. The receipt must be outside source and both host/Linux exports, in its private host directory. Strong prepared documents and publication registries must also stay outside that observed source and its exports; neither output may overwrite a consumed source/proof input (including an inode alias). The shared image_observations.verified_shell_image revalidates the observed build receipt, exact source, host/Linux artifacts, licenses, producer image proof and original export path. Copied/relabeled image files, stale artifacts with unchanged receipt bytes, and helper-only JSON are rejected. Candidate/compiler/tool observations come from the full host build receipt; the standalone shell image itself carries no Rust candidate payload. This is trusted-host observation, not a signature or proof against arbitrary host code forging receipts.

A distributable Kit FROM requires the observed explicit-registry publication. A local-template proof or mutable alias is not a remotely resolvable base. The final registry/config/runtime check remains a separate actual-host gate.

Prepared document and publication receipt

Preparation writes one atomic JSON document (marsh.prepared-kit-inputs/v2), not a map plus an independently replaceable .sources.json. Old v1 sidecars are ignored and never evidence for v2; they are not silently deleted. It contains:

  • inputs: selected source → descriptor arguments and explicit file mappings;
  • receipt.bindings: SHA256, size and mode of preparer, publisher, shared build_inputs.py, shared command_registry_rules.json, command registry, and explicit configuration/proof files;
  • receipt.state.canonical: canonical origin paths and actual file inventories (including directory membership), stored once rather than repeated for each Kit;
  • receipt.state.sources: admitted Git source-tree hash, independently checked staged-tree hash, descriptor name/hash, effective arguments, count and byte size;
  • receipt.shell_images: narrow strong verification requests linking a selected Kit's shellImage to exact candidate source, image-file and host build-receipt paths. Publisher re-runs the shared semantic verifier, not only a JSON hash.

Tree hashes are SHA256 of sorted, indented JSON plus a final newline. Each file record contains sha256 (with prefix), size, and mode. Staged file records use normalized modes. The publisher measures current source and copied bytes itself; it does not accept the receipt's hashes as assertions of what it staged.

Bundled Kits require this prepared document even for --validate-only. A stale script, canonical input, descriptor, Kit file, added/deleted admitted file or configuration fails before any Docker invocation. Rechecks after all native cache-only builds prevent a changed source from starting a push. Private context changes are also refused. Strong shell proof verification is repeated before Docker, after cache-only builds, before each push, and after publication, checking candidate and artifact bytes as well as the receipt. If a check fails after a push, the diagnostic says publication may have occurred. Inspect the destination before retrying.

Successful publication writes a durable content-addressed receipt named <registry>.publication-<sha256>.json before atomically replacing the ordinary command → repository@sha256:digest registry. The receipt binds the exact prepared file bytes, source/script bindings, effective arguments and staged hashes to each reported immutable digest and the exact registry hash. Existing receipts are not overwritten. Consumers can verify the receipt filename and registry_sha256. A failure before registry commit preserves the previous registry and receipts, but a crash can leave an unreferenced new receipt: the receipt and registry are separate local commits. Remote pushes can also partially succeed, leaving images and moving the mutable tag (:release, or --tag). Consumers pin by digest, so a moved tag does not change their selected image.

The receipt binds source and staged bytes to reported digests. It does not sign them, establish source trust or legal completeness, or qualify the image at runtime. Do not edit sources during preparation or publication: before/after checks can detect changes, but cannot take an atomic snapshot against a hostile same-user editor.

Explicit authoring inputs and native argument rules

--commands FILE maps command names to directories beneath --source-root. Sources must be in a Git worktree and contain one included .yaml workload descriptor. Command names/count/document bytes use the same declarative rules as Rust, crates/marsh-contracts/src/command_registry_rules.json (currently 251 names, 128 ASCII bytes per name, 1 MiB registry). Reserved names and dots are refused; numeric-leading names are allowed. That rules file is a bound preparation input. KIT_BUILD_INPUTS supplies extra args/files to the preparer:

{
  "kits/my-kit": {
    "args": {"version": "1.2.3"},
    "files": {"artifacts/tool": "/absolute/owner-controlled/tool"}
  }
}

Custom authoring Kits may also supply this legacy explicit mapping directly to --build-inputs. It receives the same staging/authority/native checks and a publication receipt, but no claim of a separate prepared-source receipt. To seal custom inputs, the shared Python API is seal_prepared(root, commands_path, inputs, bindings=(), *, shell_images=()); bind all helper/proof files. For fixture or custom shellImage overrides, supply:

shell_images=[{"source": "kits/my-kit", "image_file": str(original_image_file),
               "source_tree": str(root), "build_receipt": str(host_build_receipt)}]

source is the selected Kit path; the argument is fixed to shellImage. The publisher automatically binds the returned proof paths and shared verifier modules. The fixture requires sealed v2 inputs; raw maps and requests without the strong proof are refused. The public wrapper remains verified_shell_image(path, *, source_tree=ROOT, build_receipt=None) → (reference, bindings); it fails closed when the receipt is absent. prepare-kit-inputs.map_tree(source, destination) → (files, inventory) remains compatible. Explicit file origins are absolute; destinations cannot traverse, overwrite source files, or add Docker ignore controls.

Only descriptor argument names are accepted, not raw Docker buildArg names. Missing values use declared defaults, otherwise they are required. Strings, argument counts, patterns and YAML nesting are bounded; aliases and duplicate keys are refused. All selected private contexts undergo full native --output type=cacheonly builds before any push. Go's native regex parser is the pattern authority, including \z, POSIX classes and rejection of backreferences. It checks the effective value: a default that does not match its pattern can be replaced by a valid supplied value. Each cache-only build/push has a finite 30-minute deadline. Any failed cache-only build prevents every push. The pinned frontend does not support --call=validate; the publisher uses no frontend subrequest flag. Native compatibility must be tested against actual Buildx/frontend, never inferred from a recording executable.

Canonical aliases are compared by actual device/inode, including case aliases on case-insensitive macOS volumes. One source gets one build/input configuration; conflicting alias configurations are refused. Distinct sources must have distinct lowercase directory names because those names become repository suffixes.

Filesystem boundaries

  • Command registries are at most the shared 1 MiB bound; other JSON at most 4 MiB; each file at most 128 MiB; all staged contexts together at most 512 MiB. A Kit admits at most 10,000 files. Canonical/shared tree walks also bound directory entries and depth (64), not just file bytes.
  • Inputs must be caller-owned regular files with one hard link, no group/world writes. Reads are nonblocking and no-follow through every parent component. Canonical tree directories must be caller-owned without other writers. Trusted system ancestors are permitted, including root-owned sticky temporary roots.
  • Kit source contexts include tracked and nonignored untracked authoring files, not only committed HEAD. Tracked deletions are not resurrected. Repository .gitignore rules apply; personal excludes and .git/info/exclude do not change inputs. Every nonignored untracked input path (including canonical origins) is printed once, JSON-escaped, to stderr before the first remote builder call. Explicit origins are also listed; contents and argument values are not printed. Review the list and use repository ignore rules for Kit scratch files; inclusion does not mean a file is secret-free. Canonical trees are explicit whole-tree inputs, not Git snapshots: keep scratch/secret files out of them even if ignored. New admitted files after preparation require re-preparation.
  • Source symlinks and admitted special files are refused. Git can omit untracked special files; omitted/ignored paths are not part of the admitted context.
  • Temporary context parents are 0700. Context directories are 0755; regular files are 0644 plus original execute bits so image users can read notices even from a umask-077 checkout. Input modes/ownership are not changed.
  • Output parents are no-follow, owner-private, and outside selected source identities. The publisher retains and verifies the directory fd across Docker calls, then uses it for atomic replacement and fsync; pathname substitution cannot redirect a write.

These controls do not make an untrusted Dockerfile safe. Review build sources and nonsecret inputs before granting the builder authority.

Credential-free caller checks

python3 -W error tests/kits/test_publisher_cli.py -v

The suite calls the real CLIs on copied default Kit and fixture sources, with a recording Docker executable. It independently hashes canonical/staged bytes and checks stale additions/deletions/modifications, single-link/owner/no-follow/size refusals, namespace reservation, long paths, output boundaries, source changes during builds, publication receipt binding and preservation. It also checks cache-only-before-push call order, not native regex semantics; the recorder explicitly refuses frontend subrequest flags. The APFS alias case is skipped on case-sensitive test volumes. The recording executable cannot establish native frontend compatibility, registry readability, nonroot image behavior, or stock-SBX execution. Those require the actual host gates below and the product acceptance run.

Opt-in actual frontend regression

On the authorized host with a local Docker Engine, run the maintained live test:

python3 tests/kits/native_frontend_uat.py --run-native \
  --docker /absolute/path/to/docker \
  --buildx-plugin /absolute/path/to/docker-buildx \
  --docker-host unix:///absolute/path/to/docker.sock \
  --evidence /canonical/private/parent/new-native-evidence

This is opt-in and is not run by the recorder suite. It selects an explicit local Unix Docker endpoint and a fresh credential-free Docker configuration with the supplied Buildx plugin, records tool versions/SHA256 and source/context fences, and runs the exact five pattern cases: end-anchor \z, POSIX digits, invalid backreference, invalid default with valid override, and nested quantifier mismatch with 20,001 bytes. Frontend and conformant DHI base are pinned by digest. The fixed Dockerfile has USER agent, inert ENTRYPOINT, and no RUN instructions. The same cases also call the real publisher's cache-only route; its retained 4,096-byte input bound refuses the oversized nested case before Docker. The 20,001-byte direct-native case separately proves native finite mismatch behavior.

Each live command has a 120-second default deadline, with an override up to 600. Negative cases must return native pattern diagnostics; setup failures or an unsupported frontend fail the test. The harness can resolve public images and write builder cache, but has no push, image-load, workload, agent, or SBX step. Qualification still requires a host run against the exact publisher candidate.

Edit this page on GitHub