marsh

Troubleshooting

Use marsh's status and job records to find the failing part of a command:

marsh status
marsh jobs --tree
marsh results
  • marsh status shows the daemon, the host control directory, job limits, open shells, and Kit VMs.
  • marsh jobs --tree shows jobs under their parents.
  • marsh results shows each recent job, its exit status, and whether its cleanup was verified.
SymptomSection
The first call to an agent takes a whileThe first command is slow
Error before a shell opensmarsh will not start
An environment file or MCP connection failsEnvironment files and MCP connections
A command exits 125 or 126A job was refused
cleanup uncertain in jobs --treeCleanup uncertain and quarantine
A VM will not go awayRemoving VMs
join left a split behindA split was kept
ephemeral home capacity exhaustedEphemeral home slots
A local Kit's image digest does not matchLocal Kit builds

The first command is slow

Starting with no VMs requires the shell VM to boot and each Kit VM to boot and load its image on first use. A foreground call prints [starting NAME worker VM…] when it boots a Kit VM; background calls suppress that notice. Later calls reuse warm VMs.

To boot them up front, use marsh --load all or marsh --load claude,codex.

marsh will not start

Missing or old sbx

MessageCause and fix
stock sbx not found on PATH, followed by marsh needs Docker Sandboxes: ...The marsh formula does not install sbx. Run brew install docker/tap/sbx, then sbx login. To use an sbx elsewhere, set MARSH_SBX to its path.
stock sbx not found at PATH (MARSH_SBX)MARSH_SBX names a file that does not exist or is not a regular executable. Fix the path, or unset MARSH_SBX to use sbx on PATH.
stock SBX v0.47.0 or newer ... requiredCheck sbx version, then run brew upgrade docker/tap/sbx.

Login and policy errors from sbx

If sbx reports that you are not signed in, run sbx login.

If sbx asks for a network policy, sbx policy init balanced sets the default one.

If sbx refuses a Kit's source, allow docker.io/mcavage (or your own registry) in its Kit sources policy.

Install and configuration errors

MessageCause and fix
install must provide libexec/marsh/shell-imagebin/ and libexec/marsh/ are not under the same prefix. Reinstall with the method you used: brew reinstall mcavage/tap/marsh, rerun the install script, or copy the tarball's bin/ and libexec/ under one prefix.
A message naming commands.json or agents.jsonThat file in the control directory is malformed. Fix it. marsh status shows the control directory.

Cannot mount the directory

The message is cannot mount DIR into the VM: it contains marsh's private host state at PATH. It can also end it is inside ....

The directory you started in contains, or is inside, one of marsh's own paths:

  • the install prefix
  • the control directory
  • a Kit source
  • the daemon's runtime directory

For example, /tmp contains the daemon's private runtime directory and cannot be mounted as a project. Start marsh from your project or a scratch directory outside the protected paths.

Environment files and MCP connections

Configuration errors name the YAML file. Edit it on the Mac, then finish jobs, exit marsh shells, run marsh stop, and start marsh again.

ErrorFix
Unsupported field or schema versionUse schemaVersion: "1" and the supported fields.
File ownership, permissions, or symlinkUse a regular YAML file owned by your Mac account; remove group/other write permission. sbxenvs must be a real directory too.
Duplicate MCP name or overlapping directoryRemove the duplicate from the shared or per-command file. Lists add together; they do not override each other.
Aliases have different environmentsGive names for the same Kit identical combined configuration.
Server already has a different host-global definitionChoose another server name, or update the existing registration with sbx mcp on the Mac. Other sandboxes may use it.
Missing MCP authorizationRun sbx mcp auth NAME on the Mac and retry the job.
MCP registration or loading failedCheck the reported sbx error and retry after fixing the connection. Declared servers must load before the VM can be used.

Restart after changing YAML. Authentication changes alone can be retried without editing the file. A worker reset uses the daemon's existing YAML snapshot. Published pipeline defaults have a different failure policy: see MCP defaults.

A job was refused

A refused job starts nothing and prints a reason. A registered-name link excluded by MARSH_SPAWN exits 126; other refusals, including those through marsh run, exit 125.

MessageCause
capacity: 8 jobs8 jobs are already running under this daemon
depth limit 4the tree is already 4 jobs deep
fan-out limit 4this job already has 4 running children
... refused: same-Kit chain limit 2the call would make the same Kit three times in a row
total limit 64the tree has used its 64 launches, including refused child calls
tree already uses ... Kit VMs (...) (MARSH_TREE_KIT_VMS)live jobs in the tree already use the configured number of distinct Kits; let one finish before adding another
deadline: parent job ... has ... ms of wall time leftless than one second remains before the parent's deadline; a child cannot outlive it
spawn refused: NAME not in this job's spawn setMARSH_SPAWN or --spawn excludes it (126 from a registered-name link, 125 from marsh run)
interactive child jobs are not supported yeta child job was given a terminal; redirect its stdin and stdout

Refused child calls count toward the tree's 64-launch budget. Avoid retry loops. A depth or same-Kit-chain refusal needs a different call structure; waiting will not clear it. See job-tree limits.

Cleanup uncertain and quarantine

marsh checks that every container it starts is deleted. Sometimes it cannot confirm that, for example because the connection to a Kit VM was lost mid-job. Then marsh:

  • does not re-run the job
  • marks the job cleanup_uncertain (cleanup uncertain in jobs --tree)
  • quarantines that Kit VM, so no new jobs go to it

Run marsh workers reset KIT, using the Kit name in the message, or marsh workers reset all to retire all idle or quarantined Kit VMs.

This removes the VM, and the next call creates a new one. Files the job wrote to your project stay as they are. Review them with git status and git diff, or your own tools.

Removing VMs

CommandEffect
marsh workers reset KITRemove one idle or quarantined Kit VM
marsh workers reset allRemove all idle or quarantined Kit VMs
marsh resetRemove the scope's VMs; keep the daemon running
marsh stopRemove the scope's VMs and stop its daemon

reset and stop refuse while shells or jobs are open. Each prints the VMs it removed.

If a removal could not be verified, the command names what is left and exits nonzero.

Your project, guest home, and job results are kept. Packages installed in a VM are not.

Which VMs marsh touches

marsh only touches VMs it created. They are named marsh-k-… (Kits) and marsh-s-… (shells), and the daemon keeps a list of them. Any other VM, including ones you made with sbx, is left alone.

Remove marsh's VMs with marsh workers reset, reset, or stop, not with sbx. If you remove one with sbx, the daemon treats its state as unknown, not as cleaned up.

A removal keeps failing

Fix sbx first. sbx ls should work. Then run the same command again.

Do not delete the control directory or kill the daemon to get past it. The daemon's list of its VMs is in that directory, and without it marsh cannot prove which VMs are its own.

A split was kept

join --keep retains results deliberately. A nonzero join status, interrupted join, expired lease, or unconfirmed branch cleanup also keeps them. Bare join returns the branch status; with a downstream command, retention follows that command's status. pipefail does not change retention.

List results with marsh splits, inspect a split's state with marsh splits ID, and remove it with marsh splits rm ID. Removal deletes the forks and patches, so apply anything you want first. Join never applies a patch itself.

If a branch's VM was lost, run marsh workers reset all before rm.

See split and join for details.

Ephemeral home slots

marsh --ephemeral-home uses one of 16 private slots. It skips a slot that still holds data, which happens when a shell's cleanup was not verified. When no slot is free, it fails with ephemeral home capacity exhausted.

When nothing is using the slot, run marsh recover-home SLOT --discard, substituting its slot number. This deletes the slot's data.

Stale daemon after an upgrade

  • Upgrading sbx: marsh restarts an idle daemon on its own. Rerun marsh mcp install ... for any MCP registrations.
  • Upgrading marsh: a running daemon keeps the old binaries. Run marsh stop when idle.

Reporting a bug

Include:

  • marsh --version and sbx version
  • the exact command
  • what you expected
  • what happened (stdout, stderr, exit status)

For shell behavior, run the same script with bash -c and include both outputs. marsh aims to match Bash.

Do not paste tokens or private files. Report security problems as described in SECURITY.md.

Local Kit build does not match

marsh and Docker Sandboxes must build the same image for a local-source Kit. marsh retries once if their digests differ. If it reports "still mismatched after stock SBX rebuilt the Kit", run docker buildx ls and select a local builder (the * entry, for example desktop-linux), then retry. Both builds need to use the same BuildKit cache. See the registry design for the build and image checks.

Next: Configuration, Commands and Kits.

Edit this page on GitHub