❯ Testing
The test suite is not evidence of the product. It is the product.
A tool whose pitch is control cannot make a claim it has not verified. Guards decide what may move and the repository records what moved. The tests are why you can believe either.
The targets below are targets. They come from the design, not from a coverage run. Once CI generates the figures, this page prints the measured actuals beside them. Publishing a target as a result is the failure this page exists to disprove.
18 invariants, 18 named tests
Each safety rule is an entry in a catalogue that names the test holding it, and a test over the catalogue asserts that every named test exists and passes. The rule stops being prose: delete the enforcement and the build tells you which invariant lost its cover.
An entry may be marked pending while its code is still being written, and the check reads the marker in both directions. Leaving it on after the test lands fails as loudly as omitting it before. A marker cannot outlive the work it stands for.
| Invariant | What it says | Asserted by |
|---|---|---|
| PROTECTED_NEEDS_FORCESPEC §2 | an environment whose `mode` is `protected` refuses every upward write without --force | policy: a protected environment refuses an upward write without --force |
| REFUSAL_SURVIVES_DRY_RUNSPEC §2 / §19.1 | the refusal runs under --dry-run too; the consent does not | policy: a protected environment refuses under --dry-run as well |
| BACKUP_BEFORE_UPWARDSPEC §2 | no remote write proceeds without a restore point, or an explicit waiver | policy: an upward write needs a restore point or an explicit waiver |
| BACKUP_CAPABILITY_REFUSES§9.2 | a provider that cannot back up refuses the upward write rather than skipping it | policy: a provider that cannot back up refuses rather than skipping |
| UPWARD_HAS_A_SOURCEP2 / F-043 | an upward effect with no repo source is refused, unless it declares itself unreproducible | policy: an upward effect with no repo source is refused |
| WRITE_SCOPE_IS_PER_EFFECT§13 / §19.3 | direction lives on the effect, never on the command | policy: direction is read from the effect, not the command |
| TRACKED_NEVER_OVERWRITTENS-0006 / `docs/commands/fetch.md` | a downward fetch never overwrites a path listed in `tracked` — that is your source, not the remote's | policy: a fetch never overwrites a tracked path |
| NEVER_RAW_SQLCLAUDE.md rule 6 | URL replacement goes through wp search-replace, never raw SQL | policy: a URL rewrite must use search-replace, never raw SQL |
| PULL_REFUSES_DIRTY_TREES-0003 / ADR-002 / F-092 | a downward crossing refuses where it would overwrite uncommitted work in a path it writes — `--force` overrides | ops/crossing: a pull refuses a dirty tree in a path it writes |
| PULL_REFUSES_CONFLICTS-0003 / ADR-002 / F-092 | a downward crossing refuses any item that changed on both sides, by name — `--force` overrides | ops/crossing: a pull refuses an item that changed on both sides |
| PUSH_REFUSES_NON_FAST_FORWARDS-0004 / ADR-002 / F-092 / B-087 | an upward crossing refuses when an artifact the survey could read has moved since `base` — roles, settings, widgets and inventory; `--force` overrides, and an artifact that cannot be read is named rather than skipped | ops/crossing: a push refuses when the environment moved since base |
| SNAPSHOT_ABORTS_PULLS-0003 / `docs/commands/pull.md` / §19.5 | a failed pre-pull snapshot aborts the pull — the one step of seven that is not best-effort | ops/pull: a failed pre-pull snapshot aborts the pull |
| MAIL_GUARD_ALWAYSCLAUDE.md rule 5 / §19.4 | every pull that imports a database installs the mail guard, on every terminating path, even with scrubbing disabled | ops/pull: the mail guard is installed on every terminating path |
| SOURCE_IS_NEVER_WRITTENF-088 / CLAUDE.md rule 2 | an environment whose `mode` is `source` is refused every remote write, and no flag opens it | ops/guard: refuses any remote write to an environment whose mode is source |
| SOURCE_RELEASES_BEFORE_IT_CHANGESS-0002 / S-0007 / F-088 | `mode: source` on an environment that is still adopted is refused in both directions until `--release` takes the marker off | is refused rather than read, in the direction the mode exists for |
| CROSSING_DECLARES_SUB_WRITESCLAUDE.md rules 2 and 4 / ADR-005 | a composite crossing carries its sub-plans' effects, so the ladder sees every remote write the run will make | ops/crossing: a pull that adopts refuses a protected environment |
| REFUSAL_HAS_NO_EFFECTSSPEC §2 | a refused push runs no hook, takes no backup and transfers nothing | ops/push: a refused push has no side effects at all |
| VERIFY_BEFORE_NEXTSPEC §2 / §19.2 | each transfer is verified as it lands; a failed digest stops the fold | ops/push: a failed digest stops the fold before the next transfer |
The build fails if any named test disappears. These are not documentation of intent. They are the tests, indexed by the rule they defend.
Four tiers, four different jobs
domain/≥ 95%
None: plain values, and real temp directories where a planner stats.
Planning, precedence, parsing, canonical form.
tools/≥ 90%
None: exact argv assertions, including providers and builders.
The highest-value tests in the project: a wrong --app, a wrong host, a wrong path. This class of bug damages a site, and only argv assertions catch it.
ops/≥ 85%
A scripted session.
Step ordering, guard placement, dry-run behaviour.
cli/≥ 70%
Manifest-driven conformance.
Flag plumbing, exit codes, envelope shape.
Every seam is tested twice
One conformance suite runs against every implementation of a seam: cloudron, ssh and local for hosts; bricks, none and elementor for builders. A behaviour that only holds for the first implementation fails at once. The second implementation exists to catch that.
Two deliberate breaks confirmed it: drop a fix hint from a command, or let a host decide its own read/write intent instead of honouring the caller's, and the run fails.
Three structural tests
- The layering check
- Upward imports fail. Sibling imports stay legal, because a rule against those invents a
shared/directory within a month. - The invariant catalogue check
- Every entry names a test that exists and passes.
- The envelope-shape check
--jsonkeeps the documented shape, so anything parsing it can rely on the contract rather than on the current release.
No test needs a live host or a running Docker daemon. The subprocess boundary is the seam, and it is scripted.