Forelead handbook

Appearance

Upgrade and rollback ​

Upgrading means swapping the running images for a new build and running the database migrations. The one rule: never upgrade while a task is claimed, running or in gates. The upgrade script checks this for you and refuses.

Upgrade the local single-host stack ​

Run deploy/upgrade.sh (use --dry-run first to see the steps). It does the whole upgrade in one go:

  1. Check the intake configuration against the new build (changes nothing).
  2. Create the per-service database roles and their password files.
  3. Renew the operator certificate if it is past two thirds of its life.
  4. Stop the agent runner and the preflight runner.
  5. Pull, back up, migrate and recreate the stack (ht upgrade).
  6. Approve the existing runner evidence key for each runner that used it (once).
  7. Rebuild the host agent runner.
  8. Recreate the preflight sandbox network and proxy, and wait until the proxy answers.
  9. Start the preflight runner, run the end-to-end check (ht agent-runner check, then ht verify), and only then start the agent runner.

If anything fails, both runners are stopped again, FAIL is printed and the exit code is non-zero. Order matters: a runner must never run while the sandbox network is recreated, or a job claimed then cannot reach its proxy.

The script fetches the target commit from the checkout's origin. If that remote holds a stale token, export a fresh GH_TOKEN first; it is passed through an environment header and never written to .git/config.

Roll back ​

Schema changes are forward-only, so rollback depends on whether a migration ran.

  • No migration ran: set FORELEAD_PREVIOUS_IMAGE and run sh deploy/activate.sh. If startup or the loop check fails it restores the previous image by itself. It never deletes volumes.
  • A migration ran: restore the database backup together with the previous image. Images that check the schema version refuse to start against a database a newer build migrated (FORELEAD_ALLOW_NEWER_SCHEMA=1 overrides, at your own risk).
  • Older images do not have the health command or the loopback listener. Rollback also applies deploy/compose.rollback.yaml, which restores the old TCP probes and turns off the runner probe. Use the base compose.yaml again once you are back on a new image.

Before you start ​

  • Back up the Postgres volume and the external secrets.
  • Keep the previous image available.
  • Set FORELEAD_COMPOSE_FILES to the files the running stack uses; activate.sh refuses to start without it.
  • Set FORELEAD_LOOP_TEST to the operator loop driver. An upgrade does not report success until it passes (Operator loop).

Technical notes ​

  • Compose health checks verify loopback readiness only. They do not check mTLS or provider calls: verify those with real client certificates before accepting a deployment.
  • The three AWS tenant hosts start from a deploy bundle with deploy/cloud/start.sh app|runner|mirror (see AWS tenant).
  • The upgrade also asks whether it can read the old shared runner key. LEGACY_RUNNER_KEY_FILE can point at a readable copy.

Sources ​