Forelead handbook

Appearance

Restarts and leases ​

Before you restart anything that runs tasks, check that no task is active. A restart in the middle of a task makes the runner lose its lease, the job is re-queued, and the task spends one of its attempts.

The check ​

  • deploy/active-tasks.sh defines no_active_tasks. It succeeds only when no task is claimed, running or in gates, using each task's latest transition. It reads the database inside the Postgres container (forelead-postgres-1 by default).
  • It fails closed: if the database cannot be asked, it says so and fails. Do not read that as "idle".
  • upgrade.sh and enable-agent-rules.sh source it and refuse to continue when it fails.

What a lease is ​

A runner holds a job through a lease of 30 seconds, renewed while it works. If a runner stops renewing, the lease expires and the job can be claimed again with a new token. The old token can no longer renew, release or send stream messages. A runner that renews but reports nothing for 15 minutes is released and the job is re-queued.

Restarting the runner for a certificate ​

The runner certificate timer (deploy/renew-runner-cert.sh) restarts the runner only when it is idle. It asks runner idle; if the runner is busy it retries on the next timer tick and alerts if the window passes. While it works the runner is drained, and it is resumed (runner resume) afterwards, including on failure.

Technical notes ​

  • Each control plane service also has a loopback endpoint inside its container: GET /idle answers 204 when idle and 409 when busy, and POST /resume lets it take work again. They are not reachable from outside the container.
  • A task's attempt budget is 3 claims for code and research, 5 resumes in gates or published. A restart that costs an attempt is worth avoiding. See A task's life.
  • The kill switch is a separate control: a signed stop makes runners hand out no jobs and intake create no tasks, and running jobs stop at their next lease renewal.

Sources ​