Forelead handbook

Appearance

AWS tenant ​

A tenant is one customer's Forelead, running on three AWS hosts. The hosts are built by the forelead-ai/ops repository (modules/tenant); this repository supplies what runs on them. This page is the high-level map. The details are in the single-host guide, section "AWS tenant hosts".

The three hosts ​

RoleCompose projectWhat it runs
appforelead-appControl plane, broker, gateway, model watcher, proxy, migrate. Database is RDS, evidence goes to an S3 bucket.
runnerforelead-runnerThe agent runner and the preflight runner as containers, with the host Docker socket. Job containers run under gVisor.
mirrorforelead-mirrorThe package mirror (see Mirror).

How a host starts ​

All three start from one deploy bundle unpacked at /opt/forelead/current: platform/ is this repository at a commit, and instance/ holds the non-secret values (app.env, runner.env, mirror.env, mirror/ allow-lists). Secrets are files in /run/forelead/secrets (see Secrets layout). systemd runs sh platform/deploy/cloud/start.sh app|runner|mirror as root. It is idempotent and stops with a message naming any missing variable or secret file. Images come from ECR by digest.

Things worth knowing ​

  • Migrations and database roles are applied against RDS first (service-roles.sh, as forelead_owner). start.sh runs only the migrate one-shot.
  • IMDS pinning. The control plane and broker sit on an extra network (fl-imds) at fixed addresses so they alone can reach S3 and IMDS. The ops side allow-lists those addresses: do not change the bridge name or addresses.
  • Known shortcut: the control plane's GitHub-only egress is enforced by NO_PROXY, not by the network. The planned fix is an S3 gateway VPC endpoint with an egress rule.
  • Runners reach the gateway directly on the app host's private IP, port 443. start.sh refuses any other gateway URL.
  • Private images: the agent image comes from ghcr.io using the optional ghcr_pull_token.
  • Mirror host: its TLS certificate comes from Let's Encrypt through Cloudflare DNS-01.

Not covered here ​

The moving-day steps (cutting a tenant over) and the infrastructure itself live in the forelead-ai/ops repository and are not described in this one, so they are not copied here. See Switchover for the intake switchover from Switchboard.

Sources ​