Forelead handbook

Appearance

Mirror and the shared gate cache ​

Sandboxes have no internet. Everything a job downloads (Go modules, npm, PyPI, Maven, Cargo, NuGet packages) comes from one host, the package mirror, which the per-job proxy is allowed to reach. If the mirror is down or does not allow a package, builds fail with a download error.

The mirror ​

PathUpstreamServed by
/go/proxy.golang.org and sum.golang.orgCaddy (pull-through)
/npm/registry.npmjs.orgVerdaccio (pull-through cache)
/pypi/, /maven/, /cargo/, /nuget/PyPI, Maven Central, crates.io, NuGetcmd/packagemirror, allow-listed and request-logged
  • It is its own Compose project, forelead-mirror, so the main stack can be upgraded without touching it. Stop it before any docker compose down of the main stack: Docker cannot remove the ingress network while Caddy is attached.
  • It is read only (GET and HEAD). Nobody can log in to npm or publish.
  • Adding a package is a reviewed change to the instance's allow-list files (pypi.txt, maven.txt, cargo.txt, nuget.txt: exact names, one per line). The lists are re-read on every request, so no restart. Work out the names by resolving the project (for example from uv.lock); do not allow whatever the mirror happens to refuse.
  • Caddy gets a Let's Encrypt certificate for the mirror host by Cloudflare DNS-01, using the cloudflare_token secret. If that token is dead, the certificate cannot renew.
  • Check it end to end with deploy/mirror/test.sh (--prod against the real host).

The shared gate cache ​

Gates start warm from shared cache layers kept on the runner host. The kinds are go, gomod, gobuild and pnpm. A job never mounts the shared layer: it gets a private copy inside its own cache directory, thrown away with the job. Only a trusted run (a gate on the base commit, from preflight) may promote its cache into the shared layer, and every seed re-checks the layer digest. If a layer is missing, damaged or not configured, the run starts cold (log line shared cache layer unusable; running cold). If promotion fails, later runs start cold (promote shared cache failed).

The cache root must be a clean absolute path, a directory not writable by group or others, and owned by the runner user, or the runner refuses it. Preflight evidence records which layer digests a run was seeded from (cacheLayers).

Technical notes ​

  • Instance settings (MIRROR_HOST, MIRROR_INGRESS_IP, the image names, MIRROR_STATE, MIRROR_ALLOW_DIR) come from an env file kept with the instance's configuration in forelead-ai/ops. On AWS the mirror host runs deploy/cloud/start.sh mirror and keeps its cache in /var/lib/forelead/mirror-cache.
  • The job proxy permits only two HTTPS destinations, the mirror and the gateway, and rejects private DNS answers. If the mirror is private, pin it with PROXY_MIRROR_IP.
  • The gate cache issue is #1125; the code is internal/sandbox/sharedcache.go, cacheplan.go and internal/gates/basecache.go.

Sources ​