Operator runbook
Deploy, set secrets by name, check the result, and what to do when something goes wrong. Names only, never values.
This page is for the person who runs the server. It is built from docs/DEPLOY.md, docs/RUNBOOK-GATES.md, docs/RUNBOOK-BACKUP.md and infra/. Everything points at the PayPal .
Roles
| Role | Can do |
|---|---|
| Owner | Change repository secrets and the sandbox-nightly, staging and production environments, use the PayPal developer dashboard, run anything against the real sandbox from a laptop |
| Contributor | Read evidence and workflow logs, run the and the text-level voice eval, open pull requests. Never receives secrets |
Deploy
Apply the blueprint
In Render choose New, Blueprint, pick the repository and apply render.yaml. It describes one web service, gavel-host, with one instance and a 1 GB disk at /var/data. Health check path /healthz. Auto deploy is off. The plan must allow a persistent disk.
Fill the variables the blueprint leaves empty
Names only, set in the service Environment tab (see the table below).
Wire GitHub to Render
Copy the Render deploy hook. In GitHub, environment staging: secret RENDER_DEPLOY_HOOK_URL, optional secret ADMIN_KEY (same value as on Render, enables the create-auction smoke check), variable STAGING_URL.
Register the webhook
In the PayPal developer dashboard (sandbox app, Webhooks) add https://<api-host>/webhooks/paypal for PAYMENT.AUTHORIZATION.VOIDED, PAYMENT.AUTHORIZATION.DENIED, PAYMENT.CAPTURE.PENDING, PAYMENT.CAPTURE.COMPLETED, PAYMENT.CAPTURE.DENIED and PAYMENT.CAPTURE.DECLINED. Other types are acknowledged and ignored. Copy the id into PAYPAL_WEBHOOK_ID on Render only.
Run the deploy workflow and read the smoke output
Deploy main with Manual Deploy in the Render dashboard, then run scripts/smoke.sh <api-url>. The optional deploy-staging workflow does the same through the deploy hook once its secrets exist; until then it only reports SKIPPED.
Deploy the web app
The web app is a Cloudflare Worker with static assets (infra/web/wrangler.toml). Build with npm ci && npm run build -w @gavel/web, passing VITE_API_BASE and VITE_WS_BASE, then run npx wrangler deploy from infra/web (the deploy-web workflow does this on pushes to main). Add the Worker origin to CORS_ORIGINS.
Production uses the deploy-prod workflow: tag the commit or run it by hand, approval on the production environment (needs a required reviewer), deploy only if the commit is the head of main, poll GET /version until it serves the commit, then run the smoke test. If the commit never appears, the previous deploy is restored. Details: infra/rollback.md.
Secrets and variables, by name
Prop
Type
The full list the server reads is .env.example in the repository root. Never paste a value into an issue, a log or a pull request.
Post-deploy checks
| Check | Expected |
|---|---|
GET /healthz | 200 while the process runs |
GET /readyz | 200 when migrations are applied and the store answers, else 503 |
GET /warmup | 200 when the warm steps finish within 3000 ms, else 503 with a per-step report |
GET /version | {"sha":"<commit>"} of the deployed commit |
unsigned POST /webhooks/paypal | 400 |
POST /api/auctions without a key | 401 (201 with the admin key) |
By hand: ADMIN_KEY=... scripts/smoke.sh https://<api-host>. The smoke can pass against the old instance if it answers before the new build is live.
The uptime workflow probes the health routes plus the web and docs hosts on GitHub's best-effort schedule: it asks for every 15 minutes and has been observed running every 4 to 8 hours. It is on by default (UPTIME_ENABLED set to false turns it off). A failed round is re-probed after 30 seconds, and a failure that stands opens the alert issue at once. For a minute-level probe use an external monitor (see infra/uptime.md); none is configured yet.
What to do when
- Check the webhook list in the PayPal developer dashboard: the URL, the subscribed events and recent deliveries.
- Confirm
PAYPAL_WEBHOOK_IDon Render is the id of that webhook. A mismatch makes every verification fail. - An unsigned post must answer 400. A real event that fails verification is rejected, so a missing id shows up as rejected deliveries.
- Event ids are deduplicated, so a redelivery is safe. How the server treats events: webhooks.
- The retry behaviour of PayPal is UNVERIFIED (G4).
Review means the saga stopped on something it will not decide alone (an invariant break, an unknown capture that outlived ten minutes, a mismatch, or a void that failed five resumes). See saga states.
- Read what PayPal knows. List the authorizations of the sandbox account and compare with the journal export (
GET /admin/export?kind=payments&auction=<id>, admin key). - To release holds the review is keeping, call
POST /ops/auctions/<auctionId>/release-reviewwith the admin key. To void every hold of a hammered auction that is not captured, callPOST /ops/auctions/<auctionId>/void-all. Both are audited with the actoradmin, and answer 409busywhile a run holds the settlement lease. - Do not edit the journal by hand.
A timeout or 5xx on capture is an unknown answer, not a failure. The saga keeps the settlement in CAPTURING, never moves to the next bidder, and sends the same request id on every retry. After unknownDeadlineMs (600000, ten minutes) it goes to review. Wait for the deadline, then handle it as a review. See crash recovery.
Restart it on the same data directory (DATA_DIR, the disk at /var/data). The journal is saved before each effect, so the runner resumes. A resume that was cut off while holding the settlement lease waits for the lease time instead of losing it. Check /readyz, then the journal export.
Voice bids always go through the confirm prompt (G5). Without DEEPGRAM_API_KEY or GROQ_API_KEY the /audio socket falls back to cached transcripts (VOICE_STT unset means replay). For a demo use /replay/:recordingId, which makes no PayPal call. See voice and confirm.
A failed round that fails again on the re-probe raises the alert. Check /healthz first (process alive), then /readyz (store), then /warmup (it reports which step is cold). On the free tier the host sleeps; the keep-alive choice is decision D-0008 in infra/uptime.md. Warmup under 3 s after 15 minutes idle is not measured yet.
Run deploy-prod by hand with rollback set to true (or pass rollback_deploy_id). The job restores the newest earlier deploy built from a different commit and waits until /version serves it. The disk is not rolled back: a release that changed the stored data format also needs a data restore.
The web Worker rolls back with npx wrangler rollback from infra/web; the docs site rolls back to a previous deployment of its Cloudflare project. Roll web and API back together if a release changed the protocol between them. Steps: infra/rollback.md.
- Stop the server. Download the newest
ledger-snapshot.tgz.gpgartifact from thebackupworkflow and decrypt it with the private key kept off GitHub:gpg --decrypt ledger-snapshot.tgz.gpg | tar -xz. - The snapshot holds ledgers (
<auctionId>.ndjson, chain-checkable),payments/<id>.json(journal and holds),roomlog/<id>.ndjsonandpayments-global.json(processed webhook ids, kill switch state). It is a JSON view, not a byte copy: rebuildjournals.json,holds.jsonandwebhooks.jsonby hand or script. There is no import route. - The snapshot can be a day old. After a restore, reconcile against PayPal and void every authorization no restored record accounts for. Keep the kill switch engaged until then.
- This procedure is documented, not exercised. See backup.
Socket upgrades and HTTP joins share one per-address bucket. Raise RATE_JOIN_BURST (default 40; refill RATE_JOIN_PER_MIN, default 10). Each address may hold at most 20 open sockets.
- docs/DEPLOY.mdstaging deploy
- infra/rollback.mdproduction deploy and rollback
- infra/uptime.mdprobes and alert rule
- render.yamlthe Render blueprint
- .env.exampleevery variable name the project reads
- packages/server/src/http/ops.tsadmin ops routes
Last updated on