Runbooks

API Health

• check GET /health
• verify Render deploy status
• confirm environment variables were synced correctly

Webhook Issues

• inspect pos_webhook_events_raw for unprocessed rows
• run POST /api/v1/internal/cron/process-pending-webhooks
• confirm Square signature config and Clover webhook secret are present

Square Poll Issues

• run POST /api/v1/internal/cron/square-poll-payments with header X-Cron-Secret
• verify CRON_SECRET
• check token expiry and square_payments_watermark_at on pos_connections
• staging cron salesarc-staging-jobs and production cron salesarc-production-jobs call this route from scripts/render-cron.sh every 5 minutes

Clover Poll Issues

• run POST /api/v1/internal/cron/clover-poll-payments with header X-Cron-Secret
• verify CRON_SECRET
• check clover_payments_watermark_at on pos_connections
• a watermark that does not advance usually means a payment is still settling (PENDING, INITIATED, or OFFLINE_RETRYING)
• the same Render cron script calls this route after the Square poll. A 404 is treated as a skip only when that deploy does not have the route yet

Campaign Send Issues

• run POST /api/v1/internal/cron/process-campaigns
• inspect campaign_recipients status rows
• confirm RESEND_API_KEY and RESEND_FROM_EMAIL

Wallet or Redemption Issues

• verify the transaction exists
• inspect wallet_ledger
• confirm the redemption code status in redemption_codes
• check for concurrent redemption attempts

Privacy Breach

• assess whether a breach creates a real risk of significant harm
• if it does, notify affected people and the Office of the Privacy Commissioner of Canada as soon as feasible
• if affected people include Quebec residents, also notify the Commission d'accès à l'information
• keep a record of every breach for 24 months
• do not put tokens or full payloads in that record
• POST /api/v1/internal/cron/purge-expired-personal-data is an operational check that aged raw POS payloads and campaign email snapshots are cleared. It is not the breach process