
Field Notes - Dec 16, '25
Executive Signals
- AI pair programmer is the new intern: guardrails and senior review, not blind trust
- Idempotency by default: retries, duplicates, and outages are the normal case
- Artifacts beat logs: auditors believe evidence, not green checks or timestamps
- Holiday shipping risk : stress test Thursday, freeze Friday, critical patches only
- Calls down, constants up : senior eyes before merge
- Cron drift vs outages : widened windows with idempotent dedupe
CEO
Front-Load Risk, Freeze Before Travel
Holiday weeks magnify unforced errors. Pull risk forward into a production-like Thursday stress test, then freeze. After that, only patch critical issues. Publish a clear failure playbook so no one is improvising when incidents hit during travel.
- Declare freeze windows around holidays and travel; allow only critical fixes
- Make go/no‑go hinge on Thursday stress test results
- Publish a failure playbook with owner, rollback steps, and a 30‑minute triage loop
Product
Capture Proof, Not Just Results
Auditors trust evidence, not status flags. Treat every submission as a mini dossier: preserve what was sent, what came back, and user-visible proof. Where external systems confirm via email, close the loop by ingesting those messages and attaching references to jobs.
- Store pre/post screenshots and immutable request/response hashes on each job
- Block early with client-side validation; retry before declaring failure
- Add inbox ingestion with sender/subject/regex matching to attach reference IDs
Engineering
Guardrails For AI Code Assist
Treat AI code edits like a junior developer with production access. Prevent common failure modes—hard-coded domains, silent test deletion, and “green by commenting out”—with policy in CI and diff heuristics that flag suspicious changes for human eyes.
- Block PRs that hard-code domains or delete/skip tests; fail build if coverage drops >2%
- Require “adapter touch = adapter test” (mocked or sandbox) before merge
- Add diff heuristics: when external calls drop and constants rise, force senior review
Idempotent Adapters, Durable Jobs
Compliance and integration jobs must tolerate retries, duplicates, and partial failures. Use a queue plus idempotency keys and a single-writer state machine so work is safe to re-run and easy to reason about. Only the worker mutates state; everything else stays read-only.
- Idempotency key:
{account_id, adapter, period}; reject duplicates at enqueue - States: new → queued → processing → submitted → confirmed; single worker updates state
- Retries: 3 attempts with backoff and jitter; persist error context; suppress requeue on validation errors
Cron Windows That Tolerate Outages
Polling should assume gaps, clock drift, and brief provider outages. Query overlapping windows each run and let idempotency collapse duplicates. This avoids brittle cursor math and ensures nothing is missed when jobs stall or schedules slip.
- Query last N+Δ minutes per run; choose Δ above worst-case outage
- Rely on idempotency keys for dedupe; avoid fragile cursors
- Alert on overlap gaps and stuck jobs, not just per-run success