Field Notes - Nov 10, '25

Executive Signals

• Variance beats vibes: detect drift against baselines, not absolute clocks
• One pipe to truth: centralize failures, mirror context where ops actually works
• Land, then instrument: ship value before telemetry, guardrails gate scale
• Determinism has a place: scripted for stable paths, agents for the long tail
• Split loops, ship faster: fast adapter tests, deliberate at-scale confidence before go-live

Customer Success

Single Error Pipe to Slack and CRM

Treat automation failures as first-class incidents. Funnel text logs and screenshots into a dedicated Slack incident channel, and mirror a concise status back to the linked compliance case in the CRM so ops stays informed without chasing engineering.

• Include payload: adapter, environment, entity ID, run ID, error class, message, screenshot URL
• Dedupe alert bursts within a 5-minute window to prevent floods
• Write back to the CRM case with cause, retry/disposition, and owner

Engineering

Alert on Variance, Not Vibes

Page on change versus absolutes. Track wall-clock runtime per submission and compare against rolling per-adapter baselines so you catch drift early, not just slow jobs.

• Maintain p50/p95 and σ; alert if runtime > baseline + 2σ for 3 runs or >30% over 10 runs
• Store timings centrally; review weekly heatmaps to spot site changes
• Treat big negative deltas as wins, but verify they aren’t silent failures

Land-Then-Instrument

Sequence matters. First land one functional end-to-end pass in a prod-like environment. Then add observability and code quality, and only scale once stability is proven.

• Day 0: deploy and prove a single pass in prod-like
• Day 1: add Sentry, SonarQube, structured security logs; backfill key alerts
• Gate expansion on 24 hours without P1s and <1% error rate

Run a Bake-Off: Agentic vs Scripted Automation

Agentic isn’t mandatory everywhere. Build a harness that runs both styles on the same flow and choose per adapter based on data, not preference.

• Measure over 100 runs: success rate, median/p95 latency, weekly maintenance touches
• Prefer scripted on stable, deterministic paths; agentic for high-change long tails
• Re-evaluate quarterly; promote or demote as site volatility shifts

Split Testing Into Adapter-Level and At-Scale

Keep developer loops fast while earning go-live confidence through a separate at-scale plan with QA/Compliance. Decouple tight unit signals from production readiness.

• Adapter-level: deterministic smoke tests with fixtures and synthetic data
• At-scale: volume schedule, coverage matrix, pass/fail gates, rollback steps
• Attach a failure taxonomy so incidents route to the right owner quickly

Time-Box Upstream Dependencies

When an upstream owner is firefighting, protect momentum with explicit check-ins and parallel tracks that keep critical work moving.

• Record the next check-in (e.g., Wednesday) with a one-line status
• Maintain a “blocked-but-progressing” lane for error pipe, testing, and infra
• Auto-escalate at T+7 days or any slip beyond one sprint