Work
Observability for AI Workflows
The monitoring, alerting, and testing stack for a durable AI workflow surface, owned end to end.
Brief
No one owned monitoring end to end for the team’s durable AI workflow surface. Failures and quality regressions were often caught late, if at all.
Approach
- End-to-End Metrics: Defined and instrumented Prometheus metrics (response times, error rates by type) across the workflow surface, and built the backing Grafana dashboards.
- On-Call & Alerting: Set up automated severity alerting tied to runbooks. Went on call, supported senior engineers, and wrote the onboarding documentation.
- Live Debugging: Handled critical resolutions by SSHing directly into affected regions to debug live, rolling back or modifying Helm charts and deployments.
- Testing & Security: Wrote the full test suite (unit to load tests) and kept dependency/security patches current across all regions.
- User Feedback: Funneled lightweight feedback forms into the pipeline as a secondary satisfaction signal.
Result
Mean time to detect production issues dropped by more than half. Critical incidents were typically resolved within a few hours of paging. The runbooks and documentation became the reference other engineers used to onboard onto on-call.