Responding to application alerts
Configure targets and response thresholds for your application's workload. The framework does not establish an application's latency SLO, error budget, on-call team or rollback approval.
Investigate an alert
- Identify the application version, deployment time, affected route or job, tenant boundary and trace/run IDs. Redact credentials and customer data.
- Compare metrics and traces with the application's baseline. Distinguish exporter delivery failure from an actual workload failure.
- Check dependencies, recent changes, queue retry volume and cache behavior. Authenticate before inspecting tenant-specific resources.
- Apply the application's tested recovery or rollback procedure. Verify health/readiness and the original failing request or job afterward.
For cache anomalies, inspect key cardinality, tenant/host variants and hit rate; never prewarm or share private responses merely to improve a metric. For a job failure storm, bound retries and verify idempotency before replaying work. For slow HTTP requests, compare spans and dependency latency before changing limits or capacity.
Optional AI assistance
Provide a redacted metric summary and ask for hypotheses with supporting evidence. Treat output as advice. Verify each proposed action and use your normal authorization and rollback process for production changes.
See telemetry, notifications, anomaly detection and operational templates.