Telemetry — Alerts

Overview

The alerts subsystem is a registry of named monitoring rules (AIAlertOptions) that the AIObservabilityWorker evaluates on a periodic interval. Each rule defines which metric to watch, how to detect a problem (threshold comparison or anomaly scoring), and where to route the notification.

Rules are registered imperatively at startup. The registry is a simple in-memory array — there is no persistence. It is intentionally lightweight so that alert definitions can live alongside the code they monitor.

Key Concepts

• Threshold mode — compares the current metric value against a fixed threshold using an operator. Simple and predictable.
• Anomaly mode — passes the metric value through RollingAnomalyDetector and fires when the z-score exceeds sensitivity. Adapts to the metric's natural baseline automatically.
• Registry functions — registerAIAlert, listAIAlerts, and resetAIAlerts manage the in-memory rule store.
• RegisteredAIAlert — the wrapper stored in the registry. Contains the rule options plus metadata about the decorated method (target, methodName).

API Reference

registerAIAlert(rule: RegisteredAIAlert): void

Adds a rule to the global registry. The worker reads the registry on every evaluation cycle, so rules registered after the worker has started are picked up on the next tick.

import { registerAIAlert } from '@tagna/udiot/telemetry';

registerAIAlert({
  target: 'OrdersService',
  methodName: 'processOrder',
  options: {
    name: 'High error rate',
    metric: 'orders.error_rate',
    mode: 'threshold',
    operator: '>',
    threshold: 5,
    severity: 'critical',
  },
});

listAIAlerts(): RegisteredAIAlert[]

Returns a shallow copy of all registered rules. Safe to iterate without locking.

resetAIAlerts(): void

Empties the registry. Call in afterEach / afterAll in tests to prevent rule bleed between test cases.

AIAlertOptions

All fields are optional to allow incremental configuration.

AlertOperator

type AlertOperator = '>' | '>=' | '<' | '<=' | '==' | '!=';

RegisteredAIAlert

interface RegisteredAIAlert {
  target: string;      // class or module name
  methodName: string;  // method or function name
  options: AIAlertOptions;
}

target and methodName are used by notifiers to identify the origin of the alert in log output and Slack messages.

Basic Usage

Threshold alert: error rate above 5%

import { registerAIAlert } from '@tagna/udiot/telemetry';
import { getMetricsRegistry } from '@tagna/udiot/telemetry';

// Register the rule at startup
registerAIAlert({
  target: 'HttpServer',
  methodName: 'requestHandler',
  options: {
    name: 'Error rate too high',
    description: 'HTTP 5xx error rate exceeded 5%. Check recent deployments.',
    mode: 'threshold',
    metric: 'http.error_rate',
    operator: '>',
    threshold: 5,
    severity: 'critical',
    playbookUrl: 'https://wiki.internal/runbooks/high-error-rate',
  },
});

// Somewhere in your request handler, update the metric
const registry = getMetricsRegistry();
function recordRequest(isError: boolean) {
  registry.increment('http.requests.total');
  if (isError) {
    registry.gauge('http.error_rate', computeErrorRate());
  }
}

Anomaly alert: response time deviation

registerAIAlert({
  target: 'PaymentsService',
  methodName: 'chargeCard',
  options: {
    name: 'Response time anomaly',
    description: 'Payment processing time has deviated significantly from baseline.',
    mode: 'anomaly',
    metric: 'payments.response_ms',
    sensitivity: 2.5,   // z-score threshold
    windowSize: 60,     // last 60 samples form the baseline
    minSamples: 10,     // require 10 samples before firing
    severity: 'warning',
    channel: 'notify_team',
  },
});

Using the condition DSL string

The worker can parse simple "metric operator threshold" strings as a fallback when metric, operator, and threshold are not set individually:

registerAIAlert({
  target: 'QueueWorker',
  methodName: 'processJob',
  options: {
    condition: 'queue.depth > 1000',
    severity: 'warning',
  },
});

Percentage signs are stripped before parsing, so "error_rate > 5%" is equivalent to "error_rate > 5".

Common Patterns

Centralized rule file — define all alert rules in a single alerts.ts module and import it in your application entry point. This keeps monitoring rules discoverable and co-located.

Per-environment thresholds — read threshold from environment variables so production and staging have different sensitivity levels without code changes.

Testing alert rules — call resetAIAlerts() before each test, register the rule under test, construct an AIObservabilityWorker with a custom metricsProvider, call worker.evaluate(), and assert the notification in the onAlert callback.

Integration Notes

Alert rules are evaluated by AIObservabilityWorker (see ai-observability.md). The registry is passive — registering a rule has no effect until a worker reads it. You can register rules before or after creating the worker; the worker reads listAIAlerts() on every evaluation cycle.

Pitfalls

• If both metric (direct) and condition (DSL) are set, metric takes precedence.
• If mode is omitted and the alert has a threshold, the worker defaults to threshold mode. If it has sensitivity but no threshold, the worker infers anomaly mode. Set mode explicitly to avoid ambiguity.
• resetAIAlerts() mutates the module-level array in place. It affects all workers sharing the same process, so use it only in test environments.