Telemetry — LLM Incident Summarizer

Overview

When an alert fires, the AIObservabilityWorker calls an IncidentSummarizer to generate a human-readable explanation and recommendation. UDIOT ships two implementations:

• LLMIncidentSummarizer — calls an OpenAI-compatible chat completions API and returns a structured JSON response containing a summary, recommendation, and confidence score.
• RuleBasedIncidentSummarizer — a zero-dependency fallback that generates summaries from deterministic templates based on the incident type, severity, and anomaly statistics.

The two are designed to be used together: LLMIncidentSummarizer accepts a fallback option so that if the API call fails, the rule-based summarizer is invoked automatically.

Key Concepts

• IncidentSummarizer interface — both implementations share the same summarize(context): Promise<IncidentInsight> contract. You can implement your own by satisfying this interface.
• IncidentContext — the input to the summarizer, built by AIObservabilityWorker from the current alert, metric value, and optional anomaly result.
• IncidentInsight — the output: a short summary, an optional recommendation, and an optional confidence score between 0 and 1.
• Structured output — LLMIncidentSummarizer uses OpenAI's response_format: json_schema to enforce a typed JSON response from the model.
• Environment variable configuration — all LLM options can be sourced from environment variables, making it easy to configure without code changes.

API Reference

LLMIncidentSummarizer

import { LLMIncidentSummarizer } from '@tagna/udiot/telemetry';

const summarizer = new LLMIncidentSummarizer(options?);

Constructor: LLMIncidentSummarizerOptions

Note: Both endpoint and apiKey are required at construction time. If either is missing (and cannot be resolved from environment variables), the constructor throws an Error.

.summarize(context: IncidentContext): Promise<IncidentInsight>

Calls the LLM API with a structured prompt describing the incident. On API failure, delegates to the fallback summarizer and logs a warning.

IncidentContext

The input built by AIObservabilityWorker and passed to the summarizer:

IncidentInsight

The output returned to AIObservabilityWorker and attached to AlertNotification:

IncidentSummarizer interface

interface IncidentSummarizer {
  summarize(context: IncidentContext): Promise<IncidentInsight>;
}

Implement this to build a custom summarizer (e.g., routing to a different LLM, applying policy-based filters, or caching summaries).

Environment variables

createLLMSummarizerFromEnv(): IncidentSummarizer | undefined

A convenience factory that reads all configuration from environment variables. Returns undefined if TELEMETRY_SUMMARIZER_URL or an API key is not set. Useful for zero-config initialization:

import { createLLMSummarizerFromEnv, RuleBasedIncidentSummarizer } from '@tagna/udiot/telemetry';

const summarizer = createLLMSummarizerFromEnv() ?? new RuleBasedIncidentSummarizer();

RuleBasedIncidentSummarizer

A deterministic, zero-dependency implementation that produces meaningful summaries without any external API.

Anomaly mode summary format:

Detected {zScore}σ anomaly on {metricName} ({value} vs µ={mean}) | Δ +{percentDelta}%

Threshold mode summary format:

Threshold breach on {metricName}: observed {value} vs limit {threshold}

Confidence score — for anomaly incidents, confidence is computed as min(0.99, |zScore| / (sensitivity * 2)). A z-score of 5 with sensitivity 2.5 gives a confidence of min(0.99, 1.0) = 0.99. For threshold incidents, confidence defaults to 0.6.

Recommendations by severity:

If playbookUrl is set on the alert rule, the recommendation becomes "Review playbook: {playbookUrl}" regardless of severity.

Basic Usage

Production setup: LLM primary with rule-based fallback

import {
  LLMIncidentSummarizer,
  RuleBasedIncidentSummarizer,
  AIObservabilityWorker,
} from '@tagna/udiot/telemetry';

const summarizer = new LLMIncidentSummarizer({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-4o-mini',
  temperature: 0.2,
  fallback: new RuleBasedIncidentSummarizer(), // automatic fallback on API failure
});

const worker = new AIObservabilityWorker({
  intervalMs: 30_000,
  summarizer,
  // Deliver through onAlert using an application-owned notification integration.
});

worker.start();

Environment-variable-driven setup

export TELEMETRY_SUMMARIZER_URL=https://api.openai.com/v1/chat/completions
export TELEMETRY_SUMMARIZER_API_KEY=sk-...
export TELEMETRY_SUMMARIZER_MODEL=gpt-4o-mini
export TELEMETRY_SUMMARIZER_PROVIDER=openai

import { createLLMSummarizerFromEnv, RuleBasedIncidentSummarizer, AIObservabilityWorker } from '@tagna/udiot/telemetry';

const worker = new AIObservabilityWorker({
  summarizer: createLLMSummarizerFromEnv() ?? new RuleBasedIncidentSummarizer(),
});
worker.start();

Sending to an OpenAI-compatible endpoint (e.g., Ollama, Azure OpenAI)

const summarizer = new LLMIncidentSummarizer({
  provider: 'generic',
  endpoint: 'http://localhost:11434/api/generate',
  apiKey: 'not-required-but-must-be-non-empty',
  model: 'llama3.2',
});

Use provider: 'generic' for any API that accepts a { model, prompt, temperature } body and returns the result directly (not wrapped in an OpenAI choices array).

Custom summarizer

import { IncidentSummarizer, IncidentContext, IncidentInsight } from '@tagna/udiot/telemetry';

class MyCustomSummarizer implements IncidentSummarizer {
  async summarize(context: IncidentContext): Promise<IncidentInsight> {
    // Build your own summary logic
    return {
      summary: `Alert on ${context.metricName}: value ${context.metricValue}`,
      recommendation: 'Check the dashboard.',
      confidence: 0.7,
    };
  }
}

Common Patterns

Development without an API key — use RuleBasedIncidentSummarizer directly. It requires no network access and produces useful output for all alert types.

Caching — wrap LLMIncidentSummarizer in a caching layer keyed on context.alert.options.name + context.triggerType to avoid calling the API for identical incidents firing repeatedly.

Cost control — set model: 'gpt-4o-mini' (the default) for low cost. gpt-4o-mini produces high-quality incident summaries at a fraction of the cost of full-size models for this structured task.

Pitfalls

• The LLMIncidentSummarizer constructor throws synchronously if both endpoint and apiKey cannot be resolved. Wrap construction in a try/catch and fall back to RuleBasedIncidentSummarizer if the LLM is optional in your environment.
• The 'generic' provider does not use the json_schema response format. The model must return JSON in the expected shape ({ summary, recommendation?, confidence? }) through prompt engineering alone.
• LLM API latency adds to the time between an alert firing and a notification being dispatched. For time-sensitive alerts (severity 'critical'), consider using RuleBasedIncidentSummarizer or a cached response to ensure notifications arrive quickly.