Monitoring AI systems in production: drift, performance, and bias

What to monitor beyond infrastructure metrics

Deploying a model is not the end of the engineering work. Production AI systems can degrade when input data changes, when labels arrive late or not at all, or when application behavior shifts around the model. AI observability is the practice of collecting the signals needed to detect those failures, explain them, and decide whether to retrain, roll back, or add guardrails.

What to monitor beyond infrastructure metrics

CPU, memory, and request rate still matter, but they do not tell you whether a classifier is becoming less useful or whether a generative AI workflow is producing risky outputs. NIST's AI Risk Management Framework is designed to help organizations incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. In operational terms, that means monitoring should cover the data going in, the outputs coming out, the system behavior around the model, and the human process for responding to issues.

A practical monitoring plan usually separates four signal groups:

• Data quality signals: missing values, schema changes, unexpected categories, range violations, and volume anomalies.
• Distribution signals: differences between a reference dataset and live traffic, often called data drift or feature drift.
• Outcome signals: accuracy, calibration, task success, refusal behavior, or user feedback when ground truth is available.
• Service signals: latency, timeout rate, token usage, tool-call failures, and cost for LLM-backed systems.

Drift detection is a comparison problem

Drift monitoring compares a current window of production data with a baseline, such as training data, validation data, or an earlier production window. The right test depends on the data type and the cost of false alarms.

Two common statistical methods are the Population Stability Index (PSI) and the Kolmogorov-Smirnov (KS) test. PSI bins a feature's values and compares the expected and observed proportions; teams often use it for tabular features because the result is easy to track over time. The KS test compares empirical cumulative distribution functions and is commonly used for continuous numeric features. Neither method proves that model quality has dropped. A drift alert means the input distribution changed enough to investigate; you still need outcome data, slice analysis, or shadow evaluation to understand impact.

Concept drift is harder because the relationship between inputs and outputs changes. In fraud detection, recommendation, or demand forecasting, user behavior can shift while raw feature distributions look stable. When labels are delayed, teams may monitor proxy outcomes such as manual review rates, downstream conversion, or escalation volume, and treat those proxies as indicators rather than truth.

Performance, latency, and LLM traces

Traditional ML monitoring often centers on prediction quality after labels arrive. LLM and agent systems need additional runtime observability because failures can happen inside prompts, retrieval steps, and tool calls. OpenTelemetry's semantic conventions work has defined GenAI-related conventions in the OpenTelemetry GenAI semantic conventions effort, reflecting the need for consistent telemetry across model providers and frameworks. Even when a team uses a vendor-specific SDK, the underlying goal is the same: capture enough structured traces and metrics to explain why a response was slow, expensive, or wrong.

Useful production metrics include end-to-end latency, model latency, retrieval latency, timeout rate, token counts, cache hit rate, and error classes. For retrieval-augmented generation, log which documents were retrieved and which prompt template version was used. For agent workflows, record tool inputs and outputs with appropriate redaction. Logs and traces should be treated as sensitive data; prompts and model outputs can contain personal data, credentials, or proprietary content.

MLflow's tracing documentation describes OpenTelemetry-compatible tracing for LLM and agent workflows, including spans and attributes. That trace data is valuable when paired with evaluation datasets and explicit release criteria, not as a substitute for them.

Illustrative trace-field checklist (adapt to your stack): request or trace ID, user or tenant segment, model or version, prompt or template ID, retrieved document IDs, tool name and status, latency per span, token or cost metadata, guardrail decision, and a final outcome label when available.

Bias and governance signals need context

Bias monitoring is not a single dashboard number. A model can show uneven error rates across groups, but the relevant fairness question depends on the product, the decision being made, available labels, and legal or policy constraints. NIST released Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) on July 26, 2024, which organizations can use to identify risks posed by generative AI and select risk management actions aligned with their goals and priorities.

In practice, teams should define which slices matter before deployment, document why those slices were chosen, and record when data is too sparse to support a conclusion. Fairness reviews also need a path to action: disable a feature, add human review, adjust thresholds, improve training data, or limit use cases. Monitoring without an owner and an escalation path produces dashboards, not risk management.

Make alerts actionable

Every alert should answer three questions: what changed, why it matters, and what to do next. Good runbooks map a signal to an investigation step. A PSI spike on a key feature might trigger schema validation and a sample comparison. A latency increase might trigger checks for prompt size growth, retrieval fan-out, or provider incidents. A rise in harmful-output reports might trigger prompt review, guardrail evaluation, and a rollback.

Thresholds should be calibrated. Static limits are easy to start with, but production traffic has seasonality and batch effects. Review alert precision regularly; noisy monitors get ignored.

Conclusion

• Start here (week 1): pick one high-impact model or agent path, define SLOs and a named owner, and route alerts for latency, error rate, and one quality or drift proxy to a runbook.
• Start here (week 2): add trace or request IDs across the model, retrieval, tools, and application so on-call can answer “what changed?” and “who was affected?”
• Start here (month 1): review one production slice, or an explicitly sampled evaluation set for LLMs, with the model owner and product owner; decide retrain, rollback, guardrail, or accept risk.
• Owner/escalation: the service owner handles alerts; escalate unresolved quality, safety, or bias signals to the risk or governance owner named in your NIST AI RMF governance process.
• Uncertainty: when labels are delayed, slices are sparse, or LLM evaluation coverage is incomplete, treat dashboards as directional and require human review before high-impact actions.

Sources

1. NIST AI Risk Management Framework, https://www.nist.gov/itl/ai-risk-management-framework
2. NIST AI 600-1: Generative Artificial Intelligence Profile, https://doi.org/10.6028/NIST.AI.600-1
3. MLflow Tracing for LLM and Agent Observability, https://mlflow.org/docs/latest/genai/tracing/
4. MLflow documentation, https://mlflow.org/docs/latest/
5. OpenTelemetry GenAI semantic conventions repository, https://github.com/open-telemetry/semantic-conventions-genai
6. Evidently AI documentation, https://docs.evidentlyai.com/

← back to posts