HFAO — Hugging Face Agent Observatory

An open-source observability backend for AI agents — debug what your agents are doing, evaluate their outputs against expected behavior, and catch quality drift before users do. Standards-native: ingests traces through vendor-neutral OTel GenAI + OpenInference (no proprietary wire format), makes every observability primitive queryable by any MCP client, and closes the eval-trace loop in a single schema — traces become dataset items become evaluator inputs become scores become monitor triggers become traces.

Three Deployment Shapes From One Codebase

• Single-file HF Space — DuckDB embedded, the whole observatory running in one hosted notebook
• Docker Compose self-host — ClickHouse + Granian OTLP ingest, Redis Streams buffer
• Kubernetes enterprise — ClickHouse Cloud, Helm-deployed, multi-tenant

Three Pillars Commercial Competitors Can’t Easily Copy

1. Standards-nativeness done right. OTel GenAI + OpenInference on ingest, full OTLP compatibility, no proprietary wire lock-in. Commercial vendors hedge this because it commoditizes their backend — HFAO has no reason to.
2. MCP-native queryability. Traces, scores, causal edges, costs, prompts, datasets, experiments — all queryable by any MCP client. The observability backend agents debug themselves with.
3. Closed eval-trace loop. One schema, one system. Not three SaaS products glued together.

Current State

• v1.0.0 (public-release packaging) on main — Phase-1 feature parity landed at v0.5.0 (196 passed / 6 skipped); v1.0.0 reaches the public-release-packaging milestone at 270 passed / 6 skipped (CHANGELOG), with Q-20 counterfactual replay, Q-18 anomaly surfacing, and Q-19 insight routing merged. Still a source-tree milestone — no pushed tag, no published PyPI package, and no hosted Space yet: the HF-Space / Docker publish workflows are placeholder stubs pending credentials.
• Storage plane — ClickHouse backend (§4.3 DDL, §6.1 Docker shape) and DuckDB backend with parity tests between the two
• Ingest plane — OTel GenAI + OpenInference normalizer (§5), Granian OTLP server (§7.1), PII redaction (§6.5), bounded buffer with memory + Redis Streams (§7.1–§7.3), body offload at 64 KiB (§6.6)
• MCP server — FastMCP Streamable HTTP at :4319/mcp, the full §9.2 read surface plus a gated score_observation write (the queryability pillar)
• Closed eval-trace loop — the computation plane (causal attribution, eval engine, cost rollups, monitors, retention, parquet export) operational end-to-end (§8)
• Stage 2 counterfactual replay — landed on main post-v0.5.0 (SPEC §16 Q-20): driver-pluggable replay across the three Tier-1 frameworks (LangGraph / OpenAI Agents SDK / Claude Agent SDK), tested with deterministic in-process stub agents (no live LLM calls)
• Proactive anomaly surfacing + insight routing — also post-v0.5.0 on main (SPEC §16 Q-18/Q-19): an Insight schema + AnomalyEngine (rolling-mean Z-score, Western Electric control-chart rules, KS distribution drift, calibration drift, replay-verified) running alongside the monitor worker, plus a rule-based InsightRouter matching insights/alerts to role/user/agent subscriptions; surfaced in the cockpit Insights tab and via MCP
• Cockpit UI — Gradio single-file cockpit (§10) with the analyst tabs; auth / RBAC / multi-tenancy (§13) and framework instrumentations (LangGraph, OpenAI-Agents, Claude-Agent, smolagents, …) shipped
• Spec discipline — SPEC.md locked at v1.0.0; every commit cites a spec section. Silent deviation is forbidden — ambiguity gets an Open Question in §16

Future Directions

• A published artifact — tagged release + pip install hfao from PyPI (main is still a source-tree milestone with no pushed tag or package)
• Deploy shapes hardening — Helm chart + docker-compose single-binary/Docker/K8s shapes are in flight (open PR #22), not yet on main
• DuckLake warm-tier auto-sync (§16 Q-13, targeted v1.1)
• Console UI (SvelteKit analyst surface, §11) — scaffold exists but explicitly deferred to v2.0 per §16 Q-11; not yet a real surface

Positioning

Parity with LangSmith / Langfuse / Phoenix / Braintrust / Weave / Helicone on tracing, datasets, evals, prompts, annotation, cost, and monitoring. Beyond them on the three pillars above.

Technical Stack

• License: Apache-2.0
• Language: Python (hfao package)
• Storage: ClickHouse (self-host + enterprise), DuckDB (single-file)
• Ingest: Granian OTLP server, Redis Streams buffer, PII redaction
• Wire: OpenTelemetry GenAI + OpenInference (OTLP-compatible)
• Deploy: single-file HF Space, Docker Compose, Kubernetes (Helm)