HFAO — Hugging Face Agent Observatory

An open-source observability backend for AI agents — debug what your agents are doing, evaluate their outputs against expected behavior, and catch quality drift before users do. Standards-native: ingests traces through vendor-neutral OTel GenAI + OpenInference (no proprietary wire format), makes every observability primitive queryable by any MCP client, and closes the eval-trace loop in a single schema — traces become dataset items become evaluator inputs become scores become monitor triggers become traces.

Three Deployment Shapes From One Codebase

• Single-file HF Space — DuckDB embedded, the whole observatory running in one hosted notebook
• Docker Compose self-host — ClickHouse + Granian OTLP ingest, Redis Streams buffer
• Kubernetes enterprise — ClickHouse Cloud, Helm-deployed, multi-tenant

Three design choices that differentiate HFAO

1. Standards-nativeness done right. OTel GenAI + OpenInference on ingest, full OTLP compatibility, no proprietary wire lock-in. Commercial vendors hedge this because it commoditizes their backend — HFAO has no reason to.
2. MCP-native queryability. Traces, scores, causal edges, costs, prompts, datasets, experiments — all queryable by any MCP client. The observability backend agents debug themselves with.
3. Closed eval-trace loop. One schema, one system. Not three SaaS products glued together.

Current State

• v1.0.0 (public-release packaging) on main — Phase-1 feature parity landed at v0.5.0 (196 passed / 6 skipped); v1.0.0 reaches the public-release-packaging milestone at 270 passed / 6 skipped (CHANGELOG), with Q-20 counterfactual replay, Q-18 anomaly surfacing, and Q-19 insight routing merged. Still a source-tree milestone — no pushed tag, no published PyPI package, and no hosted Space yet. The HF-Space / Docker publish workflows that used to stamp a green “success” on every push while echoing and building nothing were deleted in PR #29 (2026-08-03); tests/acceptance/test_workflows_are_honest.py now fails any workflow that names itself build/publish/deploy, runs automatically, and only echoes.
• Try it — from a source checkout, scripts/demo.sh walks the whole eval-trace loop with zero API keys and no cloud account: migrate, boot the cockpit, emit OTel GenAI spans from a sample agent, query the trace back, seed a golden dataset, and run a gated eval. It uses uv if present, otherwise an installed hfao. Landed in PR #23 (2026-06-09)
• Storage plane — ClickHouse backend (§4.3 DDL, §6.1 Docker shape) and DuckDB backend with parity tests between the two
• Ingest plane — OTel GenAI + OpenInference normalizer (§5), Granian OTLP server (§7.1), PII redaction (§6.5), bounded buffer with memory + Redis Streams (§7.1–§7.3), body offload at 64 KiB (§6.6)
• MCP server — FastMCP Streamable HTTP at :4319/mcp, the full §9.2 read surface plus a gated score_observation write (the queryability pillar)
• Closed eval-trace loop — the computation plane (causal attribution, eval engine, cost rollups, monitors, retention, parquet export) operational end-to-end (§8)
• Stage 2 counterfactual replay — landed on main post-v0.5.0 (SPEC §16 Q-20): driver-pluggable replay across the three Tier-1 frameworks (LangGraph / OpenAI Agents SDK / Claude Agent SDK), tested with deterministic in-process stub agents (no live LLM calls)
• Proactive anomaly surfacing + insight routing — also post-v0.5.0 on main (SPEC §16 Q-18/Q-19): an Insight schema + AnomalyEngine (rolling-mean Z-score, Western Electric control-chart rules, KS distribution drift, calibration drift, replay-verified) running alongside the monitor worker, plus a rule-based InsightRouter matching insights/alerts to role/user/agent subscriptions; surfaced in the cockpit Insights tab and via MCP
• Cockpit UI — Gradio single-file cockpit (§10) with the analyst tabs; auth / RBAC / multi-tenancy (§13) and framework instrumentations (LangGraph, OpenAI-Agents, Claude-Agent, smolagents, …) shipped
• Deploy shapes — Helm chart (helm/hfao) plus the docker-compose shapes (docker-compose.yml, docker-compose.clickhouse.yml) landed on main in PR #22 (2026-06-09), covered by tests/acceptance/test_ac_deploy_compose_and_helm.py
• Spec discipline — SPEC.md locked at v1.0.0; every build-plan commit cites a spec section (the §15.3 rule 1 convention, reproduced in CLAUDE.md; chores, CI, and merge commits are outside it). Silent deviation is forbidden — ambiguity gets an Open Question in §16

Future Directions

• A published artifact — tagged release + pip install hfao from PyPI (main is still a source-tree milestone with no pushed tag or package)
• DuckLake warm-tier auto-sync (§16 Q-13, targeted v1.1)
• Console UI (SvelteKit analyst surface, §11) — scaffold exists but explicitly deferred to v2.0 per §16 Q-11; not yet a real surface

Positioning

Scope: HFAO targets the same capability surface as LangSmith, Langfuse, Phoenix, Braintrust, Weave and Helicone — tracing, datasets, evals, prompts, annotation, cost and monitoring. No head-to-head evaluation against any of them has been run; the three differentiators above are design choices, not measured advantages.

Technical Stack

• License: Apache-2.0
• Language: Python (hfao package)
• Storage: ClickHouse (self-host + enterprise), DuckDB (single-file)
• Ingest: Granian OTLP server, Redis Streams buffer, PII redaction
• Wire: OpenTelemetry GenAI + OpenInference (OTLP-compatible)
• Deploy: single-file HF Space, Docker Compose, Kubernetes (Helm)