BitNet SME — Self-Optimizing DSPy Expert Service

A FastAPI service that serves specialized domain experts as one typed contract, built on DSPy 3.2. Each domain is a real dspy.Module; a router dispatches the question; LiteLLM handles provider routing; MLflow traces every run; and a dspy.Evaluate harness drives MIPROv2/GEPA prompt optimization. A 1-bit bitnet.cpp local provider is an optional fallback for offline / PII / air-gapped workloads. (Renamed from “BitNet SME Expert v2.0” — v2 was string stubs with no DSPy or BitNet code; v3 rebuilt the expert layer for real.)

📖 GitHub Repository

What runs today

• Per-domain dspy.Module experts under app/dspy_modules/ — a math module (dspy.ReAct over seven SymPy tools plus a deterministic arithmetic fast-path), a code module, and a general module (dspy.ChainOfThought, with an opt-in dspy.ReAct hybrid-RAG path over LanceDB with a BGE-family embedder + bge-reranker-v2-m3 when RAG_ENABLED). A router module dispatches by domain.
• Provider routing via LiteLLM — per-role LM specs (router / math / code / general) with DSPY_LM_<ROLE> env overrides, so any provider (or the local bitnet.cpp server) can back any role.
• MLflow tracing — autolog wired into the DSPy configuration.
• An optimization loop with an enforced eval gate — scripts/optimize.py compiles each domain with MIPROv2/GEPA; scripts/run_eval.py runs dspy.Evaluate per domain against per-domain thresholds. eval.yml runs on every pull request to main (a required status check via branch-protection.yml), posting a per-domain score table with Wilson 95% CIs to the PR. A committed receipt (eval/receipts/math-miprov2.json) records a real MIPROv2 win on math: baseline 0.62 → 0.80 (+0.18) on a held-out split (50-row train / 50-row holdout per domain).
• An MCP server — app/mcp_server.py (official mcp SDK, dspy-sme-mcp console script) exposes the math/code/general experts as Model Context Protocol tools over the same DSPy programs.
• One typed FastAPI contract with the usual plumbing (Pydantic v2 schemas, SlowAPI rate limiting, CORS, Docker multi-stage build).

The air-gapped thesis

The defensible niche isn’t “we run BitNet” — the model is open-weight. It’s the corpus + a distillation/eval pipeline packaged to run fully offline on a CPU, for sovereignty/regulated buyers (legal, finance, defense, HIPAA) where the data never leaves the enclave. Paired with SUM (verifiable, citeable answers offline) and InfiniteContext (on-device retrieval without a cloud vector DB), it forms an offline expert stack that cloud incumbents structurally can’t ship.

What’s next (honest gaps)

• Eval receipts beyond math. The committed receipt covers the math domain (+0.18 MIPROv2 win on a 50-row held-out split); code and general have train/holdout gold sets but no committed baseline-vs-compiled win yet. Scaling the splits past 50/domain would tighten the Wilson CIs further.
• A measured BitNet receipt. scripts/bitnet_demo.sh measures real tok/s through the DSPy path and writes a dated eval/receipts/bitnet-<date>.txt, but it needs real hardware to run, so no measured figure is committed yet (the README deliberately ships none until the demo produces one).
• A published/installable artifact. No GitHub release, package, or hosted demo yet — the prerequisite for promoting this past in-development.

Technical Stack

• Framework: Python ≥3.12 (uv-managed) / FastAPI / Pydantic v2
• Optimization: DSPy 3.2 (dspy.Module / dspy.ReAct / dspy.ChainOfThought, dspy.Evaluate, MIPROv2 / GEPA)
• Provider routing: LiteLLM (per-role LM specs, env overrides)
• Tracing: MLflow autolog
• Retrieval (opt-in): LanceDB + a BGE-family embedder & bge-reranker-v2-m3 (hybrid RAG) for the general expert
• Local/offline: bitnet.cpp 1-bit server (OpenAI-compatible)
• Domain tools: SymPy (math)
• Ops: Docker multi-stage build, Makefile, pytest, ruff