BitNet SME — Self-Optimizing DSPy Expert Service

A FastAPI service that serves specialized domain experts as one typed contract, built on DSPy 3.2. Each domain is a real dspy.Module; a router dispatches the question; LiteLLM handles provider routing; MLflow traces every run; and a dspy.Evaluate harness drives MIPROv2/GEPA prompt optimization. A 1-bit bitnet.cpp local provider is an optional fallback for offline / PII / air-gapped workloads. (Renamed from “BitNet SME Expert v2.0” — v2 was string stubs with no DSPy or BitNet code; v3 rebuilt the expert layer for real.)

📖 GitHub Repository

What runs today

• Per-domain dspy.Module experts under app/dspy_modules/ — a math module (dspy.ReAct over seven SymPy tools plus a deterministic arithmetic fast-path), a code module, and a general module (dspy.ChainOfThought, with an opt-in dspy.ReAct hybrid-RAG path over LanceDB with a BGE-family embedder + bge-reranker-v2-m3 when RAG_ENABLED). A router module dispatches by domain.
• Provider routing via LiteLLM — per-role LM specs (router / math / code / general) with DSPY_LM_<ROLE> env overrides, so any provider (or the local bitnet.cpp server) can back any role.
• A substrate-agnostic deployment playbook — the specialization layer is the constant and the model underneath is a config flip: the same DSPY_LM_<ROLE> vars point router / math / code / general at HF Inference Providers, a Modal-hosted vLLM (modal deploy scripts/modal_serve.py), a local bitnet.cpp server, or a frontier API. docs/deployment-modal-hf.md is the written playbook; scripts/modal_serve.py and scripts/modal_finetune.py ship with it, covered by tests/test_modal_scripts.py. No substrate is hosted anywhere today — this is the recipe, not a running deployment.
• MLflow tracing — autolog wired into the DSPy configuration.
• An optimization loop and an eval harness — scripts/optimize.py compiles each domain with MIPROv2/GEPA; scripts/run_eval.py runs dspy.Evaluate per domain against per-domain thresholds. A committed receipt (eval/receipts/math-miprov2.json) records a real MIPROv2 win on math: baseline 0.62 → 0.80 (+0.18) on a held-out split (50-row train / 50-row holdout per domain).
• A CI eval gate, wired but never exercised — .github/workflows/eval.yml does run on every pull request to main, and it holds the per-domain thresholds, the Wilson 95% CI score table and the PR-comment step. But the eval step is gated on an OPENAI_API_KEY secret and the repository has no secrets set, so the step has been skipped on every run that reached it and no eval has ever actually run: the job passes green without evaluating anything, and no PR has ever received the score table. branch-protection.yml holds the intended required-check configuration; it is workflow_dispatch-only, has never been run, and main currently has neither branch protection nor a ruleset, so the check is not required either. Treat this as a harness that exists, not a bar that has been enforced.
• An MCP server — app/mcp_server.py (official mcp SDK, dspy-sme-mcp console script) exposes the math/code/general experts as Model Context Protocol tools over the same DSPy programs.
• One typed FastAPI contract with the usual plumbing (Pydantic v2 schemas, SlowAPI rate limiting, CORS, Docker multi-stage build).
• A refusal guard on the math tools — the SymPy tools previously capped expression length and evaluation time, which bounded latency but not work: 9^9^9^9 is seven characters, sails past the length cap, and allocates until the process dies. app/dspy_modules/math_module.py now parses the expression with ast first and refuses unbounded exponentiation (literal exponents are capped at 5,000) before anything reaches SymPy, with the old caps kept as backstops. Shipped 2026-08-03. It closes the sympify-DoS item of the six-item audit issue #19; the issue itself is still open, and the /query auth item was explicitly out of scope for that commit.
• A local pre-push gate — lefthook.yml runs make lint on pre-commit and make lint && make test on pre-push, mirroring the CI gate on the developer’s machine. It is inert until someone runs lefthook install.

The air-gapped thesis

The defensible niche isn’t “we run BitNet” — the model is open-weight. It’s the corpus + a distillation/eval pipeline packaged to run fully offline on a CPU, for sovereignty/regulated buyers (legal, finance, defense, HIPAA) where the data never leaves the enclave. Paired with SUM (verifiable, citeable answers offline) and InfiniteContext (on-device retrieval without a cloud vector DB), it forms an offline expert stack that cloud incumbents structurally can’t ship.

What’s next (honest gaps)

• Eval receipts beyond math. The committed receipt covers the math domain (+0.18 MIPROv2 win on a 50-row held-out split); code and general have train/holdout gold sets but no committed baseline-vs-compiled win yet. Scaling the splits past 50/domain would tighten the Wilson CIs further.
• A measured BitNet receipt. scripts/bitnet_demo.sh measures real tok/s through the DSPy path and writes a dated eval/receipts/bitnet-<date>.txt, but it needs real hardware to run, so no measured figure is committed yet (the README deliberately ships none until the demo produces one).
• A published/installable artifact. No GitHub release, package, or hosted demo yet — the prerequisite for promoting this past in-development.

Technical Stack

• Framework: Python ≥3.12 (uv-managed) / FastAPI / Pydantic v2
• Optimization: DSPy 3.2 (dspy.Module / dspy.ReAct / dspy.ChainOfThought, dspy.Evaluate, MIPROv2 / GEPA)
• Provider routing: LiteLLM (per-role LM specs, env overrides)
• Tracing: MLflow autolog
• Retrieval (opt-in): LanceDB + a BGE-family embedder & bge-reranker-v2-m3 (hybrid RAG) for the general expert
• Local/offline: bitnet.cpp 1-bit server (OpenAI-compatible)
• Domain tools: SymPy (math)
• Deployment substrates (config, not hosted): HF Inference Providers, Modal-hosted vLLM, local bitnet.cpp, frontier APIs
• Ops: Docker multi-stage build, Makefile, pytest, ruff, Lefthook local pre-commit/pre-push gate