GAINS — Gesture-Assisted Intelligent Note Scribe

A multimodal note-taking system you drive hands-free: speak your notes, and a head-nod commits what you just said. Speech runs through Whisper, head pose runs through a MediaPipe Tasks pipeline, and committed text flows through a plug-in chain over a ZeroMQ bus.

What Makes It Different

Most note-taking tools optimize for keyboard input. GAINS is designed for situations where your hands are busy or a keyboard isn’t practical — standing meetings, lab work, cooking, workshops. Voice captures content; a simple nod commits it, so you never have to touch the machine.

Voice + Nod Capture

Speech is transcribed with Whisper. Head pose — the nod-to-commit signal — is tracked with MediaPipe Tasks, running as a Python vision service. A standard webcam is all that’s required.

Capture → Commit

Speech is transcribed verbatim with Whisper; a nod commits the current utterance into the note. Committed text flows through a plug-in chain — including an optional LLM grammar-fixer that rewrites the wording (it is prompted to keep the meaning identical, and the raw transcript is kept alongside the rewrite in the JSON export, though the .txt and .md exports carry only the rewrite) — before it’s written out. That grammar plugin is the one component here that calls a cloud API (OpenAI); it is inactive unless OPENAI_API_KEY is set. The note is captured text, faithfully transcribed; parsing speech into structured sections / action items is a planned direction, not yet built.

Service Architecture

A ZMQ bridge decouples the desktop shell from the Python inference services, so speech processing and head-pose detection each run as independent workers communicating over a typed message bus (gesture.nod, text.committed, …). A pytest suite in tests/ covers plug-in discovery, ASR model-name resolution and the head-pose pitch extractor, plus a ZeroMQ XSUB/XPUB round-trip that re-implements the proxy rather than importing the shipped hub (the hub itself is import-checked only); the camera and audio services are parse-checked rather than run in CI.

Technical Stack

• Shell: Tauri 2 + Leptos 0.8 desktop app (Rust/WASM UI, cross-platform)
• Services: Python microservices for speech (ASR/TTS) and head pose
• Transport: ZeroMQ plug-in bus between shell and services
• Head pose: MediaPipe Tasks (nod detection)
• Voice: faster-whisper (CTranslate2) ASR + Piper TTS; an optional LLM grammar-fixer in the commit chain (rewrites wording, prompted to keep meaning identical; calls the OpenAI API, off unless OPENAI_API_KEY is set)
• Layout: services/ Python workers · tauri-app/ desktop shell

Current State

Modernized in May 2026 after an earlier pause, and still actively maintained: the Phase 1 rebuild (repo consolidation, Tauri 2 + Leptos 0.8 upgrade, MediaPipe Tasks migration, PR-CI) landed mid-May, with dependency bumps merged to main and CI green on main as of 2026-08-03 (a RUSTSEC-advisory break that had been red on main since 2026-07-13 was cleared in commit 6e5b7a1). Still pre-release, though: the v0.3.0–v0.6.0 tags predate the modernization and their release-build workflow failed, so there is no published GitHub Release and no signed downloadable build a stranger can install — the voice-and-nod pipeline runs from source. Read the public surface as an active work-in-progress, not a shipping product.