GAINS — Gesture-Assisted Intelligent Note Scribe

A multimodal note-taking system you drive hands-free: speak your notes, and a head-nod commits what you just said. Speech runs through Whisper, head pose runs through a MediaPipe Tasks pipeline, and committed text flows through a plug-in chain over a ZeroMQ bus.

What Makes It Different

Most note-taking tools optimize for keyboard input. GAINS is designed for situations where your hands are busy or a keyboard isn’t practical — standing meetings, lab work, cooking, workshops. Voice captures content; a simple nod commits it, so you never have to touch the machine.

Voice + Nod Capture

Speech is transcribed with Whisper. Head pose — the nod-to-commit signal — is tracked with MediaPipe Tasks, running as a Python vision service. A standard webcam is all that’s required.

Capture → Commit

Speech is transcribed verbatim with Whisper; a nod commits the current utterance into the note. Committed text flows through a plug-in chain — including an optional LLM grammar-fixer that preserves the original wording — before it’s written out. The note is captured text, faithfully transcribed; parsing speech into structured sections / action items is a planned direction, not yet built.

Service Architecture

A ZMQ bridge decouples the desktop shell from the Python inference services, so speech processing and head-pose detection each run as independent workers communicating over a typed message bus (gesture.nod, text.committed, …). Tests covering the ZMQ bridge and service integration live next to the services.

Technical Stack

• Shell: Tauri 2 + Leptos 0.8 desktop app (Rust/WASM UI, cross-platform)
• Services: Python microservices for speech (ASR/TTS) and head pose
• Transport: ZeroMQ plug-in bus between shell and services
• Head pose: MediaPipe Tasks (nod detection)
• Voice: faster-whisper (CTranslate2) ASR + Piper TTS; an optional LLM grammar-fixer (verbatim-preserving) in the commit chain
• Layout: services/ Python workers · tauri-app/ desktop shell

Current State

Modernized in May 2026 after an earlier pause, and still actively maintained: the Phase 1 rebuild (repo consolidation, Tauri 2 + Leptos 0.8 upgrade, MediaPipe Tasks migration, PR-CI) landed mid-May, with dependency bumps merged to main and CI passing through early June 2026. Still pre-release, though: the v0.3.0–v0.6.0 tags predate the modernization and their release-build workflow failed, so there is no published GitHub Release and no signed downloadable build a stranger can install — the voice-and-nod pipeline runs from source. Read the public surface as an active work-in-progress, not a shipping product.