CapsLockX 2.0: Rust rewrite with voice, brainstorm, TTS

[snomiao]

Summary

• Ground-up Rust rewrite targeting cross-platform (macOS first)
• SenseVoice local STT (95%+ accuracy) + Gemini cloud STT + MLX local LLM correction
• Brainstorm agent (Space+B) with 4 LLM providers, 10+ tools, persistent chat history
• TTS with 5-tier fallback chain (ElevenLabs → Gemini → OpenAI → msedge → native)
• Sandboxed JS engine (rquickjs) + Wolfram math engine (Woxi)
• Voice overlay with waveform, subtitles, drag/resize/close
• Window cycling via CGWindowID (stable across arrange/minimize)
• Mouse clamp to screen bounds (multi-monitor safe)
• Launch at login via LaunchAgent
• Browser WASM adapter with SenseVoice voice input

Test plan

• Build: cd rs && cargo build -p capslockx-macos --release
• Voice input: Space+V hold to record, release to transcribe
• Brainstorm: Space+B with selected text, Enter to send
• Window cycling: Space+Z forward, Space+C arrange
• Mouse: Space+WASD movement clamped to screen edges
• TTS: Agent auto-speaks translations

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR introduces CapsLockX 2.0 as a Rust rewrite (macOS-first) and adds major AI/voice features (STT, TTS, brainstorm agent), plus new platform adapters, tooling, docs, and CI/release support.

Changes:

• Adds new Rust core capabilities: voice pipeline (local+cloud STT, correction), TTS fallback chain, agent/tools, background task manager, and audio capture.
• Adds macOS adapter implementation (CGEventTap hook, tray + prefs UI, voice overlay/capture/system audio).
• Updates packaging/CI: npm launcher downloads per-platform binaries; builds macOS in CI/release; expands documentation and plans.

Reviewed changes

Copilot reviewed 72 out of 93 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tmp/capslockx-mac	Adds submodule pointer for macOS adapter snapshot
tmp/brainstorm	Adds submodule pointer for brainstorm snapshot
rs/test-results.txt	Adds recorded manual test results artifact
rs/test-manual.ahk	Adds manual QA helper for Shift+HJKL selection
rs/dev-watch.sh	Adds cargo-watch script for auto rebuild/restart
rs/core/src/tts.rs	Introduces multi-tier TTS fallback implementation
rs/core/src/task_manager.rs	Adds background task manager with timeouts
rs/core/src/stt_corrector.rs	Adds incremental LLM-based STT correction
rs/core/src/state.rs	Extends config with STT/brainstorm/LLM settings
rs/core/src/platform.rs	Extends platform trait (audio, text input, mods)
rs/core/src/modules/voice_player.html	Adds HTML utility to play voice notes with subtitles
rs/core/src/modules/mouse.rs	Updates mouse/scroll phases & adds shift-scroll behavior
rs/core/src/modules/mod.rs	Adds brainstorm + voice modules and wiring
rs/core/src/modules/edit.rs	Adds atomic modifier-aware key tapping for macOS
rs/core/src/local_whisper.rs	Adds local whisper.cpp wrapper with autoscaling
rs/core/src/local_sherpa.rs	Adds local SenseVoice wrapper with auto-download
rs/core/src/lib.rs	Exposes new core modules (agent, STT, TTS, etc.)
rs/core/src/engine.rs	Improves trigger bypass logic + held-key syncing
rs/core/src/cloud_stt.rs	Adds Gemini cloud STT transcription helper
rs/core/src/bin/test-llm.rs	Adds quick LLM client test binary
rs/core/src/bin/test-agent.rs	Adds agent tool test runner binary
rs/core/src/bin/stt-server.rs	Adds persistent local STT server binary
rs/core/src/bin/stt-quick.rs	Adds one-shot STT helper binary
rs/core/src/bin/stt-compare.rs	Adds local vs cloud STT benchmark/compare tool
rs/core/src/bin/stt-bench.rs	Adds SenseVoice vs Whisper benchmark with WER/CER
rs/core/src/bin/sherpa-test.rs	Adds standalone mic capture + SenseVoice test binary
rs/core/src/bin/clx-agent.rs	Adds standalone CLI agent chat tool
rs/core/src/audio_capture.rs	Adds cross-platform mic capture via cpal
rs/core/src/acc_model.rs	Adds option to drive ticks externally (hook thread)
rs/core/Cargo.toml	Adds dependencies for audio/STT/TTS/JS/math engines
rs/adapters/windows/src/output.rs	Updates close_tab to cmd-or-ctrl helper
rs/adapters/windows/src/hook.rs	Drives AccModel ticks via SetTimer on hook thread
rs/adapters/macos/src/tray.rs	Adds macOS tray icon via ObjC FFI
rs/adapters/macos/src/prefs_html.html	Adds macOS preferences UI (WKWebView content)
rs/adapters/macos/src/main.rs	Adds macOS adapter entry point
rs/adapters/macos/src/key_map.rs	Adds macOS keycode mapping
rs/adapters/macos/src/hook.rs	Adds CGEventTap hook w/ pass-through/suppress logic
rs/adapters/macos/src/config_store.rs	Adds persistent config store for macOS
rs/adapters/macos/src/bin/voice-standalone.rs	Adds standalone voice pipeline binary
rs/adapters/macos/src/bin/test-vpio.rs	Adds VoiceProcessingIO AEC test binary
rs/adapters/macos/src/bin/test-cycle.rs	Adds direct window cycling stability test
rs/adapters/macos/build.rs	Links required macOS frameworks
rs/adapters/macos/Cargo.toml	Adds macOS adapter crate
rs/adapters/linux/src/output.rs	Updates close_tab to cmd-or-ctrl helper
rs/adapters/browser/www/vite.config.js	Adds dev server config for browser adapter
rs/adapters/browser/www/sherpa/sherpa-onnx-vad.js	Adds sherpa VAD JS helper
rs/adapters/browser/www/sherpa/.gitignore	Ignores wasm/data model artifacts
rs/adapters/browser/www/package.json	Adds browser adapter web package config
rs/adapters/browser/www/.gitignore	Ignores node_modules
rs/adapters/browser/src/platform.rs	Updates close_tab to cmd-or-ctrl helper
rs/Cargo.toml	Adds macOS adapter to workspace
plan/voice-input/voice-modes.md	Adds voice modes design notes
plan/voice-input/TODO.md	Adds voice implementation plan checklist
plan/voice-input/README.md	Adds voice feature spec
package.json	Switches npm bin to node launcher & includes new artifacts
bin/capslockx.mjs	Adds cross-platform launcher/downloader script
TODO.md	Adds voice feature plan reference + macOS drag issue note
.playwright-cli/page-2026-03-18T17-55-02-733Z.yml	Adds Playwright snapshot artifact
.playwright-cli/page-2026-03-18T17-53-48-281Z.yml	Adds Playwright snapshot artifact
.playwright-cli/page-2026-03-18T17-51-22-570Z.yml	Adds Playwright snapshot artifact
.playwright-cli/page-2026-03-18T17-50-03-292Z.yml	Adds Playwright snapshot artifact
.playwright-cli/page-2026-03-18T17-49-54-727Z.yml	Adds Playwright snapshot artifact
.playwright-cli/page-2026-03-18T17-49-44-295Z.yml	Adds Playwright snapshot artifact
.github/workflows/release-rust.yml	Adds macOS release build + alters artifact matching behavior
.github/workflows/ci-rust.yml	Adds macOS CI build/test/clippy
docs/dev/window-cycle-stability.md	Documents window cycling stability fixes
docs/dev/modifier-bypass.md	Documents modifier+Space bypass design
docs/dev/dual-track-stt.md	Documents dual-track STT architecture
docs/dev/agent-test-matrix.md	Documents agent tool test matrix
docs/Roadmap.md	Rewrites roadmap for Rust v2.0 scope/plans

Files not reviewed (1)

• rs/adapters/browser/www/package-lock.json: Language not supported

Comments suppressed due to low confidence (15)

rs/core/src/modules/mouse.rs:1

• Horizontal scrolling is applied multiple times: dx scrolls inside both branches and then again unconditionally on line 139. This will double-scroll whenever dx != 0 (and also double-scroll in Shift mode). Remove the unconditional if dx != 0 { p.scroll_h(dx * 3); } or restructure so each axis is emitted exactly once per call.
  rs/core/src/platform.rs:1
• The implementation always uses LCtrl, but the docstring says it should use Cmd on macOS. This will break common macOS shortcuts (e.g., Cmd+W for close tab). Make the modifier conditional on target_os (use KeyCode::LWin on macOS; KeyCode::LCtrl elsewhere) or provide a platform override for macOS and keep the default consistent with its behavior.
  rs/core/src/task_manager.rs:1
• task_kill sets a flag and updates status, but the running task never observes kill, so nothing is actually stopped. To make kill functional, run tasks cooperatively by passing the Arc<AtomicBool> (or a cancellation token/channel) into func, and ensure the task checks it periodically and exits early when set; alternatively, remove the public kill API and clearly indicate tasks are non-cancellable.
  rs/core/src/task_manager.rs:1
• task_kill sets a flag and updates status, but the running task never observes kill, so nothing is actually stopped. To make kill functional, run tasks cooperatively by passing the Arc<AtomicBool> (or a cancellation token/channel) into func, and ensure the task checks it periodically and exits early when set; alternatively, remove the public kill API and clearly indicate tasks are non-cancellable.
  rs/adapters/browser/www/vite.config.js:1
• The dev server binds to all interfaces (0.0.0.0) and requires TLS keys from hard-coded /tmp paths. This is easy to misconfigure and can unintentionally expose a dev server on a LAN. Prefer defaulting host to 127.0.0.1 and sourcing key/cert paths (or HTTPS enablement) from environment variables with safe fallbacks.
  rs/core/src/local_whisper.rs:1
• The module-level docs say upgrade happens when inference is >1.25x realtime, but UPGRADE_THRESHOLD is 5.0 (and the code uses a 'budget ratio' derived from wall-available time). Update the documentation to match the actual scaling heuristic to avoid misleading tuning/expectations.
  rs/core/src/local_whisper.rs:1
• The module-level docs say upgrade happens when inference is >1.25x realtime, but UPGRADE_THRESHOLD is 5.0 (and the code uses a 'budget ratio' derived from wall-available time). Update the documentation to match the actual scaling heuristic to avoid misleading tuning/expectations.
  rs/core/src/tts.rs:1
• Invalid base64 characters are silently treated as 0 via unwrap_or(0), which can corrupt audio output without surfacing an error. Return an error when a character is not found in the alphabet (except = padding), or use a well-tested base64 crate to decode and validate input.
  rs/core/src/tts.rs:1
• Audio is written to fixed filenames in /tmp. Concurrent or overlapping speak() calls can race and overwrite each other's output, producing wrong audio playback. Use unique temp paths (e.g., include PID + timestamp/random suffix) and consider deleting the temp file after playback completes.
  rs/core/src/tts.rs:1
• Audio is written to fixed filenames in /tmp. Concurrent or overlapping speak() calls can race and overwrite each other's output, producing wrong audio playback. Use unique temp paths (e.g., include PID + timestamp/random suffix) and consider deleting the temp file after playback completes.
  rs/core/src/platform.rs:1
• The default type_text silently drops many common characters (e.g., '-', '=', ',', '/', ';', '''), which can corrupt typed STT output or user prompts. Either extend the mapping to cover a complete US-ANSI punctuation set, or change the fallback behavior to something lossless (e.g., clipboard paste) when encountering an unmapped character.
  rs/core/src/platform.rs:1
• The default type_text silently drops many common characters (e.g., '-', '=', ',', '/', ';', '''), which can corrupt typed STT output or user prompts. Either extend the mapping to cover a complete US-ANSI punctuation set, or change the fallback behavior to something lossless (e.g., clipboard paste) when encountering an unmapped character.
  rs/core/src/platform.rs:1
• The default type_text silently drops many common characters (e.g., '-', '=', ',', '/', ';', '''), which can corrupt typed STT output or user prompts. Either extend the mapping to cover a complete US-ANSI punctuation set, or change the fallback behavior to something lossless (e.g., clipboard paste) when encountering an unmapped character.
  rs/dev-watch.sh:1
• pkill -f \"target/release/capslockx\" can match and kill unrelated processes whose command lines happen to contain that substring. Consider using a PID file, pkill -x with an exact process name when possible, or filtering by the full path to the spawned binary to avoid collateral termination.
  rs/test-results.txt:1
• The PR description marks the test plan items as completed, but the committed rs/test-results.txt shows failing cases (5/8 passed). If this file is meant as an authoritative test artifact, it should reflect a passing run (or be excluded from the PR) to avoid conflicting signals about readiness.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pushing release-built binaries back to main from a release workflow is risky (mutates default branch, can create unexpected CI loops, merge conflicts, and provenance issues). Prefer attaching artifacts only to GitHub Releases and publishing npm packages from the release artifacts (or a dedicated distribution branch), rather than committing binaries into source control.

Suggested change

	# Commit binary to main so npm package includes it.
	- name: Commit macOS binary to main
	run: |
	git config user.name "github-actions[bot]"
	git config user.email "github-actions[bot]@users.noreply.github.com"
	git pull origin main --rebase
	git add capslockx-macos-arm64
	git diff --cached --quiet || git commit -m "chore: update macOS binary for ${{ github.ref_name }}"
	git push origin main