Building a Local Voice Input Tool Just to Talk to AI

[!TIP] Audience: Windows users and indie developers who talk to AI all day, find typing too slow, and don’t want their audio uploaded to the cloud. Goal: Walk through why I picked this stack, what the real latency numbers look like, and the engineering traps I hit getting audio from the mic onto the screen. Problem → Fix:

• GPU is maxed out by games and can’t run big models → Pick SenseVoice (sherpa-onnx) on pure CPU; 5.6s of audio takes 0.29s
• Simulated keystrokes can’t type Chinese, and overwriting the clipboard wipes out screenshots → Paste via clipboard, but back up and restore the entire clipboard by format
• Hard crashes on model load, miscalculated latency, and build scripts deleting user data → Measure everything first, and automate steps that used to rely on human memory

Why Talk to AI Instead of Typing

I build small tools for myself, and AI does most of the heavy lifting. Every day I spend hours typing prompts into Claude Code. Once a task gets slightly complex—with background context and constraints—typing it all out turns into physical labor.

So I built VoiceInput. It works simply: hold down a hotkey to talk, let go, and the text shows up right at your cursor. Recognition runs entirely on your machine, and no audio ever leaves it. Right now, it’s Windows-only.

I built this strictly to replace typing when talking to AI. Text landing in the Claude Code input box looks no different from manual typing, and it’s up to the AI to decide what to do next—this tool stays out of it. That’s also why I skipped all the “voice command to control your PC” stuff. Your voice just gets turned into plain text, nothing more.

In daily use, it comes down to two default hotkeys:

• Hold Ctrl+Alt+Space to talk: Let go, and the text is pasted right at the cursor.
• Hold Ctrl+Alt+Enter to talk: Let go to paste the text and automatically hit Enter—handy when you want to send a message right after speaking.

Both can be changed in settings. If another program has already grabbed one, the self-test tells you.

The app is fully portable. Configs, history, and logs live right next to the executable, and nothing touches the registry. The only exception is if you turn on “Launch at startup”, which drops a shortcut into the Windows Startup folder.

Choosing the Engine: Why SenseVoice

Before building my own, I looked at an existing open-source voice input app, but it didn’t work for me: its local engine options didn’t include SenseVoice, and the other model it offered didn’t support Chinese.

As for why I picked SenseVoice (via sherpa-onnx) over the much more famous faster-whisper, the decision came down to measured numbers. My GPU is routinely maxed out running 4 game instances at once, so a voice tool has to run on pure CPU without touching the GPU. I tested both on pure CPU using the exact same 5.6-second audio clip:

Beyond the engine, two other implementation choices mattered:

• Why paste via clipboard instead of simulating keystrokes: Another project of mine already proved that character-by-character simulated keystrokes on Windows can’t type Chinese. So this time I write to the clipboard, simulate Ctrl+V, and restore the previous clipboard contents once pasting finishes.
• Pinning the model version to 2024-07-17: I nearly fell into a huge trap downloading the model. On the same release page, there’s a newer 2025-09-09 build, and both files are the exact same size at 226MB. However, the 2025-09-09 build is actually fine-tuned specifically for Cantonese—swapping it in completely broke Mandarin, English, Japanese, and Korean. You have to stick to the 2024-07-17 version.

Pitfalls: “No Error” Doesn’t Mean “Verified”

Over the course of this project, my log of pitfalls grew to 43 entries. Looking back, one lesson runs through all of them: just because it didn’t throw an error doesn’t mean it works. Here are 5 classic examples where I assumed A, only for real testing to slap me in the face with B.

1. Mistaking Inference Time for User-Perceived Latency

For a long time, all my docs claimed latency was “0.29 seconds”, and I thought it was blazing fast. Yet after more than a dozen iterations, I had never once measured the actual end-to-end time.

When I finally added instrumentation and measured from the moment my finger left the hotkey to the moment text appeared on screen, the total time was actually 380 ms. Breaking down that 380 ms:

• Model inference: 79% (that ~0.29s part).
• Pre-paste wait: 21%.
• Text post-processing + clipboard ops: Under 1 ms combined.

That didn’t change my engine choice, but throwing around “0.29 seconds” underestimated real latency by about 30%. Once you include that wait time in the end-to-end calculation, the real-world gap against faster-whisper is 99x, not the 116x you get from looking at inference alone.

2. “Restoring the Clipboard” Only Worked for Plain Text

As mentioned earlier, text goes onto the screen via “write to clipboard → paste → restore original clipboard”. At first, I figured reading out whatever was in the clipboard, saving it, and putting it back afterward was enough. The code ran without errors.

Later, when I went hunting for untested assumptions, I realized this logic only held up for plain text—my backup only saved text, and my tests only used plain text. Any other format silently lost data:

• Screenshots in the clipboard: Hold the hotkey, say a sentence, and your screenshot is gone. When chatting with AI, this is fatal—you take a screenshot of an error to show the AI, speak a quick description, and the image vanishes.
• Copied files in the clipboard: Say a sentence, and the copied file degrades into a plain file:/// path string.
• Rich text in the clipboard: After pasting and restoring, all rich text formatting is stripped away.

I changed it to snapshot and restore the entire clipboard across formats, handling images through a dedicated image API. While fixing this, I also found out that blocking paste calls couldn’t restore rich content on a real Windows desktop; switching to async fixed it. Unit test clipboards bypass the OS pipeline, so they never caught the difference.

3. An Old System32 DLL Hijacked Model Loading and Hard-Crashed

At one point, whenever the app tried to load the model, it hard-exited right inside the C++ layer without a word. Python couldn’t catch any exception at all.

I guessed wrong twice while debugging:

1. First I thought the model version was too new for the runtime. I manually parsed the model header, saw the version was quite old, and ruled that out.
2. Next I blamed a conflict with another package installed in the environment. I uninstalled that package and tested again—same crash, ruled out again.

Eventually I tracked it down: Windows had an old onnxruntime.dll sitting in System32, and the process loaded that one first at startup. The fix was simple—install the package that bundles its own matching DLL so the app loads the library from its own directory first, and the crash vanished on the spot.

4. Half the Microphones in the Dropdown Were Broken

When building the audio settings page, I dumped every input device enumerated by the OS straight into the dropdown. I assumed users could pick any of them and start recording. Actual testing showed that selecting half of those options failed to open the audio stream at all.

The culprit was WASAPI. Half the enumerated list consisted of WASAPI devices, and WASAPI shared mode is super rigid: it only accepts the device’s native sample rate (48000Hz on my machine). Meanwhile, my app hardcoded the recording sample rate to 16000Hz to feed the model. Because the rates didn’t match, picking any WASAPI device immediately failed to record.

On a related note, to keep users from picking a dead or wrong device, I added a rule to the mic calibration wizard: you have to record a real sentence, run it through recognition, and pass a word error rate check—never judge by volume meters alone. The system default input device is often “stereo mix / loopback”, which shows a bouncing volume meter whenever your PC plays music. Looking at a level meter alone won’t catch that. If a user picks it by mistake, pressing the hotkey does nothing, and they’ll just assume the app is broken.

5. The Build Script Deleted My Own Config and History

The most painful bug happened on 2026-08-26. I was repackaging the app, and when the build script hit the “clean old artifacts” step, it casually wiped out my personal config.toml and history.db right along with them—gone for good.

This happened because of the portable app layout: I dogfood the app myself, so my personal data files lived in the exact same folder as the build output. Before every build, I used to rely on my own memory to “manually back up config and history first”. Experience proves that any engineering step relying on human memory might as well not exist.

After getting burned, I changed the build script immediately: the script now backs up user data itself before cleaning and restores it after packaging, keeping the backup strictly outside the artifact directory. I don’t trust my memory anymore.

(Hotkeys had a similarly annoying trap: the original default Ctrl+Win gets triggered by any superset combination. Switching Windows virtual desktops with Ctrl+Win+Left/Right, or pressing Ctrl+Win+X for another app, would falsely trigger recording. I noticed it “conflicting with WeChat” in daily use, which turned out to be this exact superset issue. Worse yet, an early test I wrote actually locked in this bug as expected behavior, until I rewrote the test to assert that such conflicts must be reported. The default is now Ctrl+Alt+Space.)

What’s Not Done Yet & Limitations

Right now this tool covers my daily chatting with Claude Code, but a few things are still unfinished, so I’ll lay the caveats out up front:

• Engine accuracy hasn’t been benchmarked on real human speech: Everything I know about its accuracy comes from the test audio bundled with the model and synthetic speech. I haven’t systematically measured word error rates on real recordings of my own voice yet. If real speech turns out too error-prone, I may need to hook up a cloud API fallback or add a local LLM pass for error correction.
• Silence hallucinations are blocked by a front-end filter: In testing, feeding pure silence into the model causes it to hallucinate and randomly spit out things like a single Korean character. Right now I work around this by running silence detection first and dropping silent audio before it ever reaches transcription.
• Platform lock-in: As mentioned, everything from hotkeys and clipboard handling to audio APIs is wired specifically for Windows. It only runs on Windows.

Download

Grab the portable build on the Downloads page (about 207 MB, model included, so you just unzip and run—no extra download). The source is on GitHub under the MIT license: https://github.com/yoyoset/voiceinput