Local voice: from Whisper to text-to-speech
Why voice is the next step
Local AI has already proven itself for text: chat, summaries, translations. But voice is the channel we actually live in — meetings, dictation, assistance, accessibility. Bringing voice to local means controlling the entire loop: from microphone to response, without a single byte leaving your hardware.
This guide shows how to build a fully local voice pipeline: speech-to-text with Whisper, text-to-speech with Piper or Kokoro-82M, and the pieces that connect them.
From audio to text: Whisper and its implementations
Whisper by OpenAI is the reference model for automatic transcription. Trained on over 680,000 hours of audio, it supports 99 languages and offers sizes ranging from the tiny model (39M parameters) to large-v3 (1.55B). The large-v3-turbo model (809M) offers a good trade-off between quality and speed.
To run it locally with acceptable performance, the most widely used implementations are:
- faster-whisper — a reimplementation based on CTranslate2, up to 4 times faster than openai/whisper with the same accuracy and less memory. Supports 8-bit quantization on both CPU and GPU.
- whisper.cpp — C/C++ version, ideal for ARM (Raspberry Pi, Apple Silicon) and CPU. Works without a GPU.
The Word Error Rate (WER) is the standard metric for comparing transcriptions: a low WER means few errors per word. For English, the large models remain the most accurate; for everyday use on modest hardware, medium or small with 8-bit quantization are already very useful.
From text to voice: Piper and Kokoro-82M
Two local voices stand out for quality and ease of use:
Piper — neural-based TTS built on VITS, with pre-trained models in multiple languages (including English). Lightweight: runs on CPU, even on Raspberry Pi 4, with real-time latency. Ideal for quick responses, voice notifications, voice assistants.
Kokoro-82M — 82 million parameters, VITS architecture, 80 supported languages, Apache-2.0 license for the models. Voice quality is surprisingly high for its size; the English voice af_heart is among the most natural in the catalog. It runs efficiently on CPU, making it an excellent option for those who want a local voice without a dedicated GPU.
Building the pipeline
A minimal local voice pipeline has three stages:
- Capture — microphone or audio file (WAV, MP3, OGG).
- Transcription (STT) — Whisper (faster-whisper or whisper.cpp) → text.
- Synthesis (TTS) — Piper or Kokoro → response audio.
Between the stages you can insert a local LLM (Ollama with Qwen, Llama, Mistral) to generate the text response before synthesis. The result is a complete voice assistant that works without a connection.
Quick example with faster-whisper and Piper
Transcription:
from faster_whisper import WhisperModel
model = WhisperModel("large-v3-turbo", compute_type="int8")
segments, info = model.transcribe("meeting.wav", language="en")
text = " ".join(s.text for s in segments)
print(text)
Synthesis:
echo "Summary: the meeting covered three points." | piper --model en_US-medium --output_file reply.wav
Hardware requirements
| Component | Minimum (STT+TTS CPU) | Recommended (STT+TTS GPU) |
|---|---|---|
| CPU | 4 cores (x86_64 or ARM) | 8+ cores |
| RAM | 8 GB | 16 GB |
| GPU | not required | NVIDIA 8+ GB VRAM (or Apple Silicon M1/M2) |
| Disk | ~2 GB (models) | ~4 GB |
With large-v3-turbo quantized int8 on CPU, transcription runs in real time on recent machines. Kokoro-82M on CPU has a latency of a few seconds for short phrases.
Practical use cases
- Dictation — speak, and the text lands in your notes or email app.
- Meetings — real-time transcription plus a summary from a local LLM.
- Accessibility — voice synthesis for documents, emails, notifications.
- Personal assistant — full pipeline with LLM: voice question, voice answer, all local.
Products like TalkScribe show that the entire loop (Whisper plus a local Qwen LLM on M-series GPU) can live in a native macOS app, no cloud required.
Where to go next
Once the basic pipeline works, the natural next steps are:
- Punctuation and capitalization — Whisper handles these internally; for real-time flows, consider faster-whisper with beam size 1.
- Voice activity detection (VAD) — to segment audio when someone speaks, without streaming a continuous feed.
- Fine-tuning — if your voice or accent produces high WER, a small dataset (10-30 minutes) can improve the model.
Voice is the point where local AI stops being an experiment and becomes an assistant that lives with you, on your hardware, always available.
Article generated with AI
Enjoyed this article?
Subscribe to the newsletter for new tutorials and guides.