The rise of batteryless energy-harvesting platforms could enable ultra-low-power, long-term, maintenance-free deployments of sensors.
This project explores the deployment of the OpenAI Whisper audio transcription model onto embedded devices, initially starting with the rPi and moving onto smaller devices.
1 Weeknotes
Dan Kvit ran this over the summer of 2025 with Josh Millar, and kept weeknotes as he went. The code is on GitHub. The target is a meeting recorder that transcribes on the device rather than on a server, so the model has to keep up with speech on a Raspberry Pi 4 with 4GB of memory.
- Week 1, 18 July. Read the Whisper paper and ran whisper.cpp on a Mac and then on the Pi, where 11 seconds of audio took 18 seconds to transcribe. Also covers getting the Pi onto a switch and onto eduroam. Kyutai is the faster architecture for streaming, but it wants 4.2GB and the board has 4GB.
- Week 2, part 1, 23 July. Swapping to the microSD card proved Kyutai needs only 1.6GB more than the board has, but the card is far too slow to run from. Back on whisper.cpp, timing every model and every quantisation: base-q4_k takes the 11 second sample from 18.1s to 11.9s. The q2_k and q3_k quantisations come out slower rather than faster, because they are tuned for AVX2 and AVX512 and the Pi is ARM.
- Week 2, part 2, 29 July. Voxtral at 3B parameters is out of reach when Kyutai at 1B already costs ten seconds of compute per second of audio. Rather than try models one at a time, the search moved to the open ASR leaderboard for anything under 1B. The NVIDIA entries all need an NVIDIA GPU, which leaves Moonshine.
- Week 3, 30 July. Moonshine base does the 11 second sample in 2.76s against 21s for whisper.cpp base, at a slightly lower word error rate. Attached a USB microphone and got live captions running on the Pi. SPGISpeech, Earnings-22 and AMI were picked for evaluation, since the use is meetings.
- Week 4, 8 August. Laid out the repository, one directory per model behind a shared entry point for transcribing a file and for transcribing from the microphone.
- Week 5. Swapped the Earnings-22 dataset for the distil-whisper copy, the original having no dataset card and missing data.
- Notes on Kyutai, 18 August. Writes up the goal and where Kyutai stands against it. On the Pi under swap, the first word arrives 5 minutes 51 seconds in, most of that being memory access rather than computation.
- Week 6, 27 August. Why Whisper is the wrong model for streaming. Its encoder takes a fixed 30 second input, so a 2 second chunk is padded with 28 seconds of zeroes and costs the same to run. Moonshine uses rotary position embeddings and accepts whatever length it is given. Added Vosk, and a table of real-time factor alongside the latency a user actually perceives, which is the figure that decides whether a model is usable live.
- Week 7-8, 8 September. Recognising spoken commands, matching "start recording" and "stop recording" against the live transcript to control the file. Ported the moonshine.cpp example off Windows. Real-time factor turns out to vary with the length of the sample, so a single number for a model misleads.