Making a dictation app in a weekend
My friend Geoffrey Litt has been tweeting about how useful it is to dictate — literally speak out loud — to LLMs when coding:
The key is that you can get very detailed with your prompt since voice makes it so fast. Just say everything you’d say to a junior, be verbose.
If you’re doing it right, it should feel like your prompts are absurdly long. Like, I’ll sometimes ramble for minutes. You should rarely stop to re-record – if you make a mistake just keep going and correct yourself. Feel free to think aloud on the fly
Even though I can type okay — 136 WPM w/ 99% accuracy, according to MonkeyType — I’m curious about how it’d feel to speak to the computer. (I also suspect it’d be good to practice speaking clearly and off-the-cuff about technical stuff.)
Unfortunately, the app Geoffrey uses (SuperWhisper) and the handful of other ones I found all require a newer version of MacOS than the one I was running.
Since it’s out of the question to either:
- upgrade MacOS (it’ll randomly inject balloons and emoji thumbs into my camera feed!), or
- use Mac’s built-in dictation functionality (OK quality; pauses my music)
I made my own lil’ app instead: Whispertron.
My two primary requirements were:
- support my computer (MacBook Air M1, MacOS 12.7)
- global hotkey pseudomode
I never touch XCode and barely know anything about Swift, so I had Claude write most of the code via Cursor. (Claude also helped me navigate the byzantine XCode nested menus and incantations required to allow an app to access the microphone.)
The functionality itself pretty straightforward:
- There’s a menu bar item, so you can see if the app is running and quit if necessary.
- Pressing the global hotkey:
- starts recording from the default microphone into an in-memory audio buffer
- shows a little window that says “recording” near the current mouse cursor (This is my way of making a simple status notification, as my computer has “Do Not Disturb” set from 2:01 a.m. until 2:00 a.m.)
- Releasing the global hotkey then:
- downsamples the audio to 16 kHz mono
- passes it to the Whisper model (via the incredible whisper.cpp) for transcription
- injects the text into the currently focused app by synthesizing keypresses (originally Claude tried to use some accessibility APIs to insert text, but that only worked for some system controls — I figured there must be some kind of open source “text expander” app which had a solution that’d work for any app, and I found this C++ code which I had Claude translate into Swift and worked great)
- If the global hotkey was held down only briefly, instead the app copies the last transcription to the clipboard. (I figured it’d be handy to have a way to recover your transcript in case you talked while your cursor was in the wrong place.)
That’s it, that’s the whole app.
I’m extremely chuffed to have gotten it done in one weekend. According to my notes it took about 9 hours across two sessions, though that’s a bit of an overestimate as a few hours were spent noodling with word-by-word “live” transcription before I gave up and just did single shot (which is what the Whisper model is designed for).
I find it quite liberating and enjoyable to have LLMs generate code for a language/ecosystem that I don’t have any preconceived notions about. I just don’t know (or care!) about the “proper” way to make a MacOS app, how to structure my AppDelegate, organize code between methods vs. free functions, etc. Especially once I got the basic app itself working, I could just speak out loud with the LLM to request lots of boring code tweaks (“Hey thanks looks good, but can you give the popup window an opaque background so it pops out a bit more?”)
Anyway, feel free to check out the app (you’ll have to build it yourself). If you have experience writing MacOS apps properly (or just want to have a go at managing an LLM on a toy project), I’m open to pairing and/or reviewing pull requests that:
- add a single preferences window that allows you to configure the global activation hotkey
- make it easier for other people to build (is it possible without XCode?)
- graphically select and download Whisper models besides the base English one
- implement live transcription with a prinicpled approah like Whisper Streaming (I assume it’s good based on the video demos, but couldn’t run it myself because, lol, Python dependencies)
An art wall
LLMs saved me so much time making my goofy app last weekend that I still had enough time to make a lil’ art wall behind my desk!
There’s no big story here, I’m just happy with how it turned out =D
As it so happens, putting random postcards, Dutch illustrated animal books, and miscellaneous pandemic-hobby-project keyboards and flexures into assorted frames from Ikea goes a long way.
There’s something magical about starting a new project quickly and keeping momentum through completion — manifesting in this case as refusing to leave my house to buy “proper” supplies and instead cobbling together the knolled keyboards by sticky-tacking them onto the Ikea frame’s acrylic sheet, which I frosted with leftover sandpaper and placed over white fusable interfacing ironed onto white cardboard.
Next I’d like to design some kind of nicer-looking storage unit for that pile of transparent boxes underneath the shelf. I’ve got several square meters of felt and a dozen wood rounds leftover from my desk build, maybe there’s something I can do with 3d-printed tube connectors…
Misc. stuff
Dark winter days are coming, so I’m thinking of buying/making a full-spectrum SAD lamp; this guide looks reasonable, but please write if you have experience or alternative suggestions.
I recently had to measure phase offsets from a noisy signal sampled with a cheap ADC, so figured it was time to properly learn about signal analysis. Of the many textbooks I perused, the one I picked to read was Understanding Digital Signal Processing by Richard Lyons, which has a casual friendly style and is clearly written by an experienced engineer — the final chapter is simply 150 pages of “Digital Signal Processing Tricks”!
How electric trains work and why they make interesting sounds
How the US Lost the Solar Power Race to China. The world’s largest polysilicon manufacturer pivoted to that industry after a slowdown in fish-farming.
“Lincoln Lab researchers appear to have invented a viable new class of motor in 2021??? They use electrowetting of tiny water drops which sounds like high voltage low force bullshit, but no! Competitive power and torque densities to conventional motors! WTF??”
Tagged Union Subsets with Comptime in Zig – Mitchell Hashimoto
SQLite’s Code Of Ethics and an interview with Dr. Hipp after Online People Got Mad: “I could have edited the list down to just those aspects that seem relevant to coding […] But that would put me in the position of editing and redacting Benedict of Nursia, as if I were wiser than he.”
The forklift sport event of the year, Oct 25–26 outside of Frankfurt.
A Trail Gone Cold: Against the odds, a tiny Icelandic town speaks of a local Black ancestor. Geneticists and historians combine forces to uncover the man’s eventful life.
The $1,000 Wheelchair: How the YouTubers from JerryRigEverything are Making Affordable Wheelchairs Without the Red Tape
Recreating 2024 Best Picture Nominees on a Budget | (Barbie, The Holdovers, etc…)
“Here is another way to understand the contraptiony nature of helicopters: They have no narratively compelling line of descent from nature.”
Amsterdam is Dead. Easily the best surreal Dutch parkour film I’ve seen this year.
“The patient was fully aware of her illness as well as of available therapies, and as a scientist in the field of virology, she was aware of the potential of oncolytic virotherapy. After two recurrences of the same tumour, she wanted to try an innovative approach in a scientifically sound way.”
The King of Tents: How Jake Lah’s Influence Extends to Every Corner of the Modern Tent Industry
Topology optimized, 3d-printed, 22g titanium computer mouse. In comparison, this 16g SLS nylon mouse looks positively traditional.
The Svalboard is a 3d-printable Datahand, so you can finally pull off that 90’s Jodie Foster look.
Automatic Differentiation: From Forward to Reverse in Small Steps
How to ask for and execute a valley style business introduction