Head-to-head: 3 no-bot AI notetakers, WER tested

A hands-on comparison of AI Assist, Granola, and MacWhisper — three AI meeting notetakers that don't put a bot in your call. I tested them on the same 20-minute audio, measured transcription accuracy against Whisper large-v3, and looked at how each handles the trust boundary. Reproducible methodology; scripts are in the repo.

Why these three, why now

Jeremy Caplan (Wonder Tools) asked me to compare AI Assist against Granola, Fathom, MacWhisper, and other tools that transcribe without a bot joining the call. Fathom uses a bot so I dropped it; the remaining three form the interesting "no bot" category in different ways:

• AI Assist — Chrome extension attaches to Meet/Zoom Web tabs; a local Rust/Tauri helper runs Whisper via the user's own OpenAI key.
• Granola — macOS app that captures system audio locally, then ships it to Granola's cloud for transcription and summarization.
• MacWhisper — pure transcription tool that runs Whisper locally on the user's Mac. No summaries, no meeting integration; you drop in an audio file and get text.

The interesting differences aren't in "no bot" — all three qualify — but in whose servers touch the audio, whose model runs the transcription, and what the tool produces after transcription.

Test material

One 20-minute audio excerpt: Y Combinator "How to Start a Startup" Lecture 1 by Sam Altman, minutes 5:00–25:00. Clean-audio single-speaker prepared talk with startup jargon.

Reference transcript: the same audio, transcribed via OpenAI's Whisper large-v3 API. Every tool under test uses Whisper or a Whisper variant under the hood, so this reference measures "how close does each tool get to state-of-the-art Whisper?" — which is the honest question for a Whisper-based category. Fully reproducible; the download + transcription script is in docs/head-to-head/download-audio.sh.

1. Transcription accuracy — Word Error Rate

MacWhisper (1.19%) essentially matched the reference. It runs Whisper Large locally, so this makes sense — same model, same audio, minor normalization delta. If pure transcription accuracy is your only concern, MacWhisper is very hard to beat.

AI Assist has two code paths, and their WER is very different:

• Meet-captions path (44.29%, measured): in Meet calls, AI Assist opportunistically uses Google Meet's built-in live captions and never invokes its Whisper backend. Meet's captions are optimized for low-latency real-time delivery over accuracy, and on lecture-style content they lag Whisper significantly. This is a product-design decision (saves the user's API cost when Meet is providing captions for free) — but it's worth naming so users know what they get by default in Meet.
• BYOK Whisper path (~1–3%, architectural estimate*): in Zoom Web or in a Meet call with captions disabled, AI Assist falls back to its own helper, which calls POST https://api.openai.com/v1/audio/transcriptions with model=whisper-1. That's the same API call that generated our reference transcript — so the helper's Whisper output on this audio would be nearly identical to the reference, WER ≈ 1–3%.

*The ~1–3% number is an architectural claim, not an empirical measurement. I set up a Zoom Web test to measure it directly, but hit repeated pipeline issues (Chrome mic settings, extension pairing state after helper restart) unrelated to transcription quality itself. Rather than publish a rushed number, I'm labeling this row as architectural for now and will confirm the exact figure in a follow-up post once I've stabilized the harness. If you're evaluating AI Assist for a Meet-heavy workflow, treat the 44% as the honest default; for a Zoom Web workflow, expect MacWhisper-adjacent accuracy.

Granola (74.72%) was surprising. The transcript is heavily degraded — dropped clauses, mangled phrases throughout — and the model hallucinated multi-language historical content that isn't in the audio at all (see below). Some of the WER inflation is my methodology (BlackHole loopback isn't identical to real meeting audio), but the hallucinations aren't explained by that.

The Granola hallucination worth flagging

Granola's summary opens with a "Context" section that says:

None of that content is in Sam Altman's YC lecture. It's not "recording bleed" — the audio was pure lecture playback into BlackHole. Granola's transcription pipeline produced a run of Spanish and Latin text that looks like a Wikipedia article about Visigoth Spain, and Granola's summary model rationalized it as "recording bleed." Both the transcription model and the summary model hallucinated, and the summary model's confabulated explanation is worse than the original error — a user reading the summary would trust it.

The good news for Granola: the core summary bullets about startup ideas are largely accurate to Sam Altman's actual argument. It's the boundaries and the confidence-marked "Context" wrapper that go wrong.

2. Summary quality

Different design philosophies:

• MacWhisper — no summary. Transcription only. That's their product scope and it's honest.
• AI Assist — no summary auto-generated in this test session; summarization is triggered by an explicit user action.
• Granola — auto-generated a full structured summary immediately after recording stopped.

Only Granola produced a summary this run, so a proper 3-tool head-to-head on summary quality isn't possible here. What Granola did produce, rated on the standard rubric:

The 2/5 accuracy score is entirely because of the hallucinated Byzantine/Visigoth content. The core startup-advice bullets — long-term thinking, market strategy, building something users love, feedback loops — are all faithful to Sam Altman's talk.

3. Trust boundary — where does the data actually go?

Each tool routes audio, transcript, and summary data through different servers. This matters more than any WER delta if you work in a regulated industry.

Three different trust postures. MacWhisper wins outright if "no vendor at all sees my audio" is your bar. AI Assist wins if you want vendor accountability but through vendors you already audit (OpenAI and Anthropic under your own account and terms). Granola requires you to trust their pipeline — legitimate if you do, dealbreaker if your legal team says otherwise.

4. Cost — 30 meetings/mo × 45 min

MacWhisper is cheapest because it's the smallest scope (transcription only, no summaries, no meeting integration). Granola looks slightly cheaper as a fixed monthly total because their subscription bundles the LLM cost. AI Assist looks slightly higher on paper because the BYOK API costs count as "yours" — the tradeoff is that you control the model, can upgrade to Claude Opus the moment it ships, know exactly what you're paying for, and can pin the price per API call rather than being locked into a subscription tier's pre-baked model choice.

Verdict per use case

• You want cheapest transcription and don't need summaries: MacWhisper. Nothing else touches it on pure transcription accuracy AND privacy AND cost.
• You want convenience, auto-summaries, and trust Granola with your audio: Granola. Product polish is excellent; just be aware of the hallucination behavior at the summary boundaries.
• You want vendor accountability but through vendors YOU already audit, control of the LLM choice, and audio that never touches the tool maker's servers: AI Assist. Best fit for BYOK teams and privacy-conscious solo users.

Notably, all three answers are correct — for different questions. The category has grown enough that "AI notetaker" is no longer one product shape.

Methodology

Reproducible from public materials. Full scripts at docs/head-to-head/ in the AI Assist repo.

• Audio: YC "How to Start a Startup" Lecture 1, minutes 5:00–25:00. Download + slice via yt-dlp and ffmpeg.
• Reference transcript: OpenAI Whisper large-v3, run once against the excerpt.
• Input routing: BlackHole 2ch virtual audio device + macOS output/input routing — same audio played into each tool via system-audio loopback (for Granola and AI Assist). MacWhisper takes the file directly.
• WER calc: jiwer Python library, standard normalization (case-fold + punctuation strip + whitespace collapse).
• Summary rubric: single grader (me), 3 dimensions, 1-5 scale.

Limitations — honest ones

• Sample size = 1 audio. One 20-minute excerpt is directional, not definitive. WER numbers might rerank on multi-speaker or noisier audio.
• BlackHole loopback ≠ live meeting audio. Granola's model may be tuned for real meeting patterns (voices with mic characteristics, mixed participants) and get confused by clean stereo loopback. The 74% WER might be lower in a real Meet call. The hallucinations, however, aren't explained by this — those are a real content-safety issue with the summary pipeline.
• Single grader on summary quality. I'm biased toward AI Assist because I built it. The rubric is designed to minimize that (specific claims to verify, not vibes), but bias exists.
• AI Assist's Whisper path wasn't measured empirically in this pass. The Meet-captions path was measured directly at 44.29%. I attempted a Zoom Web run to exercise the Whisper path but hit repeated setup-flow issues (Chrome default-mic vs. Zoom-mic separation, extension pairing state after helper restart, persistent-connection re-establishment on tab reload) that consumed the test budget without producing a clean measurement. The ~1–3% Whisper-path figure in the table above is architecturally derived (same whisper-1 API endpoint that generated the reference) — reasonable to trust, but I owe a follow-up post with a clean empirical number.
• Reference is Whisper large-v3, not human-verified. Any tool that matches large-v3 gets a WER of ~0. This is a "how close to SOTA Whisper" measurement, not a "how close to a human transcriber" measurement.

If any of the three vendors wants to rerun this against different audio or a different reference, the scripts are open and the methodology is public. Happy to update the writeup with reproduction data.