Head-to-head: 3 no-bot AI notetakers, WER tested

A hands-on comparison of AI Assist, Granola, and MacWhisper — three AI meeting notetakers that don't put a bot in your call. I tested them on the same 20-minute audio, measured transcription accuracy against Whisper large-v3, and looked at how each handles the trust boundary. Reproducible methodology; scripts are in the repo.

Why these three, why now

Jeremy Caplan (Wonder Tools) asked me to compare AI Assist against Granola, Fathom, MacWhisper, and other tools that transcribe without a bot joining the call. Fathom uses a bot so I dropped it; the remaining three form the interesting "no bot" category in different ways:

• AI Assist — Chrome extension attaches to Meet/Zoom Web tabs; a local Rust/Tauri helper runs Whisper via the user's own OpenAI key.
• Granola — macOS app that captures system audio locally, then ships it to Granola's cloud for transcription and summarization.
• MacWhisper — pure transcription tool that runs Whisper locally on the user's Mac. No summaries, no meeting integration; you drop in an audio file and get text.

The interesting differences aren't in "no bot" — all three qualify — but in whose servers touch the audio, whose model runs the transcription, and what the tool produces after transcription.

Test material

One 20-minute audio excerpt: Y Combinator "How to Start a Startup" Lecture 1 by Sam Altman, minutes 5:00–25:00. Clean-audio single-speaker prepared talk with startup jargon.

Reference transcript: the same audio, transcribed via OpenAI's Whisper large-v3 API. Every tool under test uses Whisper or a Whisper variant under the hood, so this reference measures "how close does each tool get to state-of-the-art Whisper?" — which is the honest question for a Whisper-based category. Fully reproducible; the download + transcription script is in docs/head-to-head/download-audio.sh.

1. Transcription accuracy — Word Error Rate

MacWhisper (1.19%) essentially matched the reference. It runs Whisper Large locally, so this makes sense — same model, same audio, minor normalization delta. If pure transcription accuracy is your only concern, MacWhisper is very hard to beat.

AI Assist (44.29%) is a product-design finding, not a Whisper limitation. In this test AI Assist captured Google Meet's built-in live captions and never invoked its Whisper backend — the extension opportunistically uses Meet's own captions to save on API cost. Meet's live captions are optimized for real-time delivery over accuracy, and on lecture-style content they lag Whisper significantly. In a Zoom Web call (no built-in captions) AI Assist would fall back to Whisper via BYOK OpenAI and the number would land in MacWhisper territory. But for the Meet path specifically, this is the real behavior, and it's worth being honest about.

Granola (74.72%) was surprising. The transcript is heavily degraded — dropped clauses, mangled phrases throughout — and the model hallucinated multi-language historical content that isn't in the audio at all (see below). Some of the WER inflation is my methodology (BlackHole loopback isn't identical to real meeting audio), but the hallucinations aren't explained by that.

The Granola hallucination worth flagging

Granola's summary opens with a "Context" section that says:

None of that content is in Sam Altman's YC lecture. It's not "recording bleed" — the audio was pure lecture playback into BlackHole. Granola's transcription pipeline produced a run of Spanish and Latin text that looks like a Wikipedia article about Visigoth Spain, and Granola's summary model rationalized it as "recording bleed." Both the transcription model and the summary model hallucinated, and the summary model's confabulated explanation is worse than the original error — a user reading the summary would trust it.

The good news for Granola: the core summary bullets about startup ideas are largely accurate to Sam Altman's actual argument. It's the boundaries and the confidence-marked "Context" wrapper that go wrong.

2. Summary quality

Different design philosophies:

• MacWhisper — no summary. Transcription only. That's their product scope and it's honest.
• AI Assist — no summary auto-generated in this test session; summarization is triggered by an explicit user action.
• Granola — auto-generated a full structured summary immediately after recording stopped.

Only Granola produced a summary this run, so a proper 3-tool head-to-head on summary quality isn't possible here. What Granola did produce, rated on the standard rubric:

The 2/5 accuracy score is entirely because of the hallucinated Byzantine/Visigoth content. The core startup-advice bullets — long-term thinking, market strategy, building something users love, feedback loops — are all faithful to Sam Altman's talk.

3. Trust boundary — where does the data actually go?

Each tool routes audio, transcript, and summary data through different servers. This matters more than any WER delta if you work in a regulated industry.

Three different trust postures. MacWhisper wins outright if "no vendor at all sees my audio" is your bar. AI Assist wins if you want vendor accountability but through vendors you already audit (OpenAI and Anthropic under your own account and terms). Granola requires you to trust their pipeline — legitimate if you do, dealbreaker if your legal team says otherwise.

4. Cost — 30 meetings/mo × 45 min

MacWhisper is cheapest because it's the smallest scope (transcription only, no summaries, no meeting integration). Granola is the cheapest full notetaker at fixed cost. AI Assist looks slightly higher on paper because the BYOK API costs count as "yours" — the tradeoff is that you control the model, can upgrade to Claude Opus the moment it ships, and know exactly what you're paying for. Over 24 months, the amortized comparison flips as the license amortizes further and Granola's subscription keeps recurring.

Verdict per use case

• You want cheapest transcription and don't need summaries: MacWhisper. Nothing else touches it on pure transcription accuracy AND privacy AND cost.
• You want convenience, auto-summaries, and trust Granola with your audio: Granola. Product polish is excellent; just be aware of the hallucination behavior at the summary boundaries.
• You want vendor accountability but through vendors YOU already audit, control of the LLM choice, and audio that never touches the tool maker's servers: AI Assist. Best fit for BYOK teams and privacy-conscious solo users.

Notably, all three answers are correct — for different questions. The category has grown enough that "AI notetaker" is no longer one product shape.

Methodology

Reproducible from public materials. Full scripts at docs/head-to-head/ in the AI Assist repo.

• Audio: YC "How to Start a Startup" Lecture 1, minutes 5:00–25:00. Download + slice via yt-dlp and ffmpeg.
• Reference transcript: OpenAI Whisper large-v3, run once against the excerpt.
• Input routing: BlackHole 2ch virtual audio device + macOS output/input routing — same audio played into each tool via system-audio loopback (for Granola and AI Assist). MacWhisper takes the file directly.
• WER calc: jiwer Python library, standard normalization (case-fold + punctuation strip + whitespace collapse).
• Summary rubric: single grader (me), 3 dimensions, 1-5 scale.

Limitations — honest ones

• Sample size = 1 audio. One 20-minute excerpt is directional, not definitive. WER numbers might rerank on multi-speaker or noisier audio.
• BlackHole loopback ≠ live meeting audio. Granola's model may be tuned for real meeting patterns (voices with mic characteristics, mixed participants) and get confused by clean stereo loopback. The 74% WER might be lower in a real Meet call. The hallucinations, however, aren't explained by this — those are a real content-safety issue with the summary pipeline.
• Single grader on summary quality. I'm biased toward AI Assist because I built it. The rubric is designed to minimize that (specific claims to verify, not vibes), but bias exists.
• AI Assist's Whisper path wasn't exercised. The Meet-captions path was tested. For Zoom Web or captions-disabled Meet, AI Assist would use Whisper via BYOK OpenAI and land in MacWhisper's WER neighborhood. A follow-up test with a Zoom recording would establish that number.
• Reference is Whisper large-v3, not human-verified. Any tool that matches large-v3 gets a WER of ~0. This is a "how close to SOTA Whisper" measurement, not a "how close to a human transcriber" measurement.

If any of the three vendors wants to rerun this against different audio or a different reference, the scripts are open and the methodology is public. Happy to update the writeup with reproduction data.