People stop reading AI meeting notes the first time an action item is attributed to somebody who did not agree to it. The fix is not a better model or a longer prompt: it is requiring every extracted item to carry the exact words it came from, and deleting — in code, before anyone sees it — any item whose quote is not found in the transcript.
Why summaries are not trusted
A summary compresses, and compression is where things get invented. The reader has no way to check a claim like “the team agreed to postpone the launch” without rewatching the meeting, which is the thing the summary existed to avoid. So the reader either trusts it blindly or ignores it, and after one bad experience it is the second.
Attach a quote and the economics reverse: checking an item costs three seconds, so items get checked, so errors get caught early and the tool keeps its credit. Grounding an output in retrievable evidence is the general form; here the evidence is thirty words of transcript.
The output shape
Three lists, one schema, and a field that has to be verbatim. Ask for it explicitly and say why — the instruction that a quote will be checked measurably reduces paraphrasing.
SYSTEM = """Extract from the meeting transcript below.
Return JSON only:
{"decisions": [{"decision": str, "quote": str, "speaker": str}],
"actions": [{"action": str, "owner": str, "due": str|null,
"quote": str, "speaker": str}],
"open_questions": [{"question": str, "quote": str, "speaker": str}]}
The "quote" field must be copied from the transcript character for character,
between 8 and 40 words, and must be the passage the item came from. It will be
checked against the transcript automatically; items whose quote cannot be found
are discarded. Do not paraphrase inside quote. Do not add ellipses.
An action needs an owner who was named or who volunteered. If nobody took it,
put it in open_questions instead of guessing an owner."""
Enter fullscreen mode Exit fullscreen mode
The instruction to put ownerless work into open_questions is the one that stops the most damaging failure. A model asked for actions with owners will supply owners, and the owner it supplies is whoever was talking.
Verifying the quote in code
Exact string matching is too brittle — transcripts contain inconsistent punctuation and models tidy it. Normalise both sides and match on that, then map back to the original span for display.
# verify.py
import re, unicodedata
def normalise(s):
s = unicodedata.normalize("NFKC", s).lower()
s = s.replace("’", "'").replace("“", '"').replace("”", '"')
s = re.sub(r"[^a-z0-9' ]+", " ", s)
return re.sub(r"\s+", " ", s).strip()
def locate(quote, transcript, norm_transcript=None):
"""Return (start, end) into the ORIGINAL transcript, or None."""
nq = normalise(quote)
if len(nq.split()) < 5:
return None # too short to be evidence
# Build a map from normalised offsets back to original offsets once.
offsets, buf = [], []
for i, ch in enumerate(transcript):
n = normalise(ch)
if n:
buf.append(n)
offsets.append(i)
nt = "".join(buf)
nt = re.sub(r"\s+", " ", nt)
pos = nt.find(nq)
if pos == -1:
return None
return offsets[pos], offsets[min(pos + len(nq), len(offsets) - 1)]
def keep_verified(items, transcript):
kept, dropped = [], []
for it in items:
span = locate(it.get("quote", ""), transcript)
if span:
it["span"] = span
kept.append(it)
else:
dropped.append(it)
return kept, dropped
Enter fullscreen mode Exit fullscreen mode
Log the dropped items rather than discarding them silently. The drop rate is the single best health metric this tool has: a stable 3–5 per cent is the model tidying punctuation, and a jump to 30 per cent after a model change means it has started paraphrasing and the notes were about to become fiction.
Long meetings: windows, not one prompt
A ninety-minute meeting is roughly 13,000 words — about 17,000 tokens — which most models will accept. Accepting it and being reliable across it are different things: recall of specific details in the middle of a long context degrades, and one prompt over the whole transcript consistently under-reports the middle third.
- Split the transcript into windows of about 3,000 tokens with 300 tokens of overlap, cutting only at speaker turns.
- Extract from each window independently. This is parallelisable and each call is small, so it is fast and cheap.
- Verify every quote against the whole transcript, not just its window. Overlap means the same item can be found twice.
- De-duplicate on the normalised quote span: two items whose spans overlap by more than half are the same item, and you keep the one with the longer quote.
- Run one final pass over the merged list only — not the transcript — to order items and drop near-duplicates the span test missed.
The final pass sees a few hundred tokens rather than seventeen thousand, which is why this is cheaper than the single-prompt version as well as more accurate.
Speakers, and who owns an action
If your transcript has speaker labels, keep them in the window text as Name: text lines and require the speaker field to be one of the names present. Validate it in code the same way you validate the quote — an owner who was not in the meeting is a fabrication, and it is a set membership test.
If your transcript has no speaker labels, do not ask the model to invent them. Diarisation is a signal-processing problem, not a language one, and a model given an unlabelled transcript will assign names by plausibility. Say “owner not identified” instead; it is accurate and a human fixes it in two seconds.
The transcript sets the ceiling
Everything downstream is bounded by the transcript, and teams consistently spend their effort on the prompt when the errors are upstream of it. Four things improve a transcript more than any prompt change will.
- Per-speaker audio. If the meeting platform can give you one track per participant, take it. Speaker attribution becomes exact rather than inferred, and the whole diarisation problem disappears. This is the single largest quality difference available and it costs nothing but plumbing.
- A vocabulary hint. Most speech recognisers accept a list of terms to bias towards — product names, people’s names, acronyms. Feed it the attendee list and your product glossary. A recogniser that renders your product name three different ways produces three different clusters everywhere downstream.
- Timestamps at the word or segment level. With them, a verified quote can link to the second of audio it came from, which turns “check the quote” from reading into listening. Without them you can only ever cite text.
- The room, not the model. One participant on a laptop microphone in a hard-surfaced room degrades the transcript for everything they say, and no amount of post-processing recovers it. A headset is a cheaper accuracy improvement than any model.
Whatever recogniser you use, keep its confidence output if it exposes one and mark low-confidence spans in the stored transcript. An action item whose supporting quote sits inside a span the recogniser was unsure about should go to review regardless of how well-formed the extraction looks — that is the one place where the two confidence signals compose usefully.
The cost of an hour
60-minute meeting, ~150 words per minute of speech
= 9,000 words ~ 12,000 tokens of transcript
Windowed extraction, 3,000-token windows with 10% overlap
= 5 windows x (3,000 in + ~400 out)
= 15,000 input tokens + 2,000 output tokens
Merge pass = ~1,500 in + 400 out
TOTAL ~ 16,500 in, 2,400 out
At $0.15 / $0.60 per million (a small model's posted prices):
16,500/1e6 x 0.15 = $0.0025
2,400/1e6 x 0.60 = $0.0014
~ $0.004 per meeting, plus transcription.
Transcription is the dominant cost: speech-to-text is typically billed per
minute of audio, and at any plausible rate it is 10-100x the text cost above.
Put your provider's per-minute price in before you plan a budget.
Enter fullscreen mode Exit fullscreen mode
The per-million prices above are an illustrative small-model rate, not a quote. Model prices move; substitute the posted price for the model you actually call. The ratio that will not move is the one the arithmetic exposes — the text processing is rounding error next to the audio.
Which is worth stating plainly: this is an audio pipeline with a text stage attached, not the reverse, so the optimisations that matter are upstream of the model. And judging the result is its own problem — evaluating a summary is much harder than evaluating an extraction, which is a further argument for producing quotable items rather than prose.
What this still gets wrong
- Decisions made by silence. “Unless anyone objects, we ship Thursday” followed by nobody objecting is a decision with no quote confirming it. Quote-grounding is conservative by construction and will miss these. That is the trade you accepted.
- Sarcasm and hedging. “Sure, we could do that by Friday” is captured as an agreement complete with a verifiable quote. The quote is real; the reading is wrong. Only the human review catches this.
- Transcription errors upstream. A verified quote is verified against the transcript, not against the meeting. If the speech-to-text heard “can’t” as “can”, everything downstream is confidently wrong and no amount of grounding helps.
None of these is a reason not to build it. They are the reason the output should say “drafted from the transcript, quotes included” rather than “minutes”, and what you call the output changes how it is read more than any prompt change will.
답글 남기기