Gate G5: speech accuracy

G5 PARTIAL

QuestionCopy Anchor Link

Is speech-to-text accuracy on noisy bid audio high enough to accept bids without a manual confirm?

ResultCopy Anchor Link

Held-out noisy set, measured onceCopy Anchor Link

On a synthetic held-out set measured once (120 clips: eSpeak speech, simulated room and babble noise at 20, 10 and 5 dB, run 42567832671), neither provider met the pre-registered thresholds: Groq wrong accept 0.058 and exact plus recoverable 0.923, Deepgram wrong accept 0.092 and exact plus recoverable 0.744, against 0.05 and 0.90; this is worse than the in-sample result.

A wrong accept is a clip where the parser answered accept, a bid taken without a confirm prompt, with a pair other than the spoken bid; it is counted over all 120 clips. The parser asked for a confirm on 0.175 (Groq) and 0.233 (Deepgram) of the clips, and on the room microphone no bid is auto-accepted by construction, so these are parser answers and not bids that reached the auction. The audio is synthetic and the noise is simulated. The set was measured once and nothing was tuned on it; any later parser, option or threshold change made after reading these failures counts as in-sample. G5 stays PARTIAL. Run: 42567832671; files and sha256 in docs/evidence/G5.

Live socket runCopy Anchor Link

The live /audio socket was run once with both providers (run 42575018866): the 24 committed synthetic clips were streamed in real time with an auctioneer token to the real server process (fake PayPal), the live provider answered 24 of 24 clips for each, and no spoken bid was accepted without a confirm (5 confirm prompts, 19 clips with no room action, 0 accepted). The log scan of the server log found 0 findings with the provider key as the canary. Groq's transcript equals the spoken text for 12 of 24 clips (caption latency after the end of speech p50 191 ms, p95 489 ms); Deepgram's for 21 of 24 (its latency reads 0 ms for every clip, which is not a meaningful number). Synthetic clips, fake PayPal in the server, no human voice and no real room; the web console still has no audio sender, so its microphone button stays disabled. G5 stays PARTIAL. Run: 42575018866; files in docs/evidence/voice-live-socket.

MethodCopy Anchor Link

• Cached eval: 24 clips (24 utterances) were sent to each provider once and the outputs recorded; the cached eval scores the recorded outputs through the rule parser. It runs in the voice-eval workflow (text and cached modes run on pull requests with no keys; live mode is manual).
• Clip set one: synthetic eSpeak speech, noise-free and robotic (PR #1940).
• Clip set two: the same 24 utterances spoken by one natural text-to-speech voice (PR VOI-01). Still synthetic and noise-free. The current recordings come from the option trials of PR VOI-02 (voice-eval runs 42364412751 and 42365428369).
• Providers: Deepgram nova-3 (punctuate=true, smart_format=false) and Groq whisper-large-v3-turbo with a bid-words prompt and temperature 0. The first run used Deepgram nova-2 and Groq without a prompt.
• Pass rule for the live run (npm run gate:g5 -w @gavel/e2e -- --write-gates, 20 private noisy recordings, DEEPGRAM_API_KEY): PASS needs exact plus recoverable confirm at 0.90 or more and wrong accepts at 0.05 or less; PARTIAL needs 0.70 or more; otherwise FAIL.

Numbers, from docs/GATES.mdCopy Anchor Link

Before the change (nova-2, no prompt): exact 0.615 to 0.692, wrong accept 0.083 to 0.167. Where the change came from, as exact and wrong-accept clips, Deepgram eSpeak: main 0.615 and 3 clips; the label fix alone 0.769 and 1; plus parser fixes on the old recordings 0.846 and 0; plus provider options 1.000 and 0. The 9 clips that drove a rule went from 0.286 to 1.000 (Deepgram) and 0.857 (Groq) exact; the 15 other clips were 1.000 before and after. With 24 clips one clip moves a rate by 0.042.

System level: nothing is auto-accepted on the room microphone (0 of 24 per run, by construction); with a device microphone and a second reading that always agrees, 0 of 15 auto-accepted bids were wrong. Per-clip causes: docs/evidence/G5/ERROR-ANALYSIS.md.

Parser only: the text-level runCopy Anchor Link

Reference text goes in as recognized text, so this measures the parser and nothing about speech-to-text. On 169 cases in 23 categories (commit 5d2e253, artifact voice-eval-text): exact 0.929, wrong accept 0.000, confirm 0.272. The held-out 40 cases were not used to change the lexicon or thresholds. Thresholds stay at AUTO 0.85 and CONFIRM 0.50 (issue 478): the sweep from 0.70 to 0.95 is flat because the parser emits only 0.50, 0.95 and 1.00.

EvidenceCopy Anchor Link

• 24 clips under packages/voice/fixtures/audio and recorded outputs under stt-cache (Deepgram, Groq) and stt-cache-human.
• docs/GATES.md, section G5.

LimitsCopy Anchor Link

• Synthetic audio only: noise-free in the two 24-clip sets, simulated noise in the held-out set. No real room, no person, no accents beyond the parser's text cases.
• The 24-clip sample is small, and the parser rules and provider options were chosen after reading the failures on it.
• Latency is not measured by the text mode.
• The nightly harness skipped G5 (20 of 20 private recordings missing), so no live G5 run exists.

What GAVEL does about itCopy Anchor Link

A spoken bid is never accepted silently: it goes through the confirm gate. See voice and confirm. For demos, use the recorded replay, which makes no PayPal call.

• packages/e2e/src/gates/g5.tsthe G5 harness
• packages/e2e/src/gates/g5-cli.tsthe live run command
• packages/voice/fixturesclips and recorded provider outputs
• .github/workflows/voice-eval.ymltext, cached and live modes