Define the real listening conditions
Describe whether the target audio comes from customer support, sales conversations, service scheduling, or another telephone workflow. Include the capture path, languages, and expected recording quality. A model evaluated only on clean read speech may face quite different conditions when callers interrupt, hesitate, or speak over background activity.
Keep reference rules explicit
Specify verbatim versus cleaned transcription, number formatting, punctuation, partial words, and uncertain speech. Decide how hold music, agent scripts, and non-speech events are represented. Reference conventions influence the interpretation of recognition errors, so keep them stable and provide examples alongside the test set.
Report meaningful slices
Compare results across language, domain, call length, channel configuration, and recording conditions. Separate names, account-like strings, and specialist vocabulary when those matter to the task. A single score can conceal a model that works well on agent introductions but poorly on the caller’s actual problem.
Review source and sample fit
Keep recordings and related speakers grouped according to the intended evaluation design. Ask for a representative pilot and a clear source description. Send Fiund your transcription rules, coverage priorities, and handling requirements so any potential source is reviewed for the specific project rather than assumed suitable from an audio sample. Choose a pilot excerpt where a caller corrects an amount or spells an unfamiliar name. Compare the model output and reference against the audio at that interval. This reveals whether the evaluation conventions capture meaningful recognition errors or mainly reflect formatting choices that should be normalized consistently before interpreting the result.
Review checklist
- Describe capture and deployment conditions
- Provide reference transcription examples
- Choose domain and audio-quality slices
- Specify grouping and source-review requirements