Translate criteria into observable events
Specify the behavior you want to identify, such as an agent asking a clarifying question or repeating a proposed next step. Avoid broad labels like good call without a rubric. The recording must contain enough evidence for a reviewer to apply the definition consistently within the actual task.
Separate behavior from outcome
A frustrated caller or unresolved issue does not by itself establish poor agent behavior. Keep conversation observations, customer-reported reactions, and business outcomes as different fields. Document information that is unavailable in the recording so reviewers do not infer unseen system actions or internal policies from tone alone.
Use calibrated review rules
Provide positive, negative, borderline, and not-observable examples for each label. Require supporting intervals for important judgments and retain disagreements during pilot review. If the rubric depends on a company-specific procedure, include the relevant reviewed context instead of assuming annotators know an unstated standard.
Scope the intended use
Decide whether the dataset supports research, model evaluation, or a particular reviewed operational workflow. Source suitability and sensitive information require assessment for the project. Bring Fiund the rubric, evidence format, necessary context, and exclusions so sourcing discussions can focus on calls that genuinely support those labels. For calibration, give reviewers a call where the agent asks the required question but the caller declines to answer. Check whether the rubric records the observable action separately from the unsuccessful outcome. Borderline examples like this are useful for clarifying what a quality label actually means before it becomes a model target.
Review checklist
- Define observable behaviors with examples
- Keep outcome separate from behavior
- Include not-observable and uncertain labels
- Require supporting intervals for judgments