Methodology

01What we measure, and where we measure it from

The headline timing figure is time to first agent byte (TTFAB), measured from the recording. The interval starts at the last sample of caller speech and ends at the first sample of agent speech. It contains the endpointing wait, the vendor's own processing, and both network legs.

This reads higher than any vendor's time-to-first-byte, and it should. The vendor measures from their server. We measure what the caller heard.

Both numbers are honest. They answer different questions. A vendor optimising their server-side number is doing real work; it is simply not the number a caller experiences, and a procurement decision made on it is being made on the wrong interval.

Nothing in this document reads a vendor log, a dashboard, or a timing export. The only input is audio.

02How recordings are processed

Seven stages. Each one can reject a recording, and every rejection is named in the report before any metric is published.

The order matters: a later check cannot be trusted if an earlier one failed. Channel layout is settled before speech detection runs, because the layout decides which metrics are eligible at all, and timing is computed before transcription, because no timing figure on this site depends on a transcript.

03Which recordings we exclude, and why

Ingest runs as a ladder. A recording that fails a rung stops there and is recorded with the reason for that rung.

Raw telephone numbers never enter a call record. They are salted and hashed at ingest, and the salt is not stored alongside the recordings.

04How we tell the caller from the bot

Channel layout is the single most consequential field in the pipeline, because it determines which metrics are eligible at all.

With both speakers mixed into one channel, overlap and interruption cannot be measured. Not approximately, not with a caveat, not with a confidence interval. They are marked ineligible per call and the reason is printed where the number would have been.

The greets-first rule

Where the layout is ambiguous, the channel that speaks first in the opening two seconds is provisionally the agent, because an outbound agent greets. This is a prior, not a finding, and it is overridden by any positive evidence.

The detector only upgrades with evidence

Where it cannot establish channel separation, it defaults to the degraded interpretation, mono, and marks the richer metrics ineligible. A detector that guessed upward would inflate the metric set on exactly the recordings where the audio is worst.

05How we detect speech

Two independent detectors run over every channel. The second detector can only veto or flag a segment the first proposed. It can never create one. A pipeline where either detector can assert speech has two chances to hallucinate a turn; this one has none.

Where the two disagree about a segment's edges by more than 150 ms, the segment is discarded and counted, and the count is published in the report. A discarded segment is not a silent loss: it appears in the coverage section.

The segmentation configuration is locked and published:

The SHA-256 of the segmentation config stamps every measurement. Measurements made under different configurations are therefore distinguishable, and a batch profiled before a config change cannot be silently compared with one profiled after.

06Thresholds we use, and why each one

The overlap gate is the honest one. A 12 dB gate produced cleaner-looking output and was wrong: it discarded genuine barge-ins whenever the line was noisy, which is precisely the condition under which barge-in matters most. The gate sits at 6 dB and the noisier output is the correct output.

07Why we report percentiles and never averages

Latency distributions from real voice systems are not symmetric. They have a dense body and a long right tail, and the tail is where the failures live: the call where recognition stalled, the call where a retry fired, the call where the caller said “hello?” twice and hung up. The arithmetic mean is the statistic most efficient at hiding that shape.

So the report gives P50, P90, P95 and P99, stratified by language, duration, campaign, agent version and direction, with the number of calls behind every cell.

The percentile support rule

A percentile is published only when there are enough calls behind it.

P99 of three calls is the maximum wearing a costume.

Below the threshold the cell is withheld with the reason attached, never a zero, never a blank. The report prints something of the form withheld: P95 requires 20 calls; this stratum has 16.

There is no average anywhere in the report, and no single score, because a profile is not a grade. A single number would be a weighted collapse of the distribution, and the weights would encode somebody's opinion about which failures matter.

08How we decide whether a difference between two vendors is real

When two batches are compared, every reported difference carries a confidence interval and a minimum detectable effect, the smallest gap the comparison could have found, given how many calls you supplied.

Resampling happens over calls, not turns. One long call can contribute forty turns, and resampling turns would treat those forty as independent evidence when they are one conversation. The cluster bootstrap resamples at the call level.

09How we test ourselves

Before the instrument is allowed near customer audio, it has to recover answers it is not told. The golden corpus is sixteen synthetic files: four scenarios × two sample rates × two noise conditions, each carrying known truth to four decimal places: a gap of exactly 800 ms, an overlap of exactly 300 ms, 6.0 s of dead air, a caller turn the agent never answers.

Published tolerances: 50 ms on clean audio, 100 ms at 10 dB signal-to-noise.

The evaluator law

Tolerances may not be relaxed. Truths may not be edited. Assertions may not be deleted or skipped. This is enforced mechanically by a hook that blocks the edit, not by convention, because the one moment anybody would want to weaken a tolerance is the moment it starts failing.

10Our own margin of error

On a closed-loop probe of five consecutive calls and fifteen turns, the instrument's own error is:

We publish this before we publish anyone else's number. An instrument that will not state its own error is not an instrument.

Three assertions that currently sit outside tolerance

Three of the 48 golden assertions do not pass. All three are 10 dB signal-to-noise trailing edges, the moment a speech segment ends, on the noisiest files in the corpus.

Nine configuration changes were tested against these and rejected on measurement, not argument: tighter inward offset budgets (83 misses, against 7), a separate lower silence threshold (34–45), deferring the offset to the second detector's tail (86), high-band envelopes for fricative tails (useless, because the corpus noise is white and masks every band equally), larger 100 ms speech pads (30), and energy floors at −55 dBFS (5 misses) and −65 dBFS (7 misses) against 3 at the locked −45.

11How reports are signed and verified

Every report is signed with an Ed25519 key over the input manifest and the canonical report body. Verification runs ten named checks, in order:

1. Parses as JSON
2. The coverage section leads the report
3. The signature is the final block
4. Signature present
5. Byte-identical to the signed form
6. Report body digest matches
7. Input manifest digest matches
8. Combined digest matches
9. Public key matches a pinned key
10. Ed25519 signature verifies

Checks 5 and 6 are independent of each other: one compares bytes, the other compares a digest computed from the parsed body. A single flipped byte that improved a P50 by 10 ms is rejected by three independent checks.

Two ways a report can pass, and they are not the same thing

Verified. Every check passed against a public key you obtained independently of the file. This is a trust decision you are in a position to make.

Self-consistent. Every check passed, but the key came from the file it signs. A file that signs itself with its own enclosed key proves internal consistency and nothing about provenance. We will not print it as a green tick.

12Current limits

These are the things this method cannot currently tell you. They are listed here rather than in a footnote because they are the reason to trust the rest.

• Per-turn language attribution is not wired. The field exists in the schema and is empty. We do not fill it with a guess.
• Code-switch density is validated against synthetic known truth only: Hinglish clips built with known switch points, not hand-labelled real calls. A hand-labelled check on real audio runs before any customer report ships.
• Romanised Hindi written in Latin script counts as English outside a curated lexicon, and that lexicon is heuristic-grade. This biases density downward on exactly the transcripts where it matters most.
• Transcription error against human truth is not measurable on production audio, because production audio has no human reference transcript. What two engines produce is agreement between engines, and it is reported under that name.
• Overlap and interruption cannot be recovered from a single-channel batch, at any price, on any deadline.
• Whether response time rises with code-switch density is a hypothesis, not a finding. The mechanism is plausible: recognition degrades on code-switched speech, lower recogniser confidence plausibly delays the endpointing decision, and a delayed endpointing decision is by construction a longer wait. We can measure the stratification and let you read it. We have not asserted the causal chain, and we won't until it is measured on real customer audio.