Voice Augmentation

An agent that passes every test under clean audio conditions may fail the same tests under realistic ones. Voice benchmarking research consistently shows a pass-rate drop between clean and noisy environments, and the drop isn't just about speech recognition accuracy. Background noise, interruptions, and cross-talk test whether the agent's conversation logic holds up: turn detection, recovery from barge-in, tool-calling sequences under garbled input, and handling off-mic speech.

Augmentation is how you create those conditions. It injects realistic audio effects (noise, interruptions, side conversations) into your voice runs so you can run the same test suite under controlled adversity.

What Augmentation Does

Each augmentation strategy modifies one aspect of the audio or conversation flow during the simulation:

Strategy	What it does	When to use
noise	Mixes background noise (cafeteria, traffic, etc.) into the caller's audio at a configured SNR	Validate ASR robustness; baseline of every realistic run
barge_in	The caller interrupts the agent mid-reply with a short prompt	Test the agent's interruption handling and recovery
backchannel	Caller utters a short acknowledgement ("mm-hmm", "right") while the agent speaks	Test the agent's turn-detection logic
directed_speech	Caller turns away mid-call to address someone in their environment, in their own turn or over the agent's reply	Test the agent against off-mic cross-talk
secondary_speaker	A second voice in the room interjects, after the caller's line or over the agent's reply	Test multi-speaker robustness, and whether the agent keeps talking through speech not meant for it
cap	Driver emits multiple consecutive messages per turn (Concurrent Ask Probability)	Stress-test agent behavior when the caller sends a burst of messages
dropout	The caller goes silent for an entire turn	Test whether the agent notices dead air and re-prompts

Composition Rule

Augmentation is configured with the augmentation parameter of okareo.run_simulation(...). Composition is constrained:

You may use at most one strategy, or noise combined with one other strategy.

Valid combos: noise alone, barge_in alone, noise + barge_in, noise + backchannel, noise + dropout, etc. Invalid: barge_in + backchannel, dropout + barge_in (or dropout with any strategy other than noise), or any three strategies together.

To script different augmentations on different turns, use a per-turn plan instead of a strategy. Only noise may be set beside a plan. The app always builds a plan.

Configuring Augmentation

In the App

In the simulation form, once a voice target is selected, open Advanced Settings and scroll to Voice Simulation Settings. In the simulation builder, the Voice Augmentation card opens the same settings.

1. Background Noise: Select a profile from the dropdown (Cafeteria, Classroom, Office Babble, Traffic, or None), then set the Signal to Noise Ratio (-5 to 25 dB). Lower SNR means more noise relative to the speaker. Noise plays for the whole call.

2. Turns: everything else is set turn by turn. Each row is one turn, or a run of turns chosen from the row number's menu (Just turn N, Through turn N, To the end of the call).

   • Each row has two slots, and each holds one augmentation: On what the Driver says (Concurrent Ask, Dropout, Directed Speech or Secondary Speaker) and On the Target's reply (Barge-In, Backchannel, Background Speaker or Directed Speech). Dropout takes the whole turn.
   • A scripted Driver's steps set the turns, one row per step, with the step's words. Silence, DTMF and End conversation steps take no augmentations. When the steps come from each Scenario row, Preview with a row chooses which row's words the turns show; a row whose steps can't take the plan shows a warning.

3. Directed Speech, Barge-In, Secondary Speaker and Background Speaker say a line written from a Prompt, or Exact words. A scripted Driver says its steps word for word, so on its turns the only choice on what the Driver says is Secondary Speaker, and everything takes Exact words.

App vs SDK defaults

An augmentation added in the app plays on every turn of its row (probability 1), and interruptions start 200 to 600 ms into the target's reply. That matches the SDK's per-turn plans. The strategy tables below give the defaults for a single strategy, which differ; set values explicitly if you need the same behavior on both paths.

From the SDK

Pass an Augmentation container to okareo.run_simulation(...). Each strategy has a typed config class in okareo.augmentations; set one strategy field, or noise plus one other. To set augmentations turn by turn, pass a plan instead (see Per-turn plans).

from okareo.augmentations import Augmentation, BargeInAugmentation, NoiseAugmentation

result = okareo.run_simulation(
    name="Noise + Barge-In",
    target=target,
    scenario=scenario,
    driver=driver,
    max_turns=5,
    first_turn="driver",
    checks=["time_to_first_audio", "result_completed"],
    augmentation=Augmentation(
        noise=NoiseAugmentation(profile="cafeteria", snr_db=10),
        barge_in=BargeInAugmentation(
            probability=0.5,
            min_offset_ms=200,
            max_offset_ms=600,
            prompt="Ask for a very short polite interruption.",
        ),
    ),
)
print(f"Results: {result.app_link}")

The strategy classes are NoiseAugmentation, BargeInAugmentation, BackchannelAugmentation, DirectedSpeechAugmentation, SecondarySpeakerAugmentation, CAPAugmentation, and DropoutAugmentation, matching the strategy keys in the table above.

For example, to have the caller go silent on roughly one turn in five:

from okareo.augmentations import Augmentation, DropoutAugmentation

augmentation = Augmentation(dropout=DropoutAugmentation(probability=0.2))

Per-turn plans

A single strategy fires at random within its window. A per-turn plan scripts the call instead: which augmentation plays on which turn. Use it to reproduce a specific scenario, such as an interruption on the agent's first reply and a different one on the second. The app's Turns list builds the same plan.

Each turn has two slots, and each takes one augmentation:

Slot	Strategies
What the caller says	cap, dropout, and directed_speech / secondary_speaker without interrupt
The agent's reply	barge_in, backchannel, and directed_speech / secondary_speaker with interrupt

dropout takes the whole turn: the caller says nothing, so there is no reply to play over.

from okareo.augmentations import (
    Augmentation,
    BackchannelAugmentation,
    BargeInAugmentation,
    NoiseAugmentation,
)

augmentation = Augmentation(
    noise=NoiseAugmentation(profile="cafeteria", snr_db=10),
    plan={
        1: [BargeInAugmentation(utterance="Sorry, can I stop you there?", min_offset_ms=8000, max_offset_ms=9000)],
        "2-4": [BackchannelAugmentation(utterance="mm-hmm", probability=0.5)],
        "5-": [BargeInAugmentation(prompt="Ask to speak to a manager.")],
    },
)

Here the caller cuts in on the agent's first reply with exact words. On turns 2 to 4 the caller, half the time, says "mm-hmm" over the agent's reply. From turn 5 to the end of the call, the caller asks for a manager over every reply.

• Keys are the caller's turns, counted as in When an augmentation fires: one turn (2), a range ("2-4"), or a turn to the end of the call ("5-"). A key's augmentations play on every turn it covers. Keys can't overlap, and turns without a key have no augmentation.
• Entries are the usual strategy classes, with two differences:
  • probability defaults to 1: a planned augmentation plays. A lower value is drawn on each turn the key covers.
  • start_at_turn and end_at_turn aren't accepted, since the key sets the turns, and neither is cap's pause_ms, which applies to the whole call.
• Exact words: utterance on barge_in, directed_speech and secondary_speaker gives the words to say instead of having them written from prompt. Set one or the other. Scripted scenarios should use utterance.
• Offsets: interruptions start 200 to 600 ms into the agent's reply unless you set min_offset_ms / max_offset_ms. To land in the middle of a long answer, use later offsets, tuned to how long your agent talks.
• Only noise may be set beside a plan.

A plan that breaks these rules, for example with a second augmentation in a slot, overlapping keys, or start_at_turn on an entry, is rejected when the run is submitted, with a message that says which turn and what to change. Turns past max_turns are accepted; if the conversation ends first, they don't play.

Per-turn plans, and utterance on DirectedSpeechAugmentation and SecondarySpeakerAugmentation, require okareo 0.0.162 or later.

What a plan delivered

An interruption plays only if the agent is talking when its offset comes round, so a planned augmentation can miss. Check what each one did before judging the agent: a missed interruption means that scenario wasn't tested on that run.

In the transcript, each caller turn with a plan is marked with what played on it: delivered (with when, so you can jump to it in the recording), missed and why (for example, the agent finished before the offset), or skipped (for example, its probability didn't come up, or the connection can't interrupt). Augmentations on turns the conversation never reached are listed at the end.

The same record is in each conversation's model_metadata as augmentation_plan_results: one row per augmentation per turn it ran on, with its turn, type and status (delivered, missed, skipped or not_reached), and when they apply a reason, the key as span for a range, and when it played.

Strategy Parameters

Parameter names, types, and defaults below are from the server strategy constructors (the source of truth for what the API accepts).

When an augmentation fires

barge_in, backchannel, directed_speech, secondary_speaker and dropout take two parameters that bound when they can fire: start_at_turn (default 1) and end_at_turn (default: no limit; must be at least start_at_turn).

cap and noise do not take either parameter. cap also sets a run-wide pause between turns, which a start turn could not delay; background noise is an environment rather than an event, and on phone connections the noise bed is set up once when the call connects.

The turns they count are the caller's turns — the turns Okareo's simulated caller takes, not the agent's replies. Counting begins once the agent has actually said something, so ringing, dial tone, hold music and an empty greeting do not advance it. On a call that opens with three seconds of ringing tone, start_at_turn=1 lands on the first real turn of the conversation rather than during the tone. These are not the transcript's own turn numbers, so an augmentation can land on a later row than its number suggests.

Turns outside the window are left completely alone: the probability is not even rolled.

Firing on one specific turn

Set both ends to the same number and set the probability to 1:

Augmentation(dropout=DropoutAugmentation(probability=1.0, start_at_turn=3, end_at_turn=3))

The window decides where an augmentation may fire; the probability is still rolled inside it. With dropout's default of probability=0.1, start_at_turn=3, end_at_turn=3 would do nothing in nine runs out of ten. Each strategy has its own default, so check the table for the one you are using.

Dropout leaves an already-quiet line alone

Dropout only silences a caller turn the agent could actually react to. If the agent said nothing on the previous turn the line is already quiet, and the caller's turn is the only thing that can revive it, so it is left alone.

On an agent that replies every turn — the usual case — this never comes up: start_at_turn=N, end_at_turn=N with probability=1.0 drops turn N. It only matters for agents that can fall silent mid-call, for instance one that answers a turn with a tool call and no speech; a dropout aimed at the turn straight after that may not fire.

noise

Parameter	Type	Valid values	Default
noise_profile	string	cafeteria, classroom, office_babble, traffic	required (no default)
noise_snr_db	number	-5 to 25	10
seed	int or null	Any integer for reproducibility	null

Lower SNR = more noise relative to the speaker. 10 is moderate; try 0 to 5 for stress conditions.

tip

The UI form defaults noise_profile to cafeteria, but the API requires you to specify it explicitly. Omitting it returns a 400 error.

Aliases accepted: profile → noise_profile, snr_db → noise_snr_db. The SDK's NoiseAugmentation(profile=..., snr_db=...) uses the alias names.

barge_in

Parameter	Type	Range / Notes	Default
prompt	string	LLM instruction for the interruption phrasing. Not with utterance	required, unless utterance is set
utterance	string	Exact words for the interruption. Not with prompt	—
probability	number	0 to 1 (per agent turn)	0.2
min_offset_ms	int	0 or above	200
max_offset_ms	int	at least min_offset_ms	600
start_at_turn	int	1 or above. First caller turn this can fire on. See When an augmentation fires	1
end_at_turn	int or null	At least start_at_turn. Last caller turn this can fire on; null is no limit. See When an augmentation fires	null
seed	int or null	Reproducibility	null

Probability is per agent reply; min_offset_ms / max_offset_ms is the delay after the agent starts speaking before the interruption fires.

backchannel

Parameter	Type	Range / Notes	Default
utterance	string	Non-empty after trim	mm-hmm
probability	number	0 to 1	0.35
min_offset_ms	int	0 or above	150
max_offset_ms	int	at least min_offset_ms	450
start_at_turn	int	1 or above. First caller turn this can fire on. See When an augmentation fires	1
end_at_turn	int or null	At least start_at_turn. Last caller turn this can fire on; null is no limit. See When an augmentation fires	null
seed	int or null	Reproducibility	null

Backchannels repeat while the target is speaking. They do not consume a turn.

tip

The values above are the defaults for a single strategy. In a per-turn plan, and in the app, a backchannel starts 200 to 600 ms into the reply. Specify offsets explicitly if you want a particular window.

directed_speech

Parameter	Type	Range / Notes	Default
prompt	string or null	LLM instruction for the off-mic remark. Not with utterance	(built-in template)
utterance	string	Exact words for the remark. Not with prompt	—
probability	number	0 to 1	0.3
lpf_cutoff_hz	number	1 to 20000	800
gain_db	number	-40 to 0	-8
start_at_turn	int	1 or above. First caller turn this can fire on. See When an augmentation fires	1
end_at_turn	int or null	At least start_at_turn. Last caller turn this can fire on; null is no limit. See When an augmentation fires	null
interrupt	bool	Play the remark over the agent's reply instead of in the caller's turn. See Background speech over the agent	false
min_offset_ms	int	0 or above. Only with interrupt	200
max_offset_ms	int	At least min_offset_ms. Only with interrupt	600
seed	int or null	Reproducibility	null

Low-pass filter and gain attenuation simulate the muffled, off-mic quality of someone turning their head away from the phone.

secondary_speaker

Parameter	Type	Range / Notes	Default
secondary_prompt	string or null	LLM instruction for the second speaker's interjection. Not with secondary_utterance	(built-in template)
secondary_utterance	string	Exact words for the second speaker. Not with secondary_prompt	—
secondary_voice	string	Named voice from Okareo's voice catalog	Cathy - Coworker
probability	number	0 to 1	0.3
inter_speaker_pause_ms	int	0 to 5000. Not with interrupt	120
lpf_cutoff_hz	number or null	1 to 20000 (null disables)	800
gain_db	number	-40 to 0	-8
start_at_turn	int	1 or above. First caller turn this can fire on. See When an augmentation fires	1
end_at_turn	int or null	At least start_at_turn. Last caller turn this can fire on; null is no limit. See When an augmentation fires	null
interrupt	bool	Play the remark over the agent's reply instead of in the caller's turn. See Background speech over the agent	false
min_offset_ms	int	0 or above. Only with interrupt	200
max_offset_ms	int	At least min_offset_ms. Only with interrupt	600
seed	int or null	Reproducibility	null

Aliases accepted: prompt → secondary_prompt, voice → secondary_voice, utterance → secondary_utterance. The SDK's SecondarySpeakerAugmentation(voice=..., prompt=..., utterance=...) uses the alias names.

tip

The app defaults to Carson - Curious Conversationalist. The server default is Cathy - Coworker. Specify secondary_voice explicitly when calling via SDK.

Background speech over the agent

By default, directed_speech and secondary_speaker speak in the caller's turn, before the agent replies: a directed_speech remark is the caller's line for that turn, and a secondary_speaker remark comes right after it. Set interrupt to true to play it over the agent's reply instead. This tests whether the agent keeps talking through speech that isn't addressed to it, like someone in the caller's room asking "Is that the delivery?" while the agent reads out account details.

With interrupt on:

• The caller says their normal line, and the remark starts min_offset_ms to max_offset_ms after the agent starts replying. The defaults are 200 and 600 ms, the same as barge-in.
• The remark doesn't take a turn. The caller's next line responds to whatever the agent said.
• The remark appears in the transcript as a note beside the conversation, with who spoke it and when, not as a caller message.
• If the agent doesn't speak on that turn, or finishes before the delay, nothing plays.
• It needs a connection where the caller can interrupt the agent, as barge-in does: phone, SIP, or WebRTC, but not SIP dialed directly. On other connections the augmentation is left out and the run still succeeds.

min_offset_ms and max_offset_ms are only accepted with interrupt on, and inter_speaker_pause_ms only with it off.

In the app, add Background Speaker (a secondary_speaker with interrupt on) or Directed Speech to a turn's On the Target's reply slot.

from okareo.augmentations import Augmentation, SecondarySpeakerAugmentation

augmentation = Augmentation(
    secondary_speaker=SecondarySpeakerAugmentation(
        probability=1.0,
        voice="Cathy - Coworker",
        prompt='Ask the caller something offhand and unrelated to the call, e.g. "Is that the delivery?"',
        interrupt=True,
        start_at_turn=2,
        end_at_turn=2,
    )
)

The SDK fields interrupt, min_offset_ms, max_offset_ms, and seed on DirectedSpeechAugmentation and SecondarySpeakerAugmentation require okareo 0.0.161 or later.

cap

Parameter	Type	Range / Notes	Default
probability	number	0 to 1	0.3
pause_ms	int or null	0 to 10000, maps to turn_transition_time on the target. Not in a per-turn plan	1000

cap does not take start_at_turn or end_at_turn, and neither does noise. See When an augmentation fires.

When the probability gate fires, the driver emits multiple consecutive messages in a single turn. The number of messages comes from the driver's get_default_consecutive_messages() setting (typically 2). The pause_ms controls the gap between the driver finishing and the target responding.

dropout

Parameter	Type	Range / Notes	Default
probability	number	0 to 1 (per caller turn)	0.1
start_at_turn	int	1 or above. First caller turn this can fire on. See When an augmentation fires	1
end_at_turn	int or null	At least start_at_turn. Last caller turn this can fire on; null is no limit. See When an augmentation fires	null
seed	int or null	Reproducibility	null

Each dropped turn adds dead air until the agent speaks again or the target reply timeout elapses (silence_timeout_ms, 10 seconds by default; see Call Timeouts). The dropped turn is not counted toward turn-taking latency, and the dropped turn numbers appear in the results.

Augmentation(dropout=DropoutAugmentation(probability=1.0, start_at_turn=3))

Reading Augmented Results

Compare the augmented run's mean_scores against a clean baseline. The expected pattern:

• time_to_first_audio rises as the agent works harder against noise or interruptions.
• result_completed may drop if the agent can't handle the augmented conditions.
• response_loop flips to 0 if the agent gets stuck after an interruption.
• With dropout, the dropped caller turns are marked in the transcript so you can see how the agent handled the silence.
• With a per-turn plan, check what the plan delivered before reading the scores.

For a structured comparison between baseline and augmented runs, see Experimentation and A/B Testing.

Where to Go Next

• Voice Checks: checks that respond to augmented conditions.
• Experimentation and A/B Testing: diff augmented vs baseline runs.
• Load Testing: combine augmentation with concurrency.

Cookbook

Full runnable script: 06_augmentation.py