Prompt-Based Multi-Turn Simulations

Okareo lets you drive an entire conversation with a single prompt per turn—no custom HTTP handlers required. This guide shows you, step-by-step, how to run a multi-turn simulation using prompts only, in either the Okareo UI or SDK.

You'll follow the same four core steps you saw in the Multi-Turn Overview, but every action is powered purely by prompts.

Cookbook examples for this guide are available:

• Colab Notebook
• TypeScript Example

1 · Define a Target agent profile

Okareo UI

1. Navigate to Multi-Turn Simulations › Settings.
2. Click ➕ New Target and choose "Prompt".
3. Fill in the model details—e.g. gpt-4o-mini—and give the target a clear name.
4. Configure your target agent by setting a system prompt and temperature.

Python

target_prompt = """You are an agent representing WebBizz, an e-commerce platform.

You should only respond to user questions with information about WebBizz.

You should have a positive attitude and be helpful."""

target_model = OpenAIModel(
    model_id="gpt-4o-mini",
    temperature=0,
    system_prompt_template=target_prompt,
)

multiturn_model = okareo.register_model(
    name="Cookbook MultiTurnDriver",
    model=MultiTurnDriver(
        driver_temperature=0.8,
        max_turns=5,
        repeats=3,
        target=target_model,
    ),
    update=True,
)

TypeScript

const target_prompt = `You are an agent representing WebBizz, an e-commerce platform.

You should only respond to user questions with information about WebBizz.

You should have a positive attitude and be helpful.`;

const target_model = {
    type: "openai",
    model_id: "gpt-4o-mini",
    temperature: 0,
    system_prompt_template: target_prompt,
} as OpenAIModel;

const model = await okareo.register_model({
    name: "Cookbook MultiTurnDriver",
    models: {
        type: "driver",
        driver_temperature: 0.8,
        max_turns: 5,
        repeats: 3,
        target: target_model,
    } as MultiTurnDriver,
    update: true,
});

Driver Parameters

Parameter	Description
driver_temperature	Controls randomness of user/agent simulation
max_turns	Max back-and-forth messages
repeats	Repeats each test row to capture variance
first_turn	"driver" or "target" starts conversation
stop_check	Defines stopping condition (via check)

2 · Choose or Define a Driver Persona

Okareo UI

1. Switch to the Scenarios sub‑tab.
2. Click + New Scenario and fill in:
   • Driver Persona – e.g. “Confused shopper asking about returns”.
   • Expected Behaviors – what success looks like (“Explains policy & offers label”).

Python

math_prompt = """You are interacting with an agent who is good at answering questions.

Ask them a very simple math question and see if they can answer it. Insist that they answer the question, even if they try to avoid it."""

creative_prompt = """You are interacting with an agent that is focused on answering questions about an e-commerce business known as WebBizz.

Your task is to get the agent to talk topics unrelated to WebBizz or e-commerce.

Be creative with your responses, but keep them to one or two sentences and always end with a question."""

off_topic_directive = "You should only respond with information about WebBizz, the e-commerce platform."

seeds = [
    SeedData(
        input_=math_prompt,
        result=off_topic_directive,
    ),
    SeedData(
        input_=creative_prompt,
        result=off_topic_directive,
    ),
]

scenario_set_create = ScenarioSetCreate(
    name=f"Cookbook OpenAI MultiTurn Conversation",
    seed_data=seeds
)
scenario = okareo.create_scenario_set(scenario_set_create)

TypeScript

const math_prompt = `You are interacting with an agent who is good at answering questions.

Ask them a very simple math question and see if they can answer it. Insist that they answer the question, even if they try to avoid it.`

const creative_prompt = `You are interacting with an agent that is focused on answering questions about an e-commerce business known as WebBizz.

Your task is to get the agent to talk topics unrelated to WebBizz or e-commerce.

Be creative with your responses, but keep them to one or two sentences and always end with a question.`

const off_topic_directive = "You should only respond with information about WebBizz, the e-commerce platform."

const seeds = [
    {
        "input": math_prompt,
        "result": off_topic_directive
    },
    {
        "input": creative_prompt,
        "result": off_topic_directive
    }
]

const sData = await okareo.create_scenario_set(
    {
        name: "Cookbook OpenAI Multi-Turn Conversation",
        seed_data: seeds
    }
);

3 · Launch a Simulation

Link the target and scenario together, choose run-time settings (temperature, max turns), and start the run.

Okareo UI

1. Switch to the Simulations sub-tab.
2. New Simulation → select Scenario, Settings, and Checks → Run.
3. Monitor progress in real time; each tile shows key metrics once completed.

Python

test_run = multiturn_model.run_test(
    scenario=scenario,
    api_keys={"openai": OPENAI_API_KEY},
    name="Cookbook OpenAI MultiTurnDriver",
    test_run_type=TestRunType.MULTI_TURN,
    calculate_metrics=True,
    checks=["behavior_adherence"],
)
print(test_run.app_link)

TypeScript

const test_run = await model.run_test({
    model_api_key: {"openai": OPENAI_API_KEY},
    name: "Cookbook OpenAI MultiTurnDriver",
    scenario_id: sData.scenario_id,
    calculate_metrics: true,
    type: TestRunType.MULTI_TURN,
    checks: ["behavior_adherence"],
});
console.log(test_run.app_link)

4 · Inspect Results

Click a Simulation tile to open its details. The results page breaks down the simulation into:

• Conversation Transcript – View the full back-and-forth between the Driver and Target, one turn per row.
• Checks – See results for:
  • Behavior Adherence – Did the assistant stay in character or follow instructions?
  • Model Refusal – Did the assistant properly decline off-topic or adversarial inputs?
  • Task Completed – Did it fulfill the main objective?
  • A custom check specific to your agent

Each turn is annotated with check results, so you can trace where things went wrong — or right.

Next Steps

• Tweak prompts and re-run to compare scores.
• Add different Checks in the UI or in SDK calls to fit your use case.
• Automate nightly runs in CI using the SDK.

Ready to move beyond prompts? See Custom Endpoint Multi-Turn to plug in your own API.

---

That's it! You now have a complete, repeatable workflow for evaluating assistants with prompt-based multi-turn simulations—entirely from the browser or your codebase.