Custom-Endpoint Multi-Turn Simulations

Okareo can drive a full conversation against your running service (RAG pipeline, tool-calling agent, or any HTTP API) by mapping requests & JSON responses to a Custom Endpoint Target. This guide shows you, step-by-step, how to run a multi-turn simulation using custom endpoints, in either the Okareo UI or SDK.

You'll follow the same four core steps you saw in the Multi-Turn Overview.

Cookbook examples for this guide are available:

• Colab Notebook
• TypeScript Example

1 · Define a Target agent profile

Okareo UI

1. Go to Multi-Turn Simulations → Settings.
2. Click ➕ New Target and choose “Custom Endpoint”.
3. Enter:
   • URL & HTTP method – e.g. POST https://api.example.com/v1/chat.
   • Headers / Query Params – add auth keys if needed.
   • Body Template – supports {session_id}, {latest_message}, {message_history[i:j]}.
   • Response Session-ID Path – JSONPath to the thread / session field.
   • Response Message Path – JSONPath to the assistant’s text.
4. Click Test Start Session to preview and adjust paths until the response is highlighted correctly.

Python

start_config = SessionConfig(
    url="https://api.example.com/v1/session",
    method="POST",
    headers={"Authorization": "Bearer <TOKEN>"},
    body={"userId": "<AGENT_ID>"},
    response_session_id_path="response.id",
)

next_config = TurnConfig(
    url="https://api.example.com/v1/session/messages",
    method="POST",
    headers={"Authorization": "Bearer <TOKEN>"},
    body={"sessionId": "{session_id}", "messages": "{message_history}"},
    response_message_path="response.messages.-1.content",
)

target = CustomEndpointTarget(start_config, next_config)

driver_model = MultiTurnDriver(
    target=target,
    driver_temperature=0.7,
    max_turns=6,
    stop_check=StopConfig(check_name="behavior_adherence"),
)

model = okareo.register_model(
    name="Custom Endpoint Demo Model",
    model=driver_model,
    update=True,
)

TypeScript

const startConfig: SessionConfig = {
    url: "https://api.example.com/v1/session",
    method: "POST",
    headers: {
        Authorization: "Bearer <TOKEN>",
        "Content-Type": "application/json",
    },
    body: { userId: "<AGENT_ID>" },
    response_session_id_path: "response.id",
};

const nextConfig: TurnConfig = {
    url: "https://api.example.com/v1/session/messages",
    method: "POST",
    headers: {
        Authorization: "Bearer <TOKEN>",
        "Content-Type": "application/json",
    },
    body: { sessionId: "{session_id}", messages: "{message_history}" },
    response_message_path: "response.messages.-1.content",
};

const target: CustomEndpointTarget = {
    start_config: startConfig,
    next_config: nextConfig,
};

const driverModel = new MultiTurnDriver({
    target,
    driver_temperature: 0.7,
    max_turns: 6,
    stop_check: { check_name: "behavior_adherence" },
});

const model = await okareo.register_model({
    name: "Custom Endpoint Demo Model",
    model: driverModel,
    update: true,
});

Driver Parameters

Parameter	Description
driver_temperature	Controls randomness of user/agent simulation
max_turns	Max back-and-forth messages
repeats	Repeats each test row to capture variance
first_turn	"driver" or "target" starts conversation
stop_check	Defines stopping condition (via check)

2 · Choose or Define a Driver Persona

Okareo UI

1. Switch to the Scenarios sub‑tab.
2. Click + New Scenario and fill in:
   • Driver Persona – e.g. “Confused shopper asking about returns”.
   • Expected Behaviors – what success looks like (“Explains policy & offers label”).

Python

seeds = [
    SeedData(
        input_="Hi, I need to return a pair of shoes. What do I do?",
        result="Agent explains return policy and offers a label.",
    ),
    SeedData(
        input_="Your site keeps crashing. Why?",
        result="Agent apologises and asks for details.",
    ),
]

scenario = okareo.create_scenario_set(
    ScenarioSetCreate(
        name="Return Policy & Stability",
        seed_data=seeds,
    )
)

TypeScript

const seeds = [
    {
        input: "Hi, I need to return a pair of shoes. What do I do?",
        result: "Agent explains return policy and offers a label.",
    },
    {
        input: "Your site keeps crashing. Why?",
        result: "Agent apologises and asks for details.",
    },
];

const scenario = await okareo.create_scenario_set({
    name: "Return Policy & Stability",
    seed_data: seeds,
});

3 · Launch a Simulation

Okareo UI

1. Switch to the Simulations sub-tab.
2. Click + New Simulation → select Target, Scenario, and Checks.
3. Click Run. You can watch the progress of the simulation.

Python

run = model.run_test(
    name="Endpoint Demo Run",
    scenario=scenario,
    test_run_type=TestRunType.MULTI_TURN,
    checks=["behavior_adherence"],
)

print("View the run ➜", run.app_link)

TypeScript

const testRun = await model.run_test({
    name: "Endpoint Demo Run",
    scenario_id: scenario.scenario_id,
    type: TestRunType.MULTI_TURN,
    checks: ["behavior_adherence"],
});

console.log("View the run ➜", testRun.app_link);

4 · Inspect Results

Click a Simulation tile to open its details. The results page breaks down the simulation into:

• Conversation Transcript – View the full back-and-forth between the Driver and Target, one turn per row.
• Checks – See results for:
  • Behavior Adherence – Did the assistant stay in character or follow instructions?
  • Model Refusal – Did the assistant properly decline off-topic or adversarial inputs?
  • Task Completed – Did it fulfill the main objective?
  • A custom check specific to your agent

Each turn is annotated with check results, so you can trace where things went wrong — or right.

---

That's it! You now have a complete, repeatable workflow for evaluating assistants with multi-turn simulations—entirely from the browser or your codebase.