Skip to main content

okareo.model_under_test

stop_listener_for_run​

def stop_listener_for_run(test_run_id: Union[str, UUID]) -> bool

Stop the custom-model listener answering this Run, if this process has one.

describe_listener_for_run​

def describe_listener_for_run(test_run_id: Union[str, UUID]) -> Optional[str]

This Run's listener state in a few words, or None if this process has none.

fetch_test_run​

def fetch_test_run(
        client: Client,
        api_key: str,
        test_run_id: Union[str, UUID],
        timeout_seconds: float = TEST_RUN_POLL_TIMEOUT_SECONDS) -> TestRunItem

GET one Run with its own request timeout, so a poll can never hang the caller.

The generated client is built without a timeout (a long run_test needs none), so the timeout goes on this request alone. Not Client.with_timeout: that mutates the already-built shared httpx client, which would give run_test's hour-long POST a 30 s read timeout.

fetch_test_run_async​

async def fetch_test_run_async(
        client: Client,
        api_key: str,
        test_run_id: Union[str, UUID],
        timeout_seconds: float = TEST_RUN_POLL_TIMEOUT_SECONDS) -> TestRunItem

The listener's own copy of fetch_test_run, for use inside its event loop.

A fresh AsyncClient per call: the generated client's async client binds to the first loop that uses it, and this runs on the listener's private loop. No thread pool either: at interpreter exit the default executor refuses new work, and a watchdog that depended on it could never see the Run end.

wait_for_test_run​

def wait_for_test_run(
    fetch: Callable[[], TestRunItem],
    test_run_id: Union[str, UUID],
    poll_interval: float = 10.0,
    timeout: Optional[float] = None,
    listener_state: Optional[Callable[[],
                                      Optional[str]]] = None) -> TestRunItem

Poll a submitted Run with short requests until it is FINISHED or FAILED.

Returns the terminal Run; the caller decides what FAILED means. A poll that fails is logged and retried, never fatal on its own. Raises TestRunError only when timeout seconds pass without a terminal status.

ModelUnderTest Objects​

class ModelUnderTest(AsyncProcessorMixin)

A class for managing a Model Under Test (MUT) in Okareo. Returned by okareo.register_model()

submit_test​

def submit_test(
        scenario: Union[ScenarioSetResponse, str, UUID],
        name: str,
        api_key: Optional[str] = None,
        api_keys: Optional[dict] = None,
        metrics_kwargs: Optional[dict] = None,
        test_run_type: TestRunType = TestRunType.MULTI_CLASS_CLASSIFICATION,
        calculate_metrics: bool = True,
        checks: Optional[List[str]] = None,
        simulation_params: Optional[Any] = None,
        driver_id: Optional[str] = None,
        tags: Optional[List[str]] = None) -> TestRunItem

Asynchronous server-based version of test-run execution. For CustomModels, model invocations are handled client-side in a background thread then evaluated server-side asynchronously. For other models, model invocations and evaluation are both handled server-side asynchronously.

For custom multi-turn Targets (CustomMultiturnTarget, CustomMultiturnTargetAsync) the client-side listener thread keeps answering the server's turns after this call returns. Call wait_for_test_run with the returned id to block until the Run is FINISHED or FAILED; it stops the listener cleanly. The listener is a daemon thread, so the process must stay alive for the whole Run: exiting early fails the Run's remaining turns.

Arguments:

  • scenario Union[ScenarioSetResponse, str] - The scenario set or identifier to use for the test run.
  • name str - The name to assign to the test run.
  • api_key Optional[str] - Optional API key for authentication.
  • api_keys Optional[dict] - Optional dictionary of API keys for different services.
  • metrics_kwargs Optional[dict] - Optional dictionary of keyword arguments for metrics calculation.
  • test_run_type TestRunType - The type of test run to execute. Defaults to MULTI_CLASS_CLASSIFICATION.
  • calculate_metrics bool - Whether to calculate metrics after the test run. Defaults to True.
  • checks Optional[List[str]] - Optional list of checks to perform during the test run.
  • tags Optional[List[str]] - Optional tags to set on the created test run. These are persisted on the test run itself and can be used with scenario0.

Returns:

  • scenario1 - The resulting test run item for the submitted test run. The scenario2 field can be used to retrieve the test run.

run_test​

def run_test(
        scenario: Union[ScenarioSetResponse, str, UUID],
        name: str,
        api_key: Optional[str] = None,
        api_keys: Optional[dict] = None,
        metrics_kwargs: Optional[dict] = None,
        test_run_type: TestRunType = TestRunType.MULTI_CLASS_CLASSIFICATION,
        calculate_metrics: bool = True,
        checks: Optional[List[str]] = None,
        simulation_params: Optional[Any] = None,
        driver_id: Optional[str] = None,
        tags: Optional[List[str]] = None) -> TestRunItem

Server-based version of test-run execution. For CustomModels, model invocations are handled client-side then evaluated server-side. For other models, model invocations and evaluations handled server-side.

Arguments:

  • scenario Union[ScenarioSetResponse, str] - The scenario set or identifier to use for the test run.
  • name str - The name to assign to the test run.
  • api_key Optional[str] - Optional API key for authentication.
  • api_keys Optional[dict] - Optional dictionary of API keys for different services.
  • metrics_kwargs Optional[dict] - Optional dictionary of keyword arguments for metrics calculation.
  • test_run_type TestRunType - The type of test run to execute. Defaults to MULTI_CLASS_CLASSIFICATION.
  • calculate_metrics bool - Whether to calculate metrics after the test run. Defaults to True.
  • checks Optional[List[str]] - Optional list of checks to perform during the test run.
  • tags Optional[List[str]] - Optional tags to set on the created test run. These are persisted on the test run itself and can be used with Okareo.find_test_runs(tags=...).

Returns:

  • name0 - The resulting test run item for the completed test run.

wait_for_test_run​

def wait_for_test_run(test_run_id: Union[str, UUID],
                      poll_interval: float = 10.0,
                      timeout: Optional[float] = None) -> TestRunItem

Block until a submitted Run is FINISHED, polling with short requests.

Each poll is one GET with its own timeout; a failed poll is logged and retried. Logs every poll, and one line if the Run passes 300 seconds.

Raises:

  • TestRunError - the Run ended FAILED (with the server's failure message), or timeout seconds passed without a terminal status. On a timeout the Run may still be going, so its listener is left running; call again to keep waiting.

get_test_run​

def get_test_run(test_run_id: Union[str, UUID]) -> TestRunItem

Retrieve a test run by its ID.

Arguments:

  • test_run_id str - The ID of the test run to retrieve.

Returns:

  • TestRunItem - The test run item corresponding to the provided ID.

ModelInvocation Objects​

@_attrs_define
class ModelInvocation()

Model invocation response object returned from a CustomModel.invoke method or as an element of a list returned from a CustomBatchModel.invoke_batch method.

Arguments:

  • model_prediction - Prediction from the model to be used when running the evaluation, e.g. predicted class from classification model or generated text completion from a generative model. This would typically be parsed out of the overall model_output_metadata.
  • model_input - All the input sent to the model.
  • model_output_metadata - Full model response, including any metadata returned with model's output.
  • tool_calls - List of tool calls made during the model invocation, if any.

OpenAIModel Objects​

@define
class OpenAIModel(BaseModel)

An OpenAI model definition with prompt template and relevant parameters for an Okareo evaluation.

Arguments:

  • model_id - Model ID to request from OpenAI completion. For list of available models, see https://platform.openai.com/docs/models
  • temperature - Parameter for controlling the randomness of the model's output.
  • system_prompt_template - System role prompt template to pass to the model. Uses mustache syntax for variable substitution, e.g. {scenario_input}.
  • user_prompt_template - User role prompt template to pass to the model. Uses mustache syntax for variable substitution, e.g. {scenario_input}
  • dialog_template - Dialog template in OpenAI message format to pass to the model. Uses mustache syntax for variable substitution.
  • tools - List of tools to pass to the model.

GenerationModel Objects​

@define
class GenerationModel(BaseModel)

An LLM definition with prompt template and relevant parameters for an Okareo evaluation.

Arguments:

  • model_id - Model ID to request for LLM completion.
  • temperature - Parameter for controlling the randomness of the model's output.
  • system_prompt_template - System role prompt template to pass to the model. Uses mustache syntax for variable substitution, e.g. {scenario_input}.
  • user_prompt_template - User role prompt template to pass to the model. Uses mustache syntax for variable substitution, e.g. {scenario_input}
  • dialog_template - Dialog template in OpenAI message format to pass to the model. Uses mustache syntax for variable substitution.
  • tools - List of tools to pass to the model.

CohereModel Objects​

@_attrs_define
class CohereModel(BaseModel)

A Cohere model definition with prompt template and relevant parameters for an Okareo evaluation.

Arguments:

PineconeDb Objects​

@_attrs_define
class PineconeDb(BaseModel)

A Pinecone vector database configuration for use in an Okareo retrieval evaluation.

Arguments:

  • index_name - The name of the Pinecone index to connect to.
  • region - The region where the Pinecone index is hosted.
  • project_id - The project identifier associated with the Pinecone index.
  • top_k - The number of top results to retrieve for queries. Defaults to 5.

QdrantDB Objects​

@_attrs_define
class QdrantDB(BaseModel)

A Qdrant vector database configuration for use in an Okareo retrieval evaluation.

Arguments:

  • collection_name - The name of the Qdrant collection to connect to.
  • url - The URL of the Qdrant instance.
  • top_k - The number of top results to retrieve for queries. Defaults to 5.
  • sparse - Whether to use sparse vectors for the Qdrant collection. Defaults to False.

CustomModel Objects​

@_attrs_define
class CustomModel(BaseModel)

A custom model definition for an Okareo evaluation. Requires a valid invoke definition that operates on a single input.

Arguments:

  • name - A name for the custom model.

invoke​

@abstractmethod
def invoke(input_value: Union[dict, list, str]) -> Union[ModelInvocation, Any]

Method for taking a single scenario input and returning a single model output

Arguments:

  • input_value - Union[dict, list, str] - input to the model.

Returns:

Union[ModelInvocation, Any] - model output. If the model returns a ModelInvocation, it should contain the model's prediction, input, and metadata. If the model returns a tuple, the first element should be the model's prediction and the second element should be the metadata.

CustomMultiturnTarget Objects​

@_attrs_define
class CustomMultiturnTarget(BaseModel)

A custom model definition for an Okareo multiturn evaluation. Requires a valid invoke definition that operates on a single turn of a converstation.

start_session​

def start_session(
    scenario_input: str | None = None
) -> tuple[str | None, ModelInvocation | None]

Method for starting a multiturn conversation with a custom model

Returns:

  • str | None: session_id - the ID of the session started by the model.
  • ModelInvocation | None: model output - the model's response to the session start, if any.

end_session​

def end_session(session_id: str) -> None

Method for ending a multiturn conversation with a custom model

Arguments:

  • session_id - str - the ID of the session to end.

invoke​

@abstractmethod
def invoke(messages: List[dict[str, str]],
           scenario_input: Optional[Union[dict, list, str]] = None,
           session_id: Optional[str] = None) -> Union[ModelInvocation, Any]

Method for continuing a multiturn conversation with a custom model

Arguments:

  • messages - list - list of messages in the conversation
  • scenario_input - Optional[dict | list | str] - scenario input for the conversation

Returns:

Union[ModelInvocation, Any] - model output. If the model returns a ModelInvocation, it should contain the model's prediction, input, and metadata. If the model returns a tuple, the first element should be the model's prediction and the second element should be the metadata.

CustomMultiturnTargetAsync Objects​

@_attrs_define
class CustomMultiturnTargetAsync(BaseModel)

A custom model definition for an Okareo multiturn evaluation that uses asynchronous methods. Requires a valid invoke definition that operates on a single turn of a converstation.

start_session​

async def start_session(
    scenario_input: str | None = None
) -> tuple[str | None, ModelInvocation | None]

Method for starting a multiturn conversation with a custom model

Returns:

  • str | None: session_id - the ID of the session started by the model.
  • ModelInvocation | None: model output - the model's response to the session start, if any.

end_session​

async def end_session(session_id: str) -> None

Method for ending a multiturn conversation with a custom model

Arguments:

  • session_id - str - the ID of the session to end.

invoke​

@abstractmethod
async def invoke(
    messages: List[dict[str, str]],
    scenario_input: Optional[Union[dict, list, str]] = None,
    session_id: Optional[str] = None
) -> Awaitable[Union[ModelInvocation, Any]]

Method for continuing a multiturn conversation with a custom model

Arguments:

  • messages - list - list of messages in the conversation
  • scenario_input - Optional[dict | list | str] - scenario input for the conversation

Returns:

Union[ModelInvocation, Any] - model output. If the model returns a ModelInvocation, it should contain the model's prediction, input, and metadata. If the model returns a tuple, the first element should be the model's prediction and the second element should be the metadata.

VoiceTarget Objects​

class VoiceTarget(BaseModel)

Base class for realtime voice targets in Okareo multiturn simulation.

Voice targets are used as Target.target when calling Okareo.run_simulation(...). They execute server-side and follow the same API key pattern as other targets (pass keys via api_keys).

Subclasses define provider-specific fields and implement params().

OpenAIVoiceTarget Objects​

@_attrs_define
class OpenAIVoiceTarget(VoiceTarget)

OpenAI Realtime API voice target for Okareo multiturn evaluation.

Arguments:

  • model - Model ID for OpenAI Realtime. Default: "gpt-realtime".
  • instructions - System instructions for the voice agent. Default: "Be brief and helpful."
  • output_voice - Voice ID for TTS output. Options: "alloy", "echo", "fable", "onyx", "nova", "shimmer".

DeepgramVoiceTarget Objects​

@_attrs_define
class DeepgramVoiceTarget(VoiceTarget)

Deepgram voice target for Okareo multiturn evaluation.

Arguments:

  • model - Model ID for Deepgram. Default: "aura-2".
  • instructions - System instructions for the voice agent. Default: "Be brief and helpful."
  • output_voice - Voice ID for TTS output. Example: "aura-2-thalia-en".

TwilioVoiceTarget Objects​

@_attrs_define
class TwilioVoiceTarget(VoiceTarget)

Twilio-backed voice target for Okareo multiturn simulation.

Arguments:

  • account_sid - Twilio Account SID.
  • auth_token - Twilio auth token (treated as sensitive).
  • from_phone_number - Outbound Twilio phone number.
  • to_phone_number - Destination number to dial.
  • max_parallel_requests - Optional cap on concurrent calls.

PhoneTarget Objects​

@_attrs_define
class PhoneTarget(VoiceTarget)

Phone-number-only voice target for multiturn simulation.

Use this when Okareo should manage telephony details. You provide only the destination phone number.

Arguments:

  • phone_number - Destination phone number (E.164 format, e.g. "+15551234567").
  • max_parallel_requests - Cap on concurrent calls hitting the target.

SipTarget Objects​

@_attrs_define
class SipTarget(VoiceTarget)

Voice target reached over SIP.

Okareo places a call to sip_uri and runs a full-duplex voice simulation against the agent that answers. Use this to test any voice agent reachable at a SIP URI — for example an agent fronted by Daily, Vapi, LiveKit, or a SIP trunk.

Arguments:

  • sip_uri - Destination SIP URI, e.g. "sip:agent@your-domain.example.com".
  • sip_username - Optional SIP authentication username for the target.
  • sip_password - Optional SIP authentication password for the target.
  • max_parallel_requests - Cap on concurrent calls hitting the target.
  • sip_mode - How the call is placed. Default (unset) routes through Okareo's telephony provider. "direct" makes Okareo the SIP client: it sends the INVITE and carries the audio itself — no telephony provider in the path. Requires a target reachable at a plain sip: URI over UDP with symmetric RTP (modern platforms such as LiveKit, Vapi, Daily, and Telnyx qualify).
  • sip_from_user - Direct mode only — user part of the From/caller identity. Default "okareo".
  • ``0 - Direct mode only — offered codec: "pcmu" (default), "pcma", or "opus".
  • ``1 - Direct mode only — extra headers for the INVITE, e.g.
  • ````3 - "abc"}``.
  • 5 - Direct mode only — STUN server used for NAT discovery, as "stun:host:port"``. Defaults server-side; not normally set.
  • ``8 - Direct mode only — seconds without inbound audio before the call is failed as one-way media. Defaults server-side.

Notes:

The direct-mode keys are emitted only when set, so existing targets serialize exactly as before. Server-side this maps onto sip_mode="direct" handling in the voice target factory — a cross-repo contract: renaming keys here requires a matching server change.

VonagePhoneTarget Objects​

@_attrs_define
class VonagePhoneTarget(VoiceTarget)

Vonage-backed voice target for Okareo multiturn simulation.

Sibling of PhoneTarget (Twilio) for Okareo's Vonage voice edge (edge_type="vonage"). Unlike PhoneTarget, Vonage credentials are caller-supplied (Vonage has no Okareo-managed-telephony mode yet), so this target also threads through application_id/private_key similar to how TwilioVoiceTarget threads through account_sid/auth_token. Vonage delivers mid-call DTMF out-of-band via RFC 4733 with no media-stream teardown.

Security note — create a dedicated Vonage Application for Okareo and pass only that application's private key. A Vonage keypair is scoped to a single Application (not your whole account), is generated locally so Vonage never sees it, and is revocable independently: regenerate the application's keypair or delete the application to cut access without touching the rest of your account.

Server contract — params() emits exactly these keys, consumed by the server's Vonage edge factory. This is a cross-repo contract — changing these keys requires a matching change server-side: type, edge_type, to_phone_number, from_phone_number, application_id, private_key, max_parallel_requests.

Recording (on), DTMF delivery (both out-of-band rfc4733 + in-band), and the 16 kHz media sample rate are fixed server-side defaults — not configurable from this target.

Arguments:

  • 5 - Destination phone number (E.164, e.g. "+15551234567"). Emitted as to_phone_numberinparams()``, mirroring PhoneTarget. Alias for PhoneTarget1 — provide either.
  • PhoneTarget1 - Same as ``5; takes precedence if both are set.
  • PhoneTarget4 - Outbound Vonage phone number.
  • application_id - Vonage application ID used to mint the call JWT.
  • private_key - Vonage application private key, PEM contents (treated as sensitive). Read it from your key file, e.g. private_key=Path("private.key").read_text().
  • PhoneTarget9 - Cap on concurrent calls hitting the target.

TelnyxPhoneTarget Objects​

@_attrs_define
class TelnyxPhoneTarget(VoiceTarget)

Telnyx-backed voice target for Okareo multiturn simulation.

Sibling of VonagePhoneTarget for Okareo's Telnyx voice edge (edge_type="telnyx"). Like Vonage, Telnyx credentials are caller-supplied (no Okareo-managed-telephony mode), but Telnyx authenticates with a single static Bearer API key rather than a JWT keypair, plus a connection_id (the Call Control Application the outbound call is placed from). Telnyx delivers mid-call DTMF out-of-band via RFC 2833 with no media-stream teardown.

Security note — create a dedicated Telnyx API key for Okareo (Telnyx supports multiple keys per account) so it can be rotated or revoked independently without touching the rest of your account.

Server contract — params() emits exactly these keys, consumed by the server's Telnyx edge factory. This is a cross-repo contract — changing these keys requires a matching change server-side: type, edge_type, to_phone_number, from_phone_number, telnyx_api_key, connection_id, max_parallel_requests.

Recording (on, dual-channel), DTMF delivery (both out-of-band rfc2833 + in-band), and the 8 kHz PCMU media format are fixed server-side defaults — not configurable from this target.

Arguments:

  • 1 - Destination phone number (E.164, e.g. "+15551234567"). Emitted as to_phone_numberinparams(), mirroring `VonagePhoneTarget`. Alias for 7 — provide either.
  • 7 - Same as 1; takes precedence if both are set.
  • ``0 - Outbound Telnyx phone number.
  • 1 - Telnyx API v2 key (Bearer; treated as sensitive). Named telnyx_api_key(notapi_key) because the server reserves the api_key`` param for the voice/TTS model key.
  • ``8 - Telnyx Call Control Application id the call is placed from.
  • ``9 - Cap on concurrent calls hitting the target.

StopConfig Objects​

@define
class StopConfig()

Configuration for stopping a multiturn conversation based on a specific check.

Arguments:

  • check_name - Name of the check to use for stopping the conversation.
  • stop_on - The check condition to stop the conversation. Defaults to True (i.e., conversation stops when check evaluates to True).

StreamingStopCondition Objects​

class StreamingStopCondition()

A condition that terminates the stream when matched.

Stop conditions use OR semantics — any single match ends the stream.

Arguments:

  • value - The value to match. Supports:
    • "true" / "false" (case-insensitive) for JSON booleans
    • "*" for presence check (any non-None value)
    • Any other string for exact string comparison
  • path - Optional response.-prefixed dot-bracket path into each parsed JSON chunk. E.g., response.is_finished. When omitted, value is matched as a raw string against the SSE data: payload before JSON parsing (e.g., "[DONE]").

StreamingSelectCondition Objects​

class StreamingSelectCondition()

A condition that filters which chunks have their content extracted.

Select conditions use AND semantics — all conditions must match for a chunk's content to be extracted.

Arguments:

  • path - Required response.-prefixed dot-bracket path into each parsed JSON chunk. E.g., response.role.
  • value - The value to match at that path. Same matching rules as
  • :class:StreamingStopCondition-"true"/"false"for booleans,"*"`` for presence, anything else for exact string comparison.

StreamingConfig Objects​

class StreamingConfig()

Configuration for streaming responses from a custom API endpoint.

When attached to TurnConfig or SessionConfig, the server can consume SSE or NDJSON streams and reassemble the final response text.

The text extraction path is configured on the parent TurnConfig or SessionConfig via response_message_path. For streaming responses, set it to your chunk shape (for example: response.choices[0].delta.content).

Arguments:

  • stop - List of StreamingStopCondition rules. The stream ends when any stop rule matches (OR semantics). A stop rule without a path matches as a raw string against each SSE data: payload, such as [DONE].
  • select - List of StreamingSelectCondition rules. All select rules must match before chunk content is extracted (AND semantics). If empty, content is extracted from every chunk.

SessionConfig Objects​

class SessionConfig()

Configuration for a custom API endpoint that starts a session.

Arguments:

  • url - URL of the endpoint to start the session.
  • method - HTTP method to use for the request. Defaults to POST.
  • headers - Headers to include in the request. Defaults to an empty JSON object.
  • body - Body to include in the request. Defaults to an empty JSON object.
  • status_code - Expected HTTP status code of the response.
  • response_session_id_path - Path to extract the session ID from the response. E.g., response.id will use the id field of the response JSON object to set the session_id.

TurnConfig Objects​

class TurnConfig()

Config for a custom API endpoint that continues a session by one turn.

Arguments:

  • url - URL of the endpoint to call for each next turn.
  • method - HTTP method to use. Defaults to POST.
  • headers - Headers to include in the request. Supports mustache substitution using latest_message, message_history, and session_id variables. Defaults to an empty JSON object.
  • body - Request body. Supports mustache substitution for latest_message, message_history, and session_id variables. Defaults to an empty JSON object.
  • status_code - Expected HTTP status code of the response.
  • response_message_path - Path to extract the model message from the response JSON (for example, response.completion.message.content).
  • response_session_id_path - Path to extract session ID from the response JSON (for example, response.result.contextId) so the same conversation session can continue across turns.
  • response_tool_calls_path - Path to extract tool calls from the response.

EndSessionConfig Objects​

class EndSessionConfig()

Configuration for a custom API endpoint that ends a session.

Arguments:

  • url - URL of the endpoint to start the session.
  • method - HTTP method to use for the request. Defaults to POST.
  • headers - Headers to include in the request. Defaults to an empty JSON object.
  • body - Body to include in the request. Defaults to an empty JSON object.
  • status_code - Expected HTTP status code of the response.
  • response_session_id_path - Path to extract the session ID from the response.

AuthConfig Objects​

class AuthConfig()

Configuration for a custom API endpoint that authenticates a session.

Arguments:

  • url - URL of the endpoint to authenticate.
  • method - HTTP method to use for the request. Defaults to POST.
  • headers - Headers to include in the request. Defaults to an empty JSON object.
  • body - Body to include in the request. Defaults to an empty JSON object.
  • status_code - Expected HTTP status code of the response.
  • response_access_token_path - Path to extract the access token from the response.

CustomEndpointTarget Objects​

class CustomEndpointTarget(BaseModel)

A trio of custom API endpoints for starting a session and continuing a conversation to use in Okareo multiturn evaluation.

Arguments:

  • start_session - A valid SessionConfig for starting a session.
  • next_turn - A valid TurnConfig for requesting and parsing the next turn of a conversation.
  • end_session - A valid EndSessionConfig for ending a session.
  • auth - A valid AuthConfig for authenticating a session.
  • max_parallel_requests - Maximum number of parallel requests to allow when running the evaluation.

Driver Objects​

@_attrs_define
class Driver()

Driver configuration used to simulate the caller side of a conversation.

Registered via Okareo.create_or_update_driver(...) and used by Okareo.run_simulation(...).

Target Objects​

@_attrs_define
class Target()

Named simulation target wrapper.

Wraps one target implementation (text or voice) so it can be created, retrieved, and referenced by name in Okareo.run_simulation(...).

Simulation Objects​

@_attrs_define
class Simulation()

Simulation runtime parameters for multiturn test runs.

Includes turn controls, stop conditions, and optional augmentation settings used by Okareo.run_simulation(...).

MultiTurnDriver Objects​

@_attrs_define
class MultiTurnDriver(BaseModel)

A driver model for Okareo multiturn evaluation.

Arguments:

  • target - Target model under test to use in the multiturn evaluation.
  • stop_check - A valid StopConfig or a dict that can be converted to StopConfig.
  • driver_model_id - Model ID to use for the driver model (e.g., "gpt-4.1").
  • driver_temperature - Parameter for controlling the randomness of the driver model's output.
  • repeats - Number of times to run a conversation per scenario row. Defaults to 1.
  • max_turns - Maximum number of turns to run in a conversation. Defaults to 5.
  • first_turn - Name of model (i.e., "target" or "driver") that should initiate each conversation. Defaults to "target".
  • driver_prompt_template - Optional system prompt template to pass to the driver model. Uses mustache syntax for variable substitution, e.g. {input}.

CustomBatchModel Objects​

@_attrs_define
class CustomBatchModel(BaseModel)

A custom batch model definition for an Okareo evaluation. Requires a valid invoke_batch definition that operates on a single input.

invoke_batch​

@abstractmethod
def invoke_batch(
    input_batch: list[dict[str, Union[dict, list, str]]]
) -> list[dict[str, Union[ModelInvocation, Any]]]

Method for taking a batch of scenario inputs and returning a corresponding batch of model outputs

Arguments:

  • input_batch - list[dict[str, Union[dict, list, str]]] - batch of inputs to the model. Expects a list of dicts of the format { 'id': str, 'input_value': Union[dict, list, str] }.

Returns:

List of dicts of format { 'id': str, 'model_invocation': Union[ModelInvocation, Any] }. 'id' must match the corresponding input_batch element's 'id'.