Skip to main content

okareo.checks

CheckResponse Objects​

@_attrs_define
class CheckResponse()

Response object for code-based checks.

Attributes:

  • score - The numeric or boolean result of the check.
  • explanation - Optional string explanation for the score, if applicable.
  • metadata - Optional dictionary containing metadata about the check. E.g., output tokens, input tokens, latency, cost, etc.

BaseCheck Objects​

class BaseCheck(ABC)

Base class for defining checks

evaluate​

@staticmethod
@abstractmethod
def evaluate(
    model_output: str, scenario_input: str, scenario_result: str,
    metadata: dict, model_input: str
) -> Union[bool, int, float, CheckResponse, tuple[Union[bool, int, float],
                                                  str]]

Evaluate your model output, scenario input, scenario result, metadata, and model_input to determine if the data should pass or fail the check.

check_config​

def check_config() -> dict

Returns a dictionary of configuration parameters that will be passed to the API.

CheckOutputType Objects​

class CheckOutputType(Enum)

Enum for the type of output that the check will produce. This is used in ModelBasedCheck.

ModelBasedCheck Objects​

class ModelBasedCheck(BaseCheck)

Check that uses a prompt template to evaluate the data.

The prompt template should be a string that includes at least one of the following placeholders, which will be replaced with the actual values:

  • {model_input} -> what was sent to the model (prompt or messages)
  • {model_output} -> the raw model output
  • {scenario_input} -> the input from your Scenario
  • {scenario_result} -> the expected result from your Scenario
  • {message_history} -> the row's full conversation, ending with the model's response
  • {tool_calls} -> the model's tool/function calls
  • {tools} -> the tool/function definitions available to the model
  • {model_output_metadata} -> extra output metadata (latency, tokens, cost, etc.)
  • {simulation_message_history} -> the dialog from the linked Okareo Simulation
  • {user_only_audio} -> user-only audio payload for speaker-scoped checks

Okareo rejects any placeholder not in the list above, including the legacy aliases. Replace {model_output}0 with {model_output}, {model_output}2 with {scenario_input}, and {model_output}4 with {scenario_result}. {model_output}6 and {model_output}7 have no replacement — remove them: audio is injected as multimodal content, not through a placeholder. A rejected prompt fails on save; it is not accepted and quietly ignored.

Example of how a template could be used: "Count the words in the following: {model_output}"

The check output type should be one of the following:

  • CheckOutputType.SCORE -> this template should ask prompt the model a score (single number)
  • CheckOutputType.PASS_FAIL -> this template should prompt the model for a boolean value (True/False)
  • CheckOutputType.ANALYSIS -> this template should prompt the model for free-form text rather than a score

The 'is_audio' flag indicates whether the check is meant to evaluate audio data. If True, the system will handle the audio data before passing it to the check, meaning you do not need to add any audio-specific prompt template placeholders for this check.

__init__​

def __init__(prompt_template: str,
             check_type: CheckOutputType,
             is_audio: bool = False)

Initialize the check with a prompt template and check type

CodeBasedCheck Objects​

class CodeBasedCheck(BaseCheck)

A check that uses code to evaluate the data

To use this check:

  1. Create a new Python file (not in a notebook).
  2. In this file, define a class named 'Check' that inherits from CodeBasedCheck.
  3. Implement the evaluate method in your Check class.
  4. Include any additional code used by your check in the same file.

The output type (pass/fail vs. score) is inferred by the server from the value your evaluate method returns, so you do not need to declare it.

Example:

# In my_custom_check.py
from okareo.checks import CodeBasedCheck, CheckResponse

class Check(CodeBasedCheck):
    @staticmethod
    def evaluate(
        model_output: str, scenario_input: str, scenario_result: str, metadata: dict, model_input: str
    ) -> CheckResponse:
        # Your code here
        return CheckResponse(score=True, explanation="This is an explanation of my score.")

The evaluate method parameters can be any subset of the possible parameters. For example, if your check only needs model_output and model_input, then you can write a method like this:

def evaluate(
    model_output: str, model_input: str
) -> CheckResponse:
    # ...your check logic here...