okareo.checks

BaseCheck Objects

class BaseCheck(ABC)

Base class for defining checks

evaluate

@staticmethod
@abstractmethod
def evaluate(model_output: str, scenario_input: str, scenario_result: str,
             metadata: dict) -> Union[bool, int, float]

Evaluate your model output, scenario input, scenario result, and metadata to determine if the data should pass or fail the check.

check_config

def check_config() -> dict

Returns a dictionary of configuration parameters that will be passed to the API.

CheckOutputType Objects

class CheckOutputType(Enum)

Enum for the type of output that the check will produce. This is used in ModelBasedCheck.

ModelBasedCheck Objects

class ModelBasedCheck(BaseCheck)

Check that uses a prompt template to evaluate the data.

The prompt template should be a string that includes at least one of the following placeholders, which will be replaced with the actual values:

• {model_input} -> corresponds to the model's input
• {generation} -> corresponds to the model's output
• {scenario_input} -> corresponds to the scenario input
• {scenario_result} -> corresponds to the scenario result

Example of how a template could be used: "Count the words in the following: {generation}"

The check output type should be one of the following:

• CheckOutputType.SCORE -> this template should ask prompt the model a score (single number)
• CheckOutputType.PASS_FAIL -> this template should prompt the model for a boolean value (True/False)

__init__

def __init__(prompt_template: str, check_type: CheckOutputType)

Initialize the check with a prompt template and check type

CodeBasedCheck Objects

class CodeBasedCheck(BaseCheck)

A check that uses code to evaluate the data

To use this check:

1. Create a new Python file (not in a notebook).
2. In this file, define a class named 'Check' that inherits from CodeBasedCheck.
3. Implement the evaluate method in your Check class.
4. Include any additional code used by your check in the same file.

Example:

# In my_custom_check.py
from okareo.checks import CodeBasedCheck

class Check(CodeBasedCheck):
    @staticmethod
    @abstractmethod
    def evaluate(
        model_output: str, scenario_input: str, scenario_result: str, metadata: dict
    ) -> Union[bool, int, float]:
        # Your code here
        pass