Skip to main content

Okareo CLI

The Okareo CLI makes it easy to run model evaluation and scenario workflows in Python or TypeScript—without changing your core codebase. Designed for local use and CI/CD pipelines, it helps you validate AI behavior across flows defined in a lightweight config. With a single command, you can coordinate tests, generate reports, and even proxy model requests for runtime evaluation. The proxy supports runtime and production LLM error tracking that can feed directly into evaluations. Whether you’re iterating locally or automating checks before a prod push, the CLI keeps model validation clean, versioned, and code-native.

tip

The SDK requires an API Token. Refer to the Okareo API Token guide for more information.

Install​

Download the latest version of the okareo CLI (okareo) for your development environment.

curl -O -L https://github.com/okareo-ai/okareo-cli/releases/latest/download/okareo_darwin_arm64.tar.gz
tar -xvf okareo_darwin_arm64.tar.gz

Add Okareo to your path after unpacking:

export PATH="$PATH:[LOCAL_PATH_WHERE_YOU_UNPACKED_OKAREO]"

Run okareo -v to verify your installation before moving to next step.

If you want to use Okareo, outside of a broader unit test framework, you can use the okareo init command to bootstrap in Python or Typescript. Learn more

okareo init --language typescript

Project Structure & Configuration​

To run scenario creation and evaluation scripts Okareo CLI assumes a specific directory structure. you will need to establish a .okareo folder that incldues config.yml file and a flows folder with the scripts you want to run.

[Project]
  └── .okareo
  └──── config.yml
  └──── flows
  └────── [your_flow_script].[py|ts]

To use the Okareo CLI, you will need to establish a .okareo folder with a config.yml file inside.

The ./.okareo/config.yml file can be configured directly or for scripts written in python, javascript, or typescript

name: Project Name 
api-key: ${OKAREO_API_KEY}
project-id: ${OKAREO_PROJECT_ID}
run:
  flows:
    config:
      - name: "Your Evaluation Name"
        model-id: "<MODEL-UUID>"
        scenario-id: "<SCENARIO-UUID>"    
        type: "NL_GENERATION" | "INFORMATION_RETRIEVAL" | "MULTI_CLASS_CLASSIFICATION"
        checks:
          - "<CHECK-NAME-1>"
          - "<CHECK-NAME-2>"
          ...
          - "<CHECK-NAME-N>"
tip

On the roadmap is a facility to split and run flows in parallel. Although a seemingly simple feature, there are interesting consequences - like model warming which causes answers to change if topical questions are asked in sequence. If this is of interest, please reach out to us product@okareo.com

CLI Commands​

-h, --help​

The Okareo CLI includes help. You can get help via --help at the CLI root or on a per command basis.

machine-dev % okareo --help
The Okareo CLI is a tool to help you evaluate your use of AI/ML in your application:
To use the CLI, refer to the docs: https://docs.okareo.com/docs/sdk/cli

Usage:
  okareo [flags]
  okareo [command]

Available Commands:
  clean       Okareo CLI command to clean .okareo directory
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  init        Creates a default .okareo structure in the current directory
  proxy       Start a proxy server
  run         Okareo CLI command to run workflows.

Flags:
  -h, --help      help for okareo
  -v, --version   The current version of the Okareo CLI (default true)

-c, --clean​

The Okareo CLI bootstraps the Typescript and Javascript environment within ./.okareo.

To avoid accidentally checking in this generated code, we suggest adding a few entries to .gitignore per the Typescript and Javascript install section.

Clean specifically removes the following items:

  • .okareo/package-lock.json
  • .okareo/tsconfig.json if using typescript
  • .okareo/node_modules
  • .okareo/dist if using typescript

    This mechanism is not needed if you are using Python.

init​

Creates a default .okareo structure in the current directory

Usage:
  okareo init [flags]

Flags:
  -d, --debug             See additional stdout to debug the init process.
  -f, --force             Forces the exiting configuration to be overwritten.
  -h, --help              help for init
  -l, --language string   The language you want to configure: Python, Javascript, or Typescript. (default "python")

Global Flags:
  -v, --version   The current version of the Okareo CLI (default true)

init -d, --debug​

Provides additional information for the init runtime.

okareo init -d

init -f, --force​

This will replace the existing configuration with a new configuration file. This will NOT remove existing flows.

okareo init --force

init -l, --language​

This initializes the .okareo environment with the specified language : Python, Javascript, or Typescript.

okareo init -l ts
tip

If you are interested in an additional language, let us know: product@okareo.com


proxy​

Starts a proxy server that can handle LLM requests 

Usage:
  okareo proxy [flags]

Flags:
  -c, --config string   Path to config file (default "./cmd/proxy_config.yaml")
  -d, --debug           Enable debug mode
  -h, --help            help for proxy
  -H, --host string     Host to run the proxy server on (default "0.0.0.0")
  -m, --model string    Model to use (e.g., gpt-3.5-turbo, claude-2)
  -p, --port string     Port to run the proxy server on (default "4000")

Global Flags:
  -v, --version   The current version of the Okareo CLI (default true)

proxy -c, --config string​

To easily support runtime error tracking and evaluations, the CLI provides a proxy. To run the proxy, you will need to provide your OKAREO_API_KEY, and any other standard model keys you wan to use.

You will need to point your model base_url to the proxy endpoint. The proxy runs on port 4000 by default. You can change this with -p 4000

Example of using OpenAI with the proxy base_url in place.

client = openai.OpenAI(
  base_url="http://localhost:4000"
)


run​

The run command is the primary feature of the Okareo CLI. The run command will call each script that matches the file pattern in .okareo/config.yml.

This command is useful locally during development and can be used directly or can also run in CI/CD using the official Okareo GitHub Action.

If the evaluation config is in the yml or the SDK uses the JSONReporter.log(), run will automatically ouput evaluations results to ./.okareo/reports

machine-dev % okareo run --help
Okareo CLI 'runs' can include multiple flows that perform a variety of tasks from scenario generation to model evaluation.

Usage:
  okareo run [flags]

Flags:
  -c, --config string   The Okareo configuration file for the evaluation run. (default "./.okareo/config.yml")
  -d, --debug           See additional stdout to debug your flows.
  -f, --file string     The Okareo flow script you want to run. (default "ALL")
  -h, --help            help for run
  -r, --reports string  The folder where eval results are made available. Defaults to ./.okareo/reports/ 

Global Flags:
  -v, --version   The current version of the Okareo CLI (default true)

run -c, --config​

With this you can create bespoke config files with different matching patterns.

okareo run -c ./.okareo/retrieval.config.yml

A few example configurations that you could create include:

- all.config.yml could macth all flows - intent.config.yml could match flows for evaulating intent - ci.config.yml could match flows you want to run in CI/CD - prod.config.yml could match flows necessary for automated validation prior to production push


run -d, --debug​

If you are concerned that flows are being skipped or would like more information, --debug provides more detailed output.

This facility is primarily used by Okareo during development of the CLI but is perfectly safe for daily use as well.

okareo run -d

run -f, --file​

There are times during development when you will want to run a specific flow to drive and evaluate finetuning. The file flag allows you to specify the flow you want to run.

tip

The --file command takes the name of the flow you want to run without the file type designation. For example projects vs projects.py. This is a conscious decision that abstracts the language from the flow. Because Okareo allows more than one config file to exist, it is possible to have more than one language available in a single .okareo environment.


run -h, --help​

Provides help output on run.

okareo run -h

run -r, --reports string​

The CLI/SDK default to output evaluation reports to ./.okareo/reports. However if an alternative location is preferred, this flag can set the location to use for the output.

okareo run -r analytics

This will output evaluation reports to ./.okareo/analytics. Specify an absolute path to a directory if you'd like to save reports outside ./.okareo