Okareo CLI
The Okareo CLI provides a means to run scenario/model workflows in python or typescript without modifying your larger project/repo. You can use the CLI locally or as a coordinator of scenario/model validation in CI.
The SDK requires an API Token. Refer to the Okareo API Key guide for more information.
Overview
The Okareo CLI assumes a specific config structure that incldues a config.yml
for context and a flows
folder with the flows you want to run.
[Project] .okareo config.yml flows [your_flow_script].[py|ts]
Install
Download the latest version of the okareo CLI (okareo
) for your
development environment.
- macOS Silicon
- Windows
- Linux
- All Releases
curl -O -L https://github.com/okareo-ai/okareo-cli/releases/download/v0.0.19/okareo_0.0.19_darwin_arm64.tar.gz
tar -xvf okareo_0.0.19_darwin_arm64.tar.gz
# Download and Extract the archive (requires tar in PowerShell 5.1+)
Invoke-WebRequest -Uri https://github.com/okareo-ai/okareo-cli/releases/download/v0.0.19/okareo_0.0.19_windows_386.tar.gz -OutFile okareo_0.0.19_windows_386.tar.gz
tar -xvf okareo_0.0.19_windows_386.tar.gz
curl -O -L https://github.com/okareo-ai/okareo-cli/releases/download/v0.0.19/okareo_0.0.19_linux_386.tar.gz
tar -xvf okareo_0.0.19_linux_386.tar.gz
You can find all the current releases at the Okareo-CLI github repo.
Add Okareo to your path after unpacking:
export PATH="$PATH:[LOCAL_PATH_WHERE_YOU_UNPACKED_OKAREO]/bin"
Run okareo -v
to verify your installation before moving to next step.
If you want to use Okareo, outside of a broader unit test framework, you can
use the okareo init
command to bootstrap in Python or Typescript. Learn
more
- Typescript
- Python
okareo init --language typescript
okareo init --language python
Config Structure
To use the Okareo CLI, you will need to establish a .okareo
folder with
a config.yml
file inside.
The ./.okareo/config.yml
file can be configured directly or for scripts
written in python, javascript, or typescript
- YAML
- Python
- Typescript
- Javascript
name: Project Name
api-key: ${OKAREO_API_KEY}
project-id: ${OKAREO_PROJECT_ID}
run:
flows:
config:
- name: "Your Evaluation Name"
model-id: "<MODEL-UUID>"
scenario-id: "<SCENARIO-UUID>"
type: "NL_GENERATION" | "INFORMATION_RETRIEVAL" | "MULTI_CLASS_CLASSIFICATION"
checks:
- "<CHECK-NAME-1>"
- "<CHECK-NAME-2>"
...
- "<CHECK-NAME-N>"
name: Project Name
api-key: ${OKAREO_API_KEY}
project-id: ${OKAREO_PROJECT_ID}
language: "python"
run:
flows:
file-pattern: '.*\.py'
Python configurations assume that you have a poetry or similar python environment already setup. Each script in the flows directory is run independently by the CLI.
name: Project Name
api-key: ${OKAREO_API_KEY}
project-id: ${OKAREO_PROJECT_ID}
language: "typescript"
run:
flows:
file-pattern: '.*\.ts'
Typescript needs to be transpiled to run. Okareo handles this for you by
injecting a minimal package.json and dist structure inside of .okareo/dist. If
you ever suspect there to be a problem, you can use okareo clean
to
automatically remove the Okareo compilation artifacts.
When using Typescript, we suggest that you add the following lines to your
root .gitignore
file.
.okareo/node_modules
.okareo/dist
.okareo/package*
.okareo/tsconfig.json
name: Project Name
api-key: ${OKAREO_API_KEY}
project-id: ${OKAREO_PROJECT_ID}
language: "javascript"
run:
flows:
file-pattern: '.*\.js'
Javascript requires an environment to run in. Okareo handles this for you by injecting a minimal package.json and dist structure inside of .okareo/dist.
If you ever suspect there to be a problem, you can use okareo clean
to automatically remove the Okareo compilation artifacts.
When using Javascript, we suggest that you add the following lines to your root .gitignore
file.
.okareo/node_modules
.okareo/package*
On the roadmap is a facility to split and run flows in parallel. Although a seemingly simple feature, there are interesting consequences - like model warming which causes answers to change if topical questions are asked in sequence. If this is of interest, please reach out to us product@okareo.com
CLI Commands
-h, --help
The Okareo CLI includes help. You can get help via --help
at the CLI root or on a per command basis.
machine-dev % okareo --help
The Okareo CLI is a tool to help you evaluate your use of AI/ML in your application:
To use the CLI, refer to the docs: https://okareo.com/docs/sdk/cli
Usage:
okareo [flags]
okareo [command]
Available Commands:
clean Okareo CLI command to clean .okareo directory
completion Generate the autocompletion script for the specified shell
help Help about any command
init Creates a default .okareo structure in the current directory
proxy Start a proxy server using litellm
run Okareo CLI command to run workflows.
Flags:
-h, --help help for okareo
-v, --version The current version of the Okareo CLI (default true)
-c, --clean
The Okareo CLI bootstraps the Typescript and Javascript environment within
./.okareo
.
To avoid accidentally checking in this generated code, we suggest adding a few entries to .gitignore per the Typescript and Javascript install section.
Clean specifically removes the following items:
.okareo/package.json
.okareo/package-lock.json
.okareo/tsconfig.json
if using typescript.okareo/node_modules
.okareo/dist
if using typescriptThis mechanism is not needed if you are using Python.
init
Creates a default .okareo structure in the current directory
Usage:
okareo init [flags]
Flags:
-d, --debug See additional stdout to debug the init process.
-f, --force Forces the exiting configuration to be overwritten.
-h, --help help for init
-l, --language string The language you want to configure: Python, Javascript, or Typescript. (default "python")
Global Flags:
-v, --version The current version of the Okareo CLI (default true)
init -d, --debug
Provides additional information for the init runtime.
okareo init -d
init -f, --force
This will replace the existing configuration with a new configuration file. This will NOT remove existing flows.
okareo init --force
init -l, --language
This initializes the .okareo environment with the specified language : Python, Javascript, or Typescript.
okareo init -l ts
If you are interested in an additional language, let us know: product@okareo.com
proxy
Starts a proxy server that can handle LLM requests using litellm's proxy functionality
Usage:
okareo proxy [flags]
Flags:
-c, --config string Path to config file (default "./cmd/proxy_config.yaml")
-d, --debug Enable debug mode
-h, --help help for proxy
-H, --host string Host to run the proxy server on (default "0.0.0.0")
-m, --model string Model to use (e.g., gpt-3.5-turbo, claude-2)
-p, --port string Port to run the proxy server on (default "4000")
Global Flags:
-v, --version The current version of the Okareo CLI (default true)
proxy -c, --config string
To easily support tracing, the CLI provides a proxy. To run the proxy, you
will need to provide your OKAREO_API_KEY
, and any other standard model
keys you wan to use.
The Okareo proxy uses LiteLLM. As a result, you can pass in any standard LiteLLM config format you wish.
You will need to point your model base_url
to the proxy endpoint. The
proxy runs on port 4000 by default. You can change this with -p 4000
okareo proxy -c ./config_proxy.yaml
Default config_proxy.yaml
file
model_list:
- model_name: "*"
litellm_params:
model: "*"
litellm_settings:
callbacks: ["otel"]
Example of using OpenAI with the proxy base_url
in place.
- Python
- Typescript
client = openai.OpenAI(
base_url="http://localhost:4000"
)
const openai = new OpenAI({
baseURL: 'http://localhost:4000',
});
run
The run
command is the primary feature of the Okareo CLI. The run
command will call each script that matches the file pattern in
.okareo/config.yml
.
This command is useful locally during development and can be used directly or through the Okareo GitHub Action to drive flows in CI/CD
If the evaluation config is in the yml or the SDK uses the JSONReporter.log(),
run
will automatically ouput evaluations results to ./.okareo/reports
machine-dev % okareo run --help
Okareo CLI 'runs' can include multiple flows that perform a variety of tasks from scenario generation to model evaluation.
Usage:
okareo run [flags]
Flags:
-c, --config string The Okareo configuration file for the evaluation run. (default "./.okareo/config.yml")
-d, --debug See additional stdout to debug your flows.
-f, --file string The Okareo flow script you want to run. (default "ALL")
-h, --help help for run
-r, --reports string The folder where eval results are made available. Defaults to ./.okareo/reports/ (default "reports")
Global Flags:
-v, --version The current version of the Okareo CLI (default true)
run -c, --config
With this you can create bespoke config files with different matching patterns.
okareo run -c ./.okareo/retrieval.config.yml
A few example configurations that you could create include:
-all.config.yml
could macth all flows - intent.config.yml
could match
flows for evaulating intent - ci.config.yml
could match flows you want to
run in CI/CD - prod.config.yml
could match flows necessary for automated
validation prior to production push
run -d, --debug
If you are concerned that flows are being skipped or would like more
information, --debug
provides more detailed output.
This facility is primarily used by Okareo during development of the CLI but is perfectly safe for daily use as well.
okareo run -d
run -f, --file
There are times during development when you will want to run a specific flow to drive and evaluate finetuning. The file flag allows you to specify the flow you want to run.
The --file command takes the name of the flow you want to run without the
file type designation. For example projects
vs projects.py
. This is a
conscious decision that abstracts the language from the flow. Because Okareo allows
more than one config file to exist, it is possible to have more than one language
available in a single .okareo
environment.
run -h, --help
Provides help output on run.
okareo run -h
run -r, --reports string
The CLI/SDK default to output evaluation reports to ./.okareo/reports
.
However if an alternative location is preferred, this flag can set the
location to use for the output.
okareo run -r analytics