Skip to main content

CircleCI

The Okareo CLI is easy to use within the CircleCI ecosystem. You can use it for CI/CD including synthetic scenario generation, evaluation, and deployment readiness.

You can also use the CLI to drive fine tuning pipelines directely from CircleCI. To learn more about the Okareo CLI, please refer to the CLI documentation.

tip

The SDK requires an API Token. Refer to the Okareo API Key guide for more information.

Usage​

The Okareo CLI provides easy access to the Okareo SDKs and APIs. It does not presume a specific language/version. As a result, make sure to setup either Typescript or Python as part of your workflow steps.

The following example would appear in your .circleci/config.yml file.

version: 2.1

orbs:
  node: circleci/node@5.0.2

jobs:
  summarization:
    executor: node/default
    working_directory: ~/cookbook/demos
    steps:
      - checkout:
          path: ~/cookbook
      - run:
          name: Install Okareo CLI
          command: |
              curl -O -L https://github.com/okareo-ai/okareo-cli/releases/download/v0.0.14/okareo_0.0.14_linux_386.tar.gz
              tar -xvf okareo_0.0.14_linux_386.tar.gz
              chmod +x bin/okareo
              echo 'export PATH=$PATH:$PWD/bin' >> $BASH_ENV
      - run:
          name: Check Okareo Version
          command: okareo --version
      - run:
          name: Run Summarization
          command: okareo run -f summarization

  reporting:
    executor: node/default
    working_directory: ~/cookbook/demos
    steps:
      - checkout:
          path: ~/cookbook
      - run:
          name: Install Okareo CLI
          command: |
              curl -O -L https://github.com/okareo-ai/okareo-cli/releases/download/v0.0.14/okareo_0.0.14_linux_386.tar.gz
              tar -xvf okareo_0.0.14_linux_386.tar.gz
              chmod +x bin/okareo
              echo 'export PATH=$PATH:$PWD/bin' >> $BASH_ENV
      - run:
          name: Check Okareo Version
          command: okareo --version
      - run:
          name: Summarization Report
          command: okareo run -f x_summarization_logger

workflows:
  version: 2
  build:
    jobs:
      - summarization:
          context:
            - Demo-Env
      - reporting:
          requires:
            - "summarization"
          context:
            - Demo-Env
info

Other LLM API keys can be passed using the standard CircleCI context mechanism.

Workflow Options​

By install the Okareo CLI, you have full access to the APIs and SDKs of Okareo. See okareo run in examples above. You can pass a variety of options to the okareo CLI command.

In addition to your .circleci/config.yml file Okareo CLI expects the following directory structure to find flow scripts to execute. Under the flows/ directory there could be Python (.py) or Typescript (.ts) scripts that Okareo CLI will execute.

[repo root]/
│
└─ .okareo/
   │
   ├─ config.yml
   │
   └─ flows/
      │
      └─ [your_flow_script].[py|ts]

You could run okareo init -l typescript or okareo init -l python to initialize the above directory structure within your VCS repo using your language of preference. See CLI documentation on how to install the CLI and options it supports.

Viewing Results in Okareo​

You can view the results of your synthetic scenario generations and model evaluations in https://app.okareo.com/. Action stdout will typically include direct links to newly generated scenarios in format similar to:

https://app.okareo.com/project/<project UUID>/scenario/<scenario UUID>

or for evaluations:

https://app.okareo.com/project/<project UUID>/eval/<evaluation UUID>

Reporters (Typescript)​

The Typescript SDK provides reporters that can provide information directly into the CI console. This provides high level visibility of evaluation results within CircleCI.

Example Summarization console output for a failing test: Okareo Diagram

Example Retrieval Report Okareo Diagram

Learn more about reporters in the Typescript SDK.