GitHub Action

The Okareo GitHub Action provides easy access to the Okareo CLI and APIs. You can use it for CI/CD including synthetic scenario generation, evaluation, and deployment readiness.

You can also use the action to drive fine tuning pipelines directely from GitHub. To learn more about the Okareo CLI, please refer to the CLI documentation.

tip

The SDK requires an API Token. Refer to the Okareo API Key guide for more information.

Usage

The Okareo GitHub Action provides access to the Okareo APIs and SDKs. It does not presume a specific language/version. As a result, make sure to setup either Typescript or Python as part of your workflow steps. In the case of GitHub, node is more generally available. However if you want to use a specific version, you will need to configure that in the workflow.

The following example would appear in your .github/workflows/workflow.yml or similar file:

Typescript

name: Typescript Example
env:
  SKIP_YARN_COREPACK_CHECK: true
  PROJECT_BUILD_ID: ${{ github.run_number }}
  OKAREO_API_KEY: ${{ secrets.OKAREO_API_KEY }}
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  
on:
  push:
    branches:
      - main

jobs:
  summarization:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./demos
    permissions:
      contents: 'read'
      id-token: 'write'
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Okareo Action
        uses: okareo-ai/okareo-action@v2.5
      
      - name: Summarization Evaluation
        run: |
          okareo -v
          okareo run -f summarization

Python

name: Python Example
  env:
    PROJECT_BUILD_ID: ${{ github.run_number }}
    OKAREO_API_KEY: ${{ secrets.OKAREO_API_KEY }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    
  on:
    push:
      branches:
        - main

  jobs:
    cookbook-python:
      runs-on: ubuntu-latest
      defaults:
        run:
          working-directory: ./cookbook/py-minimal-ci
      permissions:
        contents: 'read'
        id-token: 'write'
      steps:
        - name: Checkout
          uses: actions/checkout@v4
          
        - name: Set up Python
          # This is the version of the action for setting up Python, not the Python version.
          uses: actions/setup-python@v5
          with:
            python-version: '3.x'

        - name: Okareo Action
          # Notice the specific version of the Okareo GitHub Action you want to use
          uses: okareo-ai/okareo-action@v2.5
        
        - name: Python Example
          run: |
            okareo -v
            okareo run

Other LLM API keys can also be passed in via environment variables:

steps:
- uses: actions/checkout@v4
- uses: okareo-ai/okareo-action@v2
- name: Run Validation
  run: okareo run
  env:
    OKAREO_API_KEY: ${{ secrets.OKAREO_API_KEY }} # required
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} # optional for models you build with

Workflow Options

This GitHub action sets up the right environment and runs Okareo CLI. See okareo run in examples above. You can pass a variety of options to the okareo CLI command.

In addition to your .github/workflows/workflow.yml file Okareo CLI expects the following directory structure to find flow scripts to execute. Under the flows/ directory there could be Python (.py) or Typescript (.ts) scripts that Okareo CLI will execute.

[repo root]/
│
└─ .okareo/
   │
   ├─ config.yml
   │
   └─ flows/
      │
      └─ [your_flow_script].[py|ts]

You could run okareo init -l typescript or okareo init -l python to initialize the above directory structure within your GitHub repo with your language of preference. See CLI documentation on how to install the CLI and options it supports.

Viewing Results in Okareo

You can view the results of your synthetic scenario generations and model evaluations in https://app.okareo.com/. Action stdout will typically include direct links to newly generated scenarios in format similar to:

https://app.okareo.com/project/<project UUID>/scenario/<scenario UUID>

or for evaluations:

https://app.okareo.com/project/<project UUID>/eval/<evaluation UUID>

Reporters (Typescript)

The Typescript SDK provides reporters that can provide information directly into the CI console. This provides high level visibility of evaluation results within GitHub.

Example Summarization console output for a failing test:

Example Retrieval Report

Learn more about reporters in the Typescript SDK.