Model Configuration

Okareo uses model.json to define which model providers are available, how to reach them, and which models serve as targets, evaluators, and drivers. In standalone mode, there is no hosted control plane to discover model connections -- model.json is the local source of truth.

The file is mounted read-only into the services container at /app/model.json.

Categories

Each top-level key in model.json maps to a role:

• target_models -- models you want to test and evaluate (the system under test).
• evaluation_models -- models used to grade and judge outputs (LLM-as-a-judge). A stronger evaluator yields more stable scoring.
• driver_models -- models used to generate synthetic prompts and simulations.

Field reference

Field	Description	Notes
providerName	Provider type	azure, vllm, openai
providerDisplayName	Friendly label shown in the Okareo UI	--
api_base	Provider endpoint URL	vLLM: http(s)://<host>:<port>/v1/ (trailing /v1/ required). Azure: https://<resource>.openai.azure.com/
api_version	API version	Azure only (e.g., 2024-12-01-preview)
api_key	API key reference	Recommended: name of an env var on the services container (e.g., EVAL_API_KEY). Discouraged: a literal key string.
models[].name	Provider-scoped model identifier	Azure: azure/<deployment-name>. vLLM: model name as served. OpenAI: gpt-4o, etc.
models[].displayName	Human-readable name in UI	--
models[].model_id	Reserved role ID	Evaluator: text_eval_model. Embeddings: text_embedding_eval_model. Driver: default_driver_model.

tip

Keep api_key as an env-var reference and store the actual secret only in your deployment secret mechanism.

Strong defaults

Most teams get the best first-run experience with OpenAI for targets + evaluation (if outbound access is allowed), or vLLM for fully private / air-gapped deployments.

Practical defaults:

Role	Recommended model
Targets	gpt-4o + gpt-4o-mini
Evaluator	gpt-4.1-mini
Embeddings	text-embedding-3-small
Driver	gpt-4.1-mini

tip

For a first install, pick one provider for targets + evaluator + driver to reduce moving parts.

Provider examples

Cloud providers (OpenAI + Azure evaluator)

{
  "target_models": [
    {
      "providerName": "openai",
      "providerDisplayName": "OpenAI",
      "api_key": "OPENAI_API_KEY",
      "models": [
        { "name": "gpt-4o", "displayName": "GPT-4o" },
        { "name": "gpt-4o-mini", "displayName": "GPT-4o mini" }
      ]
    }
  ],
  "evaluation_models": [
    {
      "providerName": "azure",
      "providerDisplayName": "Azure (Eval + Embeddings)",
      "api_key": "EVAL_API_KEY",
      "api_base": "https://{AZURE_RESOURCE_NAME}.openai.azure.com/",
      "api_version": "2024-12-01-preview",
      "models": [
        {
          "model_id": "text_eval_model",
          "name": "azure/{EVAL_DEPLOYMENT_NAME}",
          "displayName": "Evaluator"
        },
        {
          "model_id": "text_embedding_eval_model",
          "name": "azure/{EMBEDDING_DEPLOYMENT_NAME}",
          "displayName": "Embeddings"
        }
      ]
    }
  ],
  "driver_models": [
    {
      "providerName": "azure",
      "providerDisplayName": "Azure (Driver)",
      "api_key": "EVAL_API_KEY",
      "api_base": "https://{AZURE_RESOURCE_NAME}.openai.azure.com/",
      "api_version": "2024-12-01-preview",
      "models": [
        {
          "model_id": "default_driver_model",
          "name": "azure/{DRIVER_DEPLOYMENT_NAME}",
          "displayName": "Driver"
        }
      ]
    }
  ]
}

vLLM (fully private)

{
  "target_models": [
    {
      "providerName": "vllm",
      "providerDisplayName": "vLLM (Target)",
      "api_base": "http://{VLLM_HOST}:8000/v1/",
      "api_key": "VLLM_API_KEY",
      "models": [
        {
          "model_id": "sut_model",
          "name": "meta-llama/Llama-3.1-8B-Instruct",
          "displayName": "Llama 3.1 8B (Target)"
        }
      ]
    }
  ],
  "evaluation_models": [
    {
      "providerName": "vllm",
      "providerDisplayName": "vLLM (Evaluator)",
      "api_base": "http://{VLLM_EVAL_HOST}:8000/v1/",
      "api_key": "VLLM_API_KEY",
      "models": [
        {
          "model_id": "text_eval_model",
          "name": "meta-llama/Llama-3.3-70B-Instruct",
          "displayName": "Llama 3.3 70B (Evaluator)"
        }
      ]
    }
  ],
  "driver_models": [
    {
      "providerName": "vllm",
      "providerDisplayName": "vLLM (Driver)",
      "api_base": "http://{VLLM_HOST}:8000/v1/",
      "api_key": "VLLM_API_KEY",
      "models": [
        {
          "model_id": "default_driver_model",
          "name": "meta-llama/Llama-3.3-70B-Instruct",
          "displayName": "Llama 3.3 70B (Driver)"
        }
      ]
    }
  ]
}

Setting up vLLM

Prerequisites

• GPU-backed machine
• Network connectivity from the Okareo services container to vLLM host:port
• Hugging Face account with access approval for gated models

Install and serve

1. Create a Python environment:

python3 -m venv vllm-env
source vllm-env/bin/activate
pip install --upgrade pip
pip install vllm huggingface-hub

2. Authenticate to Hugging Face (if required for gated models):

huggingface-cli login

3. Start serving a model:

# Single GPU
vllm serve RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8400 \
  --gpu-memory-utilization 0.92

# Multiple GPUs
vllm serve RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 65000 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 2

4. Validate:

curl -sS "http://{VLLM_HOST}:8000/v1/models" | head

GPU sizing reference

Model flavor	GPU class	max-model-len
70B FP8 (quantized)	1x A100 80 GB	~8k
70B FP8 + tensor-parallel 2	2x A100 80 GB	~65k
70B full precision	4x A100 80 GB	~65k

tip

--max-model-len and concurrency drive GPU memory usage. Start conservative.

vLLM configuration details

• api_base -- must be reachable from the services container and must end with /v1/. Avoid localhost unless vLLM runs in the same container.
• api_key -- if vLLM has no auth, set VLLM_API_KEY=sk-none. If auth is enforced, set the real key.
• name -- must match the served model identifier your endpoint expects.
• model_id -- for the evaluator use text_eval_model, for the driver use default_driver_model.

Environment variable reference

Variable	Description	Required	Default
EVAL_API_KEY	API key for evaluation model provider	Yes	--
EVAL_MODEL_NAME	Evaluator model name (e.g., azure/<deployment>)	Yes	--
EMBEDDING_MODEL_NAME	Embedding model name	Yes	--
VLLM_API_KEY	vLLM API key	No	sk-none
OPENAI_API_KEY	OpenAI API key (if using OpenAI)	If applicable	--

---

Next step: Operations and Maintenance

Related: Deploy with Docker | Deploy on Kubernetes