Skip to main content

Getting Started with Real-Time Error Tracking

Building reliable LLM and agentic applications isn’t just about preventing crashes. It’s about catching the quiet failures—when your agent technically responds, but does the wrong thing. That could mean making up a refund policy, skipping required steps, or going off on an inappropriate tangent with overconfidence. These errors don’t show up in logs or crash reports. They just happen—and they affect real users.

Okareo helps you catch and fix those invisible failures. It acts as an always-on watchdog that monitors every interaction with your agents and monitors what they say, decide, and do. Whether your app is fielding support requests, guiding internal workflows, or responding to sales leads, Okareo helps you spot when things silently go off the rails.

In this guide, we’ll walk through how to connect your app to Okareo in minutes—so you can start detecting issues like hallucinated answers, broken flows, or safety violations in real time.

Why Teams Use Error Tracking?​

Real-Time Detection Prompt evals and offline tests miss the moment.

  • A customer-facing agent responded with “we’ll cover that cost,” even though the refund policy said otherwise. There was no alert—until it was flagged as a scope violation.

Replay + Root Cause See the full reasoning chain: tools, memory, prompts, and all decision steps.

  • A customer reported getting stuck during onboarding. Nothing showed up in logs. Replaying the agent revealed it skipped an identity verification step—because the memory state was overwritten mid-flow. Silent failure, invisible without a replay.

Failure Pattern Detection Surface patterns like off-topic responses, stuck conversations, or silent fallbacks.

  • A sales assistant started looping back to the same question whenever users asked about compliance.

Scope & Safety Monitoring Define what your agent is allowed to say and do. Get alerted the moment it deviates.

  • A support bot started suggesting legal advice after a retrieval miss.

How It Works​

To enable error tracking Okareo will inspect your LLM requests and responses and evaluate them for issues in real time. There are several options to collect your LLM requests and responses. You can use OpenTelemetry standard, cloud or self-hosted proxy, or send data points directly to Okareo via APIs. You control what data is shared and stored.

Okareo Proxy in Context

Issues vs. Errors: What’s the Difference?​

Throughout this guide, we will talk about "issues” and “errors” in the context of LLM/Agent behavior. Distinction between the two helps developers prioritize and understand problems:

  • Issues (Behavioral Issues): An Issue represents any notable unwanted behavior or outcome from the AI, even if the system didn’t technically fail. These are the subtle logic or content problems that traditional monitoring would miss. For example, if your customer support bot suddenly starts discussing financial advice when it’s not supposed to, that’s an issue. The app didn’t crash and the model gave a response – but it’s a behavior you consider incorrect or off-policy. Okareo would detect this condition and provide an explanation along with the dialog history showing where the finance topic came up. You can quickly grasp why it was flagged and decide on a fix (e.g., adjust the prompt or add a content filter for finance terms).

Issue Example

  • Errors (System/Configuration Errors): An Error usually refers to a more concrete failure in the model interaction – something that is undeniably wrong in execution. These often correspond to exceptions, misconfigurations, or invalid model responses. For example, using a wrong parameter or model setting or hitting a rate limit can be considered errors. Imagine you set the model’s temperature to an out-of-bounds value, or sent context that is too large for a given model – those are errors.

Error Example

In practice, the Okareo dashboard and notifications may label certain model responses as issues and others as errors. Issues are derived from checks failing, whereas errors might come from the model or system events (like exceptions or explicit refusals). Both are important: issues tell you where your model's behavior could be improved or corrected, and errors tell you where your request completely failed. Okareo’s real-time tracking covers both sides – catching silent logic issues and explicit errors – giving you a full picture of your system’s reliability.