Agentforce Session Traces

Salesforce Agentforce keeps its own record of every agent session in Data 360: the topic it routed each turn to, the actions it ran, the instructions its model saw, and, for voice, what it heard the caller say. This record is Salesforce's session tracing data (the Session Tracing Data Model, or STDM).

The Agentforce integration brings that record into Okareo. Once it is connected:

• Simulations get Salesforce's view. Each conversation your simulation has with an Agentforce agent is matched with its session trace, so checks can score what the agent actually did, not only what it said.
• Production sessions are monitored. Sessions that didn't come from a simulation arrive as monitoring traces in the same project.

Unlike the voice monitoring providers, Agentforce needs no webhook and nothing is pasted into Salesforce. Okareo signs in to your org with an External Client App and checks it for new sessions on a schedule you set.

Setup has three parts:

1. In Salesforce: turn on session tracing and choose the External Client App Okareo signs in with.
2. In Okareo: add an Agentforce integration to your project and test the connection.
3. Run simulations in that project. Nothing is configured on your Targets.

Set up Salesforce

A Salesforce admin does these steps once per org.

Turn on session tracing

Agentforce records sessions in Data 360 only while session tracing is on.

1. In Setup, open Einstein Audit, Analytics, and Monitoring Setup.
2. Turn on Agentforce Session Tracing.

Salesforce takes a few minutes to set up session tracing in Data 360 after you turn it on. Session tracing records only sessions that start after it is on.

Set up an External Client App

Okareo signs in with the OAuth client credentials flow, so the app acts as one run-as user and nobody signs in interactively. Agents and their session traces belong to the org, not to an app: any app whose run-as user can read them works, and it doesn't change how your Targets or other apps reach your agents.

Use the app your Targets already use. If your Targets call the Agentforce Agent API, you already have an External Client App with the client credentials flow. Add the cdp_query_api scope to it (and api, if it isn't there), and make sure its run-as user can read session tracing data in Data 360. Then use its consumer key and secret.

Or create a dedicated app. A separate app lets you give Okareo a run-as user that only reads, and lets you rotate its secret without touching your Targets. It's also the fix when Salesforce refuses the shared app's access tokens (see Troubleshooting).

1. In Setup, open External Client App Manager and click New External Client App.
2. Name the app (for example, Okareo Session Traces) and enter a contact email.
3. Under the OAuth settings, select Enable OAuth. If Salesforce asks for a callback URL, enter any HTTPS URL; the client credentials flow doesn't use it.
4. Add these OAuth scopes:
   • Manage user data via APIs (api)
   • Perform SQL queries on Data Cloud data (cdp_query_api)
5. Select Enable Client Credentials Flow.
6. Under Security, leave Issue JSON Web Token (JWT)-based access tokens for named users unchecked, and save the app.
7. On the app's Policies tab, edit the OAuth Policies: enable the client credentials flow and set Run As (Username) to a user who can read Agentforce session tracing data in Data 360. The run-as user of your Agent API app works, since it already sees your agents.
8. Click Consumer Key and Secret and copy both. Salesforce first emails a verification code to the address on your Salesforce user. The consumer key is the client ID and the consumer secret is the client secret.

Before you leave Salesforce, note your My Domain URL (Setup → My Domain). It looks like https://acme.my.salesforce.com.

Find your Agent IDs (if needed)

By default the integration collects sessions from every agent in the org. To limit it to certain agents, you choose them from the org's agents after testing the connection, so you usually don't need their IDs. If Okareo can't list the agents, the Agents field asks for IDs instead. To find an agent's ID:

1. In Setup, enter Agent in the Quick Find box and select Agentforce Agents.
2. Click the agent's name.
3. Copy the ID at the end of the page's URL. It starts with 0Xx and is 18 characters long.

This is the same agent ID the Agent API uses. If your agent was built in the new Agentforce Builder and its URL doesn't show the ID, see Salesforce's Get the Agent ID.

Add the Agentforce integration

An Agentforce integration belongs to one Okareo project, and every trace it collects lands in that project. Create it in the project where you run your Agentforce simulations.

1. In Okareo, open User and Org Settings → Integrations → Voice Monitoring and click New Integration.

2. Choose Agentforce and fill in:

   • My Domain: your org's My Domain URL, for example https://acme.my.salesforce.com.
   • Client ID and Client secret: the consumer key and secret from your External Client App.

3. Click Test connection and fix anything that fails (see Troubleshooting). Create integration becomes available once the test passes. If you then change the My Domain or the credentials, test again.

4. Optionally, set:

   • Agents: search the org's agents by name or ID and choose up to 20. Each shows how many sessions it had in the last 30 days and when the latest started, and the most recently active are listed first. Leave it empty to collect sessions from every agent in the org. If the agents can't be listed, enter Agent IDs instead.

     An agent is listed once a version of it has been committed in Agentforce Builder; a new agent that is still a draft doesn't appear yet. After you commit one, test the connection again to refresh the list.

   • Polling: when Okareo looks for sessions. Leave these blank to use the defaults:

     Field	Default	What it means
     Wait after a session ends	60 s	How long Okareo waits after a session ends before collecting it, so Salesforce has time to record all of it.
     Check every	60 s	How often Okareo checks your org for sessions. The minimum is 15 seconds.
     Give up after	1800 s (30 minutes)	How long after a session starts Okareo keeps waiting for it. A session that is still incomplete then is collected as recorded so far.

5. Click Create integration. Nothing is saved until you do.

Okareo stores the client ID and secret encrypted. They are write-only: Okareo never shows them again, and you change them by rotating the secret.

A project can have one active Agentforce integration per Salesforce org. To collect different agents in different projects, create an integration in each project and set its Agents.

Test the connection

Test connection works before you create the integration, while you edit its settings, and on a saved one. It shows a pass or fail with a hint for each check:

Check	Passes when
Settings	The My Domain is a valid URL, and the client ID and secret are entered.
Sign-in	Salesforce accepts the client ID and secret at your My Domain.
Session traces	Okareo can read Agentforce session tracing data in Data 360.
Agents	Every chosen agent is still in this org. It lists each agent's name. Shown when Agents is set, on a saved integration or in Edit settings.

Last validated is set when you create the integration and updates each time a test passes. Last used updates when Okareo collects a session.

Manage the integration

• Edit settings: change the My Domain, the agents, or the polling settings. Save settings is available once a connection test passes, in the form or on the integration's card just before. To change a chosen agent, click it and pick another; to remove it, click its ×. Changes apply from the next check.
• Rotate secret: replace the stored client ID and secret, for example after you regenerate the consumer secret in Salesforce.
• Deactivate and Enable: pause and resume collection without deleting the integration.
• Delete: remove the integration. Traces it already collected stay in the project.

After you rotate the secret, run Test connection again.

How collection works

Every Check every seconds, Okareo looks in your org for sessions that started within the last Give up after window, from the agents you selected.

• A session is collected once it has ended and Salesforce has recorded it, usually a few minutes after it ends.
• Each session is collected once. It doesn't come back on later checks.
• There is no backfill. Sessions that started before you created the integration aren't collected. Neither are sessions that started while it was deactivated, if they're older than Give up after when you enable it again.

Salesforce usually makes a session available within a few minutes, so expect each simulated conversation's trace to appear about 5 to 10 minutes after the conversation starts.

What you'll see

Each collected session appears in the project with source Agentforce, as a trace with a session → turn → step structure. Three kinds of datapoints come from it, so you can score each part of the agent's behavior on its own:

Datapoint	What it holds
Turn	One for each turn where the caller or the agent spoke: the conversation so far and the agent's reply. The agent's opening greeting is one too, with no conversation before it. A turn where the agent answered without words, such as a handoff to a human, has an empty reply. The session's last turn also carries the session summary: the topics, the actions the agent ran, how the session ended, and whether it was a voice or text session.
Routing decision	Which topic Agentforce chose for the caller's message, and the topics it chose from. The topics are its tools and the chosen one is its tool call, so tool-call checks can judge the choice.
Model call	One for each call a topic's agent made to its model: the instructions and messages it saw, the actions it could use, the actions it called, and their results. Salesforce's own groundedness validation of the agent's replies appears as model calls too; gen_ai.sf.prompt_template names the prompt template a call used.

For voice, caller messages are Agentforce's own speech-to-text, so you can compare what the agent heard with what the simulated caller said.

Linking to simulations

A session that a simulation started is linked to that simulation and its scenario row:

• Text: a Custom Endpoint Target that calls the Agentforce Agent API.
• Voice: a WebRTC voice Target that starts an Agentforce session and joins its call.

The simulation must run in the integration's project. A session that no simulation started, such as production traffic, is collected as a monitoring trace.

Score Agentforce traces with checks

Use monitors to run checks on Agentforce traces as they arrive. Filter each monitor to Source equals Agentforce, then narrow it to the datapoints the check is about. A metadata filter is a User Metadata filter with the key below:

To check	Filter	For example
The whole conversation (topics, escalation)	gen_ai.sf.final_turn equals true	Your own code checks on topic and escalation
What the agent heard, in voice sessions	gen_ai.sf.final_turn equals true and gen_ai.sf.modality equals voice	Your own code check comparing the transcript with what the simulated caller said
Each turn	gen_ai.sf.datapoint equals turn (add gen_ai.sf.greeting not equal true to skip the opening greeting)	Tone, helpfulness, policy checks
Tool choice	Tool Calls is set	function_call_validator, on the routing decisions and the model calls that called an action
Routing	gen_ai.sf.datapoint equals router	Whether the expected topic was chosen
Model calls	gen_ai.sf.datapoint equals llm_call	Checks on what a topic's agent saw and how it answered

Filter tool-call checks on Tool Calls rather than on model calls: a model call that answered without calling an action gives a tool-call check nothing to judge, so it passes without telling you anything.

In a code check, the same facts are in the metadata dict:

Key	Meaning
user_gen_ai.sf.topics	On the last turn: the topics the conversation was routed to, in order (a JSON list stored as text).
user_gen_ai.sf.actions	On the last turn: the actions and tools the agent ran, in order, such as escalate_to_human (a JSON list stored as text).
user_gen_ai.sf.end_type	On the last turn: how the session ended, such as CLOSED_TRANSFERRED.
user_gen_ai.sf.modality	On the last turn: voice for a voice session, text otherwise.
user_gen_ai.sf.topic	On each turn: the topic the turn was routed to.
user_gen_ai.sf.selected_target	On a routing decision: the topic chosen.

Whether a session escalated is your decision, not Okareo's. A common rule is that the agent ran a handoff action such as escalate_to_human, or the session ended CLOSED_TRANSFERRED. Routing a turn to an escalation topic isn't enough on its own, since the caller may decline the handoff.

The check below compares the conversation's topics and escalation with the expected values in each scenario row's result, for example {"topic": "Order_Management", "escalated": false}. Run it on the last turn.

agentforce_routing_check.py

import json

from okareo.checks import CodeBasedCheck, CheckResponse

HANDOFF_ACTIONS = {"escalate_to_human"}  # add your org's own transfer actions


def decode(value):
    # Lists arrive as JSON text.
    while isinstance(value, str):
        try:
            value = json.loads(value)
        except ValueError:
            break
    return value


class Check(CodeBasedCheck):
    @staticmethod
    def evaluate(scenario_result: str, metadata: dict) -> CheckResponse:
        expected = decode(scenario_result)
        if not isinstance(expected, dict):
            return CheckResponse(score=True, explanation="No expected topic or escalation in the scenario row.")
        metadata = metadata or {}
        topics = decode(metadata.get("user_gen_ai.sf.topics")) or []
        actions = decode(metadata.get("user_gen_ai.sf.actions")) or []
        escalated = bool(HANDOFF_ACTIONS & set(actions)) or metadata.get("user_gen_ai.sf.end_type") == "CLOSED_TRANSFERRED"

        problems = []
        if expected.get("topic") and expected["topic"] not in topics:
            problems.append(f"expected topic {expected['topic']}, got {topics or 'none'}")
        if "escalated" in expected:
            want = str(expected["escalated"]).lower() == "true"
            if want != escalated:
                problems.append(f"expected escalated={want}, got {escalated}")

        return CheckResponse(
            score=not problems,
            explanation="; ".join(problems) or f"Topics {topics}, escalated={escalated}.",
        )

Upload it with okareo.create_or_update_check(...) as described in Custom Checks. Check results on traces linked to a simulation roll up on the simulation run alongside its transcript checks.

Limits

• Linking: text (Agent API) and WebRTC voice sessions link to simulations. Sessions over Agentforce's phone (PSTN) or SIP channels are collected as monitoring traces but aren't linked to a simulation.
• No audio: traces contain text only (transcripts and agent steps). Call audio is not collected from Salesforce.
• Tracing start: session tracing records only sessions that start after it is on, and Okareo collects only sessions that start after the integration exists.
• One project per integration: traces land in the integration's project, and only simulations in that project are linked.

Troubleshooting

Test connection fails

Failed check	What to do
Sign-in: check the client ID and client secret	Copy the consumer key and secret again. If you regenerated the secret, rotate it in Okareo.
Sign-in: enable the client credentials flow	On the app's policies, enable the client credentials flow and set its run-as user.
Sign-in: could not reach, or no token endpoint at this My Domain	Check the My Domain spelling. Use your My Domain (Setup → My Domain), not a Lightning or site URL. If the My Domain is right, test again: Salesforce occasionally gives this answer for a moment.
Session traces	Turn on Agentforce session tracing, add the cdp_query_api scope to the app, and make sure the run-as user can read session tracing data in Data 360.
Session traces or Agents: INVALID_JWT_FORMAT	Salesforce refused the app's JWT-based access token. Use a dedicated app with Issue JSON Web Token (JWT)-based access tokens for named users unchecked, then rotate the secret in Okareo with its consumer key and secret.
Agents: not found in this org	In Edit settings, test the connection, then edit or remove the agents marked as not found.
Agents: could not be looked up	Make sure the app has the api scope and its run-as user can see your Agentforce agents. Until then, the Agents field takes typed Agent IDs.

No Agentforce trace appears

Work through these in order:

1. Wait a few minutes. Salesforce usually makes a session available within a few minutes of it ending, and Okareo collects it on its next check.
2. Check the integration. It is active, Test connection passes, and it is in the project where the simulation ran.
3. Check the agents. If Agents is set, the session's agent is one of them.
4. Check session tracing. It is on, and it was on before the session started.
5. Check the timing. The session started after the integration was created. If your org is slow to record sessions, raise Give up after.

Last used on the integration updates each time Okareo collects a session. If it never changes, no session has been collected yet.

Checks run on the wrong datapoints

Each session produces several datapoints. Narrow each monitor with the filters in Score Agentforce traces with checks, so conversation-level checks run once per session and tool-call checks run only where there are tool calls.