Agents

The retrieval bench

Check whether an agent would reach an imported API operation for something a caller might say, and understand what the bench deliberately does not do.

The bench answers one question: would this agent reach this API operation if a caller said that? It runs the same search a live turn runs and shows you what came back and how strongly. It is a retrieval test, not an execution test, and the difference decides how you read the result.

What you need

  • An agent, opened on its Tools tab. The bench is scoped to one agent, so the workspace Tool Library page does not offer it.
  • An imported OpenAPI tool whose status is Ready, selected on that agent. Nothing else is searched. A database connection has its own bench, described in Review and approve database questions.

Steps

Step 1: Open the tool from the agent

On the agent's Tools tab, open the tool. Its detail view has an Overview tab, an Endpoints tab and, once the tool is Ready, a Tests tab.

Step 2: Open the Tests tab

The Tests tab holds the bench, scoped to this tool. Once a run has been recorded, a line above the box shows "✓ Matched" or "× No match" with the query you ran.

Step 3: Type what a caller might say

The box takes plain speech, not an endpoint name. The placeholder is "e.g. when is the next flight?" and the hint above it explains the point: "Type what a caller might say. This runs the same semantic search the agent runs each time, so you can detect an incorrect match before launch."

Four suggestion chips sit under the box: "when is the next flight", "cancel my booking", "what is my baggage allowance" and "change my seat". Pressing one fills the box and runs immediately.

Step 4: Run it

Press Run beside the box, or press Enter in the box. The button reads Testing… while the search is in flight.

Get it right

The bench compares your sentence with the operation's description, so type the sentence a caller would actually say. Typing the operation's own name proves nothing: it matches itself.

Recommended

my parcel hasn't turned up and I ordered it last Tuesday

Less effective

getOrderShipmentStatus

What happens next

EchoNova takes the tools selected on this agent, keeps the imported OpenAPI ones that are Ready, narrows that to the tool you opened, and returns the operations that best match your text. What comes back is a ranked list, not a call.

Under Results each row shows a signal-bar glyph, the HTTP method, the operation's path, the tool it belongs to, and a score as a percentage. The score is a similarity rounded to a whole number and clamped between 0 and 100. It measures the distance between your wording and the operation's description. It is not a probability, and it is not comparable across different queries.

Nothing above the threshold is a result in its own right, and the bench says so: "There is no match above the specified minimum threshold. The agent will ask a clarifying question instead of calling an endpoint."

What the bench does not do

  • It does not call your API. No request reaches the operation's URL, and your server logs will show nothing.
  • It sends no arguments. You give it an utterance, not a request body, and no parameters are filled.
  • It does not use the agent's language model. Matching compares text, and the model you chose for the agent is not involved.
  • It does not create a conversation. No session is opened and no turn is recorded, so nothing appears under the agent's conversations.
  • It does not consume credits. Credits are charged for turns in a live conversation, and a bench run is not a turn.

So a green result means the agent would reach the operation, not that the operation works. Auth, base URL, request shape and your endpoint's own behaviour are all untested here. Verify those separately, then talk to the agent in the playground to see the two halves work together.

The same limit applies in reverse to the tool kinds that are never searched. A webhook, integration or system tool is in front of the agent on every turn, so there is no retrieval decision to test and the bench does not score it.

What the run records

A run's outcome is kept per tool: whether it matched, the best score, the exact query and the time. Only the tools that run searched are updated, so a scoped run leaves the other tools' results alone.

You see it in two places afterwards.

Where What it shows
The Last tested column of the agent's tools table How long ago it ran, or "Never"; hover shows the query and the top score as a percentage
The Tests tab of the tool "✓ Matched" or "× No match", with the query

The result survives a reload, and it belongs to the agent that ran it. What is searched depends on what that agent has selected, so the same tool can surface for one agent and not another.

Limits

Limit Value Applies to
Query length 500 characters Every run
Tools searched Imported OpenAPI tools with status Ready, selected on the agent Every run
Scope in the portal One tool, the one you opened The Tests tab
Hits returned Up to 3, each scoring 0.5 or higher (a live turn keeps up to 5) Every run
Run timeout 30000 ms Every run
Results kept One per tool, replaced by the next run that searches it The agent
Permission Workspace membership, nothing further Running the bench

The number of hits and the score they must reach are not adjustable from the portal today.

Troubleshooting

"Could not run the bench."

The run failed rather than returning nothing. Retry it once. If it keeps failing, wait a moment and try again: a tool that has only just finished importing may not be searchable yet.

The result is empty for every query, including the presets

Nothing was searched. Check two things in order: the tool's status is Ready rather than Importing or Failed, and the tool is selected on this agent rather than only present in the workspace library. A run with nothing to search returns an empty result rather than an error, which reads exactly like a bad match.

There is no Tests tab

The bench is offered for imported OpenAPI tools that are Ready, opened from an agent. A webhook, integration or system tool has no retrieval step to test. A database connection has its own bench instead, and only once its questions have been drafted.

The bench matches but the agent still does not call the operation

Retrieval is one of two gates. The operation surfaced, so the wording is fine; what is left is the agent's decision and the operation's own state. Check that the operation is enabled, and that its description says when it should be used.

The score looks low but the agent behaves correctly

Treat the percentage as a ranking aid rather than a pass mark. What matters is whether the operation you expected is in the list and above the ones you did not expect.

FAQ

Does a bench run cost anything?

No. It records no turn and consumes no credits.

Does the bench hit my endpoint?

No. It searches the descriptions of your operations, not the operations themselves.

Why is my webhook tool not in the results?

It is never searched. Webhook, integration and system tools are in front of the agent on every turn, so they are not ranked and carry no score.

Can I test how the agent chooses between two tools?

Not from the Tests tab, which always scopes to one tool. Talk to the agent in the playground instead, where every tool it has competes as it would in a real conversation.

Does the recorded result change when I edit the tool?

No. It holds the outcome of the last run and its query. Run it again after an edit to refresh it.

Do bench results follow the tool to another agent?

No. The result belongs to the agent that ran it, because what is searched depends on that agent's selected tools.