You are tuning an AI agent within the GoodData AI Hub. You add a doc that defines your metrics, add a reminiscence instruction about the best way to group income, open the preview and ask a query. The reply appears to be like proper.

Did it learn the doc? Did the instruction apply? Why did it decide that talent and never the opposite one?

You don’t have any technique to inform. The reply is the one factor on display.

We constructed a panel for precisely this. It’s presently known as Behind this reply, it sits underneath each response within the Agent Builder preview, and it lists the steps the agent ran to supply that response: which expertise it picked, which recollections it utilized, what it looked for and what it discovered, which metric it queried, how lengthy every step took and what number of tokens it used. A peek underneath the hood, with out drowning you in particulars.

What Is Behind This Reply?

Throughout consumer testing of the Agent Builder, individuals did the wise factor. They configured an agent, opened the preview and requested it inquiries to verify their work. Then they requested us the identical three questions each time: Did it learn my Data recordsdata? Did my Reminiscence instruction kick in? Why this talent?

We couldn’t reply any of them from the product, and neither may they. A well-grounded reply and a believable guess regarded equivalent. That could be a dangerous place to be when you find yourself about handy the agent to your clients.

“Behind this reply” is the repair. It’s a small panel, however the mechanism underneath it’s what the remainder of this put up is about.

Each Reply Is a Sequence of Steps

When the agent solutions, it does a number of issues so as. It decides which expertise apply. It hundreds recollections. It searches your data base and the semantic catalogue. It normally runs a metric question, and at last it writes the reply. Every of those has an enter, an output, a period and a token price.

So the runtime now emits one occasion per step: what ran, what it used, whether or not it labored and what it price. The step classes observe the agent’s inside workflow, applyMemory, skillRouting, knowledgeSearch, catalogSearch, metricQuery, composeAnswer, so you possibly can filter on them.

The panel is a renderer over these occasions. The occasions are additionally saved, and that’s the half that issues long run. Something we construct later about conversations, or about brokers throughout a complete group, reads the identical data.

What the Panel Reveals

Underneath every reply there’s one small line, for instance 21.2s / 97K tokens. Click on it and the panel opens with one row per step. Every row has a one-line abstract and a element view.

Apply Reminiscence

Which recollections have been set to Use at all times, which Use when related recollections matched this query and the way nicely, and what number of have been utilized in whole. In case your instruction didn’t kick in, that is the place you see it.

Talent Routing

The abilities the agent had obtainable and those it activated for this reply. Routing can occur greater than as soon as per response. A talent that was obtainable however by no means activated is the most typical tuning discovery.

Data Search

The question the agent really despatched, which is its personal rephrasing of your query, the paperwork that got here again above the similarity threshold with their scores, and the perfect match. This row solutions “did it learn my doc?” immediately.

Catalogue Search

The identical for the semantic layer: question phrases, which object varieties it searched, what it discovered and what it used.

Metric Question

Metrics, grouping, the visualization it produced, and the dimensions of the consequence.

Compose Reply

The output sort, the mannequin, and what number of follow-up options it generated.

The footer reveals the hint ID, so when somebody asks in Slack why a solution regarded odd, you possibly can ask GoodData as a substitute of guessing.

The panel doesn’t replay the mannequin’s uncooked reasoning. The chat has a separate collapsible “Reasoning” aspect for that, and the panel sticks to information concerning the run.

Placing It Collectively

Here’s what occurred once we examined it on a recent demo agent. We uploaded a doc that defines web income, wrote “floor definitions within the group data base” into the agent’s character, and requested: How will we outline web income, and what was it by area?

The reply got here again with an accurate definition and a bar chart. Good.

The panel confirmed 5 steps: reminiscence, routing, catalogue search, metric question, compose. No data search. The agent had taken the definition from the metric’s description within the semantic layer and by no means opened the doc. The character instruction was not sturdy sufficient to alter that.

With out the panel, that is invisible. The reply is appropriate, the demo goes nicely, and you discover out weeks later when a buyer asks one thing solely the doc covers. With the panel, it’s one lacking row, and also you repair the configuration earlier than anybody else sees it.

For Builders: The Steps API

The panel is one client of the occasions. Should you would quite have them as knowledge, they’re one GET away:

GET /api/v1/ai/workspaces/{workspaceId}/chat/conversations/{conversationId}/steps

{
  "steps": [
    {
      "stepId": "…", "responseId": "…", "stepIndex": 0,
      "durationMs": 5754,
      "tokens": { "input": …, "output": …, "total": … },
      "createdAt": "2026-08-04T…Z"
    }
  ]
}

A step carries solely what belongs to the entire mannequin name: order, period, spend. What every step did is on the dialog objects from the …/objects endpoint. Every merchandise has a stepId, and the place a instrument ran, a typed element with the question and the outcomes. Be a part of the 2 and you’ve got the identical image the panel attracts, prepared in your personal dashboards or your individual observability stack.

What We Bought Fallacious the First Time

The primary minimize saved every step as a dialog merchandise, in the identical desk and stream as consumer messages and assistant replies. It was low cost to construct. The objects pipeline already dealt with persistence, streaming and reload, so the steps obtained all of that totally free.

It had two prices.

Dialog objects are what will get replayed to the mannequin as historical past on the subsequent flip. So each place that constructed the mannequin’s context needed to filter the steps again out, and one supplier adapter that didn’t know the content material sort rendered it as an empty block and obtained its request rejected. We have been sustaining code to cover rows from the desk we had simply written them to.

The second price was within the element. One step can include a number of instrument calls, for example two catalogue searches. Their particulars have been merged into one abstract on the step, so two searches that discovered completely different objects with completely different scores become one blurry entry. We may not inform which question had discovered which object.

So the step turned its personal entity, with its personal desk, its personal stream occasion and the /steps endpoint above. It carries solely what belongs to at least one mannequin name. The small print of every motion dwell on the merchandise that produced them, and every merchandise factors to the step it ran in. Two searches in a single step at the moment are two data with two scores.

The rule we took from this: if it’s important to filter one thing out of the gathering you saved it in, it doesn’t belong in that assortment.

How We Suppose About It

The identical occasions reply completely different questions at completely different scopes. For one response: did it use the fitting issues? For one dialog: did the consumer need to rephrase, did the session finish proper after a failure? For one agent in a single workspace: how typically does it hit data gaps, what goes unanswered? For one agent throughout workspaces: does it behave the identical all over the place, or is a reminiscence instruction lacking in one in all them? We constructed the single-response view first as a result of the others learn the identical knowledge.

The agent’s personal report isn’t sufficient. The agent is aware of issues no one else can see, equivalent to “I searched the data base and located nothing”, and we report these. However an agent grading its personal failures under-reports. So we additionally need detection from the skin: guidelines over the hint and the dialog that spot a consumer re-asking the identical query or correcting the reply. Not each discovering is an issue, both. A workspace the place customers hold thanking the agent tells you which of them configuration to repeat.

This isn’t the utilization dashboard. GoodData already reveals AI utilization as counts, suggestions charges and adoption per workspace, constructed from metadata with none AI. The hint works on dialog content material and step occasions. The dashboard tells you the adverse suggestions fee went up. The hint tells you which of them questions prompted it and why. It additionally carries extra delicate knowledge than combination counts, so entry to it wants stricter guidelines.

The hint by no means goes again into the mannequin. No matter we report a couple of run should not change the run. That constraint is what uncovered the primary knowledge mannequin.

Past the Panel: What Comes Subsequent

The panel is step one. The saved occasions are there for the subsequent ones, and we have now prototyped sufficient to know roughly what they appear to be.

Tagging. Tag every dialog because it occurs: what subject the reply was about, and whether or not the consumer needed to ask once more or appropriate the agent. Within the prototype the agent emits these tags in the identical last name that ends the flip, so there isn’t any additional mannequin round-trip. Free-text matters drift (“income forecast” and “income projection” find yourself as two buckets), so the plan is to retailer an embedding and cluster at question time. The evaluation layer is then SQL plus vector search over tags, and the mannequin price is paid as soon as per dialog.

Ask the info. We tried an admin-facing talent. You ask in chat “what matters got here up most this week” or “the place do customers hit useless ends”, and it solutions from a set set of organization-scoped queries quite than free SQL. That was a fast technique to discover out which questions admins really care about. The recurring ones can develop into experiences.

From discovering to repair. Counsel a reminiscence instruction for a lacking definition, a data doc for a lacking subject, or, hardly ever, a change to the semantic layer. In the present day the agent can solely learn these, so the options are advisory. Data is the plain first one to make one-click.

Belief in an AI agent comes down to at least one factor

Can the individuals chargeable for it see what it did and repair it when it goes fallacious? Behind this reply offers agent authors that view for a single response, and the occasions underneath it give us the bottom for a similar view throughout conversations, workspaces and brokers. The open questions there are about privateness, not plumbing: what dialog content material an admin may even see, and for a way lengthy we hold it.

Subsequent time a solution within the preview surprises you, open the panel. Examine which talent ran, which paperwork it discovered and used, and what it price. Then repair the configuration as a substitute of guessing.

Behind this reply is rolling out within the Agent Builder preview now.

Need to see what GoodData can do for you? Request a demo