Agents are becoming the new developer platform, using semantic search with data from tools like Git, Slack, and Jira for context. In their presentation AI Meets Internal Developer Platform at KubeCon & CloudNativeCon Europe, Whitney Lee and Viktor Farcic spoke about setting guardrails to block or allow things, and using logs, metrics, and traces to understand agent behavior.
Agents are the next revolution of platforms. You want to give your developers tooling to do stuff; before it was Backstage, now it’s agents, Farcic argued.
Agents take input from a developer, combine it with the system context, and send it to a model. The model can respond and provide an answer, or execute a tool based on descriptions of what it can do, Farcic explained:
Semantic search enables an agent to find the information it needs at a given moment, Farcic said:
The sources of information can be anything: Git repositories with the actual manifests and code, pull request discussions, Slack threads where somebody explained why we don’t use that database, Jira tickets, wiki pages, Zoom transcripts of design meetings. That’s where "how we like to do things" lives.
Companies are building their own agents to make things secure. Farcic mentioned that we want to have some kind of guardrails that will block or allow certain things, things that a person can execute automatically, or that require specific approval.
With an LLM, the input can be anything that a user provides to an agent, and the output is whatever the LLM thinks it should do, Farcic said. We have no idea what the input or output is, he added; we cannot limit what will happen.
Agent traces give visibility into how developers are interacting with a platform, Lee said. They show which models an agent uses, which tools it chooses, what those calls cost in tokens, and the path it takes to accomplish a task.
Farcic suggested capturing what is happening on the agent level, using logs, metrics, and traces:
Traces allow you to see the request going from the user to the agent to the tools. They are probably the only way to understand what is happening after it happened.
OpenTelemetry has semantic conventions for GenAI; spans for model calls with the model name, token counts, tool invocations, and their arguments, Farcic explained:
With OpenTelemetry you can send the data anywhere: Jaeger or Grafana Tempo for traces, Prometheus for metrics, Loki for logs. The commercial platforms — Datadog, Honeycomb, Dynatrace, Elastic — all ingest OpenTelemetry as well.
Real traces are your test data. The questions your developers actually ask, and the paths the agent actually took, make a far better evaluation set than anything you’ll invent in a meeting, Farcic concluded.
InfoQ interviewed Whitney Lee and Viktor Farcic after their talk.
InfoQ: How can we get good results with semantic search?
Viktor Farcic: The quality of semantic search is not defined by the embedding model or the vector database. Those are commodities. It’s defined by what you put in and how you slice it.
The first mistake is assuming documentation is the source. It’s *one* source, and usually the worst one, because it’s outdated.
Second, chunking. If you embed a forty-page document as a single vector, you get one blob that is vaguely about everything and precisely about nothing. Split it into pieces that stand on their own — a section, a function, a resource definition — and keep the metadata: which repo, which team, and when.
Third, freshness. An index is not a migration you run once. If the information is six months old, the agent will confidently give you six-month-old answers, and that is worse than no answer, because you trust it. Ingestion has to be continuous, and deletions have to propagate.
Finally, know what should *not* be in the index. Semantic search is for knowledge. The current state of your cluster is not knowledge; it’s state, and it changes every second. Don’t embed it — give the agent a tool to fetch it live.
And measure it. Look at the traces and see which chunks actually came back for a given question. When the answer is wrong, the retrieval is usually the culprit, not the LLM.
Whitney Lee: Vector search helps retrieve information from different tools. As a user, you (or more likely your coding agent) don’t have to understand where the information is, and because the information is indexed by semantic similarity, you don’t have to guess the exact words used in order to find what you’re looking for.
InfoQ: How can we use traces with AI applications?
Lee: A single trace can tell you what an agent did, which helps with debugging. Looking at thousands of traces can tell you how developers are using your platform.
With agent traces at scale, agent observability becomes platform discovery. Your traces are telling the platform team what developers are trying to accomplish through the platform. That creates a feedback loop: developers interact with the agent, the traces show how the agent and the platform responded, and together they show the platform team where the developer experience can improve.
Farcic: With a normal application, when something breaks, you re-run it with the same input and watch it break again. With an agent, you can’t. Ask the same question twice, and you get two different execution paths. So the trace isn’t a debugging convenience; it’s the only evidence the thing ever happened the way it did.
I look at three things. What did the agent decide to do — which tools, in what order, with which arguments? A tool that never gets called, or always gets called wrongly, is a tool description problem, not a model problem. Second, where the time and the money went; one badly scoped semantic search can drag fifty thousand tokens into the context and you will only ever see that in a trace. Third, the audit trail. If the agent changed something in production, somebody will eventually ask who approved it, and "the AI did it" is not an acceptable answer.