Agent observability is the continuous, tamper-evident record of what an agent actually did, in production, at the level of every prompt, every tool call, every decision, and every hop to another agent or MCP server. It is the second pillar of Agent DOS, and it is the one that determines whether an incident is recoverable or simply over.
The reason observability is its own pillar is that the standard logging pattern from human-driven applications does not fit agents. A web request lives for milliseconds, has one identity, hits a small number of services, and produces a clean line in the access log. An agent run lives for minutes, holds multiple identities along its delegation chain, fans out across MCP servers and other agents, and produces a sequence of decisions that the application server has no opinion about. None of the existing log shapes capture what a security investigator needs.
See how Trust3 AI implements agent observability →
What standard logging misses for agents
Three properties of agent runs break the existing model.
Decisions happen inside the agent. When an agent picks tool A over tool B because of something in a retrieved document, the reasoning is in the agent's prompt and response history, not in any application log. Without capturing those, an investigator can see the outcome of a decision but not the cause.
Identity moves. A user asks a copilot for a report. The copilot calls a sub-agent. The sub-agent calls an MCP server, which calls a database. Without identity propagation, the database log says the service account did it. The investigator has no way to tie the query back to the originating user or even to the originating prompt.
State is ephemeral. When the agent process ends, the logs inside it usually do too. By the time someone is asking what happened, the only persistent record is the side effects: the row that was written, the email that was sent, the file that was uploaded. The chain of reasoning that led there is gone.
Observability for agents has to solve all three.
What needs to be captured
A complete agent telemetry record includes:
- Every prompt the agent received. From the user, from another agent, from a retrieval pipeline, from a tool response that fed back into reasoning.
- Every tool call the agent made. Including the arguments it constructed, the MCP server or API it called, the response it received, and how that response influenced the next step.
- Every intermediate decision. When the agent considered multiple tools, files, or actions, the choice it made and the inputs that drove the choice.
- Every A2A hop. Which agent called which, with what identity, carrying what purpose claim, returning what payload.
- Every data access event. Which dataset, which rows or columns, under which user's identity, with which policy decision.
- System metadata. Timestamps, the agent version, the model version, the tool catalog at the time of the call. Reproducing an incident requires knowing the state of the world when it happened.
The record has to be append-only and tamper-evident. If an agent compromise can rewrite its own audit trail, the audit trail is not evidence.
Identity propagation is the part most teams get wrong
The most common observability failure is not missing data. It is correct data attributed to the wrong identity.
When an agent calls another agent or an MCP server using a service account, downstream systems log that service account. From the database's perspective, the service account ran the query. From the security team's perspective, every action looks like it was performed by a small number of robot identities, none of which is useful in an investigation. Least-privilege becomes impossible because the principal asking for access is not the principal that needs the data.
The fix is to carry the originating user identity, and a purpose claim that explains why the action is being taken, through every hop of the chain. The receiving system enforces policy against the real principal and logs the real principal. The investigator gets back the chain of custody. The auditor gets back the linkage between the human request and the action that hit regulated data.
Propagation has to work across A2A, MCP, and direct data calls. Trust3 AI's implementation documents three hops of depth for A2A identity propagation, which covers the topologies most enterprises run today.
Replay as the unit of incident response
The right test of observability is whether an analyst can reconstruct what an agent did, end to end, from the record alone.
Given a flagged action ("the agent emailed a client an internal credit memo"), the analyst should be able to ask: which user initiated the chain, what was their original request, which retrievals informed the agent's reasoning, which tool calls were considered and rejected, which MCP server returned what, where in the run did the disclosure happen, and what data was exposed. Each question should be answered from the telemetry, not from interviewing the engineer who built the agent.
This is what replay means in practice. It is also what regulators are increasingly asking for. EU AI Act Article 12 obligations on logging for high-risk systems are written against precisely this property: the ability to reconstruct, after the fact, what a system did and why.
Where does your org actually stand?
The F500 benchmark scores your agent governance posture across discovery, observability, and security in 90 seconds. No signup required.
Get your score →Feeding the existing stack
Observability is most useful when it routes into the tools the security operations team already uses.
- SIEM. Agent telemetry as structured events, with the identity normalized to the originating user. Detections written against agent behavior look much like detections against any other workload once the identity is right.
- SOAR. Playbooks for agent-specific incidents: a kill-switch invocation, a prompt-injection detection, an unusual tool-call sequence.
- Data lake. Long-retention agent activity for trend analysis, drift detection, and the audit horizon.
- Audit and compliance. Periodic exports that map directly to control evidence for SOC 2, ISO 27001, and EU AI Act reporting.
The goal is to make agent activity legible to the systems and people that already run security operations.
Observability across the agent lifecycle
Design. The observability requirements are part of the agent's design record: what will be captured, where it routes, how long it is retained.
Development. Telemetry hooks are wired in at the protocol layer rather than retrofitted into agent code. Anything that depends on the agent owner remembering to log a particular event is fragile.
Pre-deployment testing. Replay is tested with synthetic incidents. If the test analyst cannot reconstruct a planted incident from telemetry alone, the observability is not done.
Deployment. Telemetry is live before the agent receives production credentials. There should be no period where the agent is in production and the audit trail is not.
Runtime. Tamper-evident capture continues. Storage retention runs against the audit horizon required by sector and region.
Continuous monitoring. Telemetry is the input to drift detection: longer prompt chains, new tools invoked, new data reached, new A2A peers. Drift surfaces as a change in the observability stream.
Decommissioning. Telemetry is retained for the documented audit horizon after the agent is retired, with chain-of-custody preserved.
Where observability aligns with standards
| Framework | Observability's contribution |
|---|---|
| NIST AI RMF | Measure (telemetry and metrics) and Manage (response to flagged behavior). |
| ISO/IEC 42001 | Operational records, monitoring, and continuous improvement evidence. |
| EU AI Act | Article 12 logging requirements and Article 14 human oversight enablement for high-risk systems. |
| SOC 2 / ISO 27001 | Audit logging, monitoring, and incident response control evidence. |
| HIPAA / PCI-DSS / GLBA | Identity-attributed access logs for regulated data reached through agents. |
Frequently asked questions
Most agent frameworks log a partial subset, in the agent's process memory, with the service-account identity, in a format that is not tamper-evident. That is useful for the developer debugging the agent. It is not what an incident responder or auditor needs.
You can, and you should. The data layer sees who reached the data, but it does not see why or under what reasoning. Without the agent's prompt and decision context, the data-layer log is a half-record.
Sample prudently on intermediate reasoning if cost is a concern, but never sample tool calls, data accesses, or A2A hops. The fact that an action was taken is not optional to log.
The capture should include them, with the same redaction and access controls as any other sensitive log content. The investigator needs to know what the agent said. The unauthorized reader of the log does not.
Yes, but with friction. Telemetry wired in at the protocol layer is the cleanest path. Adding it after the agent is in production usually means agent-side instrumentation, which depends on the agent owner and is harder to standardize.