SIVARO
One AI Agent Can't Be Trusted. Four Can't Either.
I spent the first half of 2025 debugging a system where three agents kept blaming each other for a corrupted database write. Agent A said Agent B issued the bad transaction. Agent B said Agent C's schema migration broke the table. Agent C said it never ran a migration.
All three were right.
The failure was in the orchestration layer's retry logic. Every agent's action was correct in isolation. The system as a whole was accountable to no one.
That's the problem at the heart of accountability in multi agent ai systems how it works — and it's not a theoretical concern. It's the difference between a system you can ship to a hospital in 2027 and a demo you delete after the conference.
What Accountability Actually Means in Multi-Agent Systems
Most people think accountability is logging. It's not. Logging tells you what happened. Accountability tells you who is responsible for the decision that caused it — and gives you the mechanism to do something about it.
In a single-agent system, accountability is simple. One model, one prompt, one context window, one verdict. You know exactly which code path executed.
Multi-agent systems break that contract. When you have four agents coordinating through a shared state store, a decision is rarely made by one entity. It emerges from the interaction.
So here's my working definition, refined through a dozen production deployments at SIVARO:
Accountability in a multi-agent system is the ability to trace any system-level outcome back to a specific agent's decision, in a way that is auditable, revocable, and reproducible — even when that outcome emerged from complex interactions.
Three properties matter. Let's go through each.
The Three Non-Negotiables
1. Attribution (Who decided?)
Every decision needs a caller ID. Not just "agent_3 decided to call the refund API" — but the full context: which prompt version, which parent task, which state snapshot, which retry count.
We use a trace context propagated through every hop. OpenTelemetry works, but you need to extend it with agent-specific semantics.
Here's the pattern we landed on at SIVARO after trying three different approaches:
python
# Trace context propagation — the SIVARO pattern
class AgentTraceContext:
def __init__(self, agent_id, task_id, parent_trace_id, decision_version):
self.agent_id = agent_id
self.task_id = task_id
self.parent_trace_id = parent_trace_id
self.decision_version = decision_version # prompt/model version hash
def to_headers(self):
return {
"X-Agent-ID": self.agent_id,
"X-Task-ID": self.task_id,
"X-Parent-Trace": self.parent_trace_id,
"X-Decision-Version": self.decision_version
}
# Every tool call, every LLM inference, every state mutation
# carries this context. Non-negotiable.
If an agent doesn't propagate this context, the system rejects its subsequent calls. We enforce this at the orchestration layer, not as a convention.
2. Decidability (Could it have decided differently?)
This is the one most people miss. A system is only accountable if you can trace a decision and understand what alternative decisions were available.
Think of it as counterfactual auditability. You need to be able to ask: "Given the same state, would the agent have made the same choice?"
This matters more than you'd think. In September 2025, we had a client in financial services almost get rejected by their risk department because one agent's response was stochastic — the same input produced different outputs across runs. The compliance team couldn't certify the system because they couldn't reproduce the decision.
We fixed it with deterministic decoding and constrained output schemas. It cost us some fluency in the agent's responses, but it made the system auditable.
If your agent uses temperature > 0, you need to log the seed. If you can't reproduce a decision, you can't be accountable for it.
3. Revocability (Can you stop it?)
Attribution without enforcement is just a post-mortem tool. Real accountability requires the ability to revoke an agent's authority mid-task, in response to a detected violation.
At SIVARO, we built a policy enforcement layer that sits between every agent and every tool. It's not advisory. It's a hard gate.
javascript
// Policy enforcement — hard gate between agent and tool
async function enforcePolicies(agentContext, toolCall, systemState) {
// Check if this agent has authority over the target resource
const authorization = await checkAgentPermissions(
agentContext.agent_id,
toolCall.resource
);
if (!authorization.allowed) {
// Deactivate the agent across all orchestration paths
await agentRegistry.suspend(agentContext.agent_id, {
reason: `Unauthorized access to ${toolCall.resource}`,
trace_id: agentContext.parent_trace_id,
timestamp: Date.now()
});
// Persist the violation for audit
await violationLedger.record({
agent: agentContext.agent_id,
action: toolCall,
policy: authorization.policy_id,
outcome: "SUSPENDED"
});
return { decision: "BLOCKED", reason: "Policy violation" };
}
return { decision: "ALLOWED" };
}
This isn't performance overhead. It's insurance. And in production, this gate has caught more bad behavior than any model guardrail we've deployed.
The Architecture Pattern That Makes It Work
What I'm about to describe is the coordination-trace-verification triple — the architecture we've standardized on at SIVARO for production multi-agent systems. It's not the only pattern, but it's the one that has survived contact with real workloads.
Layer 1: Coordination (The choreographer)
You need an explicit orchestration layer. Not autonomous agent-to-agent messaging. Explicit.
I know autonomous swarms are the hot thing. I also know that every production team I've talked to in 2026 has backed away from them for anything customer-facing. The failure modes are too hard to anticipate.
Your coordination layer should:
- Define tasks as state machines with explicit phases
- Log every state transition with a timestamp and actor ID
- Enforce timeouts per phase (an agent that takes 30 seconds to decide is a bug, not a feature)
- Maintain a task graph that shows dependencies between agents
We use temporal workflows for this. It gives us durable execution and replayable history. For a system where accountability matters, durable execution isn't optional — if your orchestration framework loses state, you've lost accountability.
Layer 2: Trace (The ledger)
Every action gets written to an append-only ledger. Not a database you can update. An actual immutable log.
We use a append-only event store (Kafka topic with retention infinity, compacted on the partition key). Every event includes:
- Agent ID
- Task ID
- Input hash
- Output hash
- Model version
- Timestamp
- Parent trace ID
This is the raw material for any accountability investigation. You want to ask "why did the system block this user?" — you should be able to query the ledger and reconstruct the entire decision chain.
Here's the schema we use:
sql
CREATE TABLE agent_decision_ledger (
event_id UUID PRIMARY KEY,
trace_id UUID NOT NULL,
parent_trace_id UUID,
agent_id VARCHAR(64) NOT NULL,
task_id UUID NOT NULL,
phase VARCHAR(32) NOT NULL,
decision_type VARCHAR(64) NOT NULL,
input_hash CHAR(64) NOT NULL,
output_hash CHAR(64) NOT NULL,
model_version VARCHAR(32) NOT NULL,
status VARCHAR(16) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
metadata JSONB
);
CREATE INDEX idx_ledger_trace ON agent_decision_ledger (trace_id);
CREATE INDEX idx_ledger_agent ON agent_decision_ledger (agent_id, created_at);
You will get asked: "How much storage does this cost?" Less than you think. Text tokens are small. You're logging hashes, not full payloads. A system processing 100K decisions/day costs maybe 2GB/month. Trivial.
Layer 3: Verification (The auditor)
You need automated verification that runs continuously. Not weekly reports. Real-time checks.
At SIVARO, we built a verification service that:
- Replays decisions against the ledger to detect anomalies (e.g., an agent that mutated state without a corresponding decision record)
- Checks invariant violations (e.g., "no agent can modify a record that another agent is currently processing")
- Computes drift metrics (e.g., "agent_3's decisions diverged from agent_2's expectations 14 times this hour — why?")
This service is the thing that actually catches problems before they hit your users.
The contrarian take: Most teams think the model is the source of errors. In our production data, orchestration bugs (retries, race conditions, state desync) outnumber model output errors 4:1. Verification at the orchestration layer catches more real failures than any guardrail you put on the model itself.
How to Build This for Your Team
If you're starting today, here's the practical playbook — the one we use when we onboard new teams at SIVARO.
Step 1: Define your accountability contract
Before you write any agent code, write down what "accountable" means for your specific system. Three questions:
- What decision types exist? Expense approval, code generation, customer refund, whatever. List them all.
- Who (which agent) is the decision owner for each type? This is your RACI matrix for agents.
- What is the escalation path? When the agent can't assert confidence, who gets involved?
Write this down. Review it with your team. The exercise is more valuable than the artifact.
Step 2: Instrument everything
You can't add instrumentation after the fact — by then it's too late, the interactions have already become too complex to trace. Build it in from day one.
- Add trace context to every agent call
- Add input/output hashing to every tool invocation
- Add phase logging to every state transition
Do this before you write your first agent prompt.
Step 3: Build the verification loop
Start with unit tests that check accountability invariants:
python
def test_every_decision_has_trace_context():
"""Golden test for our agent framework."""
result = agent.run(
task="analyze customer churn scenario",
context=AgentTraceContext(
agent_id="analyst_1",
task_id="task_42",
parent_trace_id="root_1",
decision_version="prompt_v3_hash"
)
)
assert result.trace_id is not None
assert result.decision_version == "prompt_v3_hash"
assert result.input_hash != result.output_hash # meaningful work happened
def test_orchestration_blocks_untraced_agent():
"""An agent without trace context cannot call tools."""
orchestrator = OrchestrationLayer()
with pytest.raises(TraceContextMissingError):
orchestrator.execute_tool_call(
agent_id="analyst_1",
tool_call={"name": "get_user_data", "args": {"user_id": 123}}
)
Then build integration tests that simulate multi-agent interactions and verify the ledger captures the full chain.
Step 4: Practice incident response with the ledger
Run a game day. Misconfigure an agent. See who catches it, and how long it takes to trace the failure through the ledger.
We did this with a client in healthcare and discovered it took their team 45 minutes to reconstruct what happened in a simulated failure. We got it down to 4 minutes by improving their query patterns and adding pre-built dashboards.
If you can't do this in under 10 minutes, your system isn't operationally accountable, regardless of how traceable it is in theory.
The Hard Trade-off: Accountability vs. Autonomy
There's a real tension here. The more accountable your system, the less autonomous your agents. Every gate, every version control, every verification step is a constraint on an agent's ability to act dynamically.
I've seen teams oscillate between extremes — fully autonomous swarms that cause production fires, and fully constrained systems that are basically elaborate if-else chains with extra steps.
You have to pick a middle path that matches your use case.
For customer-facing systems (refunds, support, content moderation): prioritize accountability. Costs: the agents feel "dumb" — they can't improvise as much. Benefit: you can certify the system to your head of legal.
For internal research tasks (code analysis, document summarization, forecasting): prioritize autonomy. Costs: you might get some inaccurate outputs. Benefit: more creative, useful results.
We built a "confidence gating" mechanism that lets agents operate in a lower-stakes mode (no code changes, no financial impact) with high autonomy, and graduate to a higher-stakes mode (with tool access, state mutation, external effects) when they need to. The gating is based on a model confidence score plus a human-in-the-loop check.
python
# Confidence gating — dual mode autonomy
def agent_execution_mode(predicted_risk: float, task: Task) -> ExecutionMode:
"""
Route agent execution based on risk prediction.
Low risk: full autonomy with trace logging.
High risk: require human confirmation.
"""
if predicted_risk < 0.2:
return ExecutionMode.AUTONOMOUS
elif predicted_risk < 0.7:
return ExecutionMode.AUTONOMOUS_WITH_LEDGER_BACKED_CONFIRMATION
else:
return ExecutionMode.HUMAN_GATED
This wasn't easy to build. It took us four iterations to get the risk prediction model accurate enough that human review volume wasn't overwhelming (we target around 8-12% of decisions requiring human confirmation — below that, the autonomy gains are marginal; above that, the system is too slow).
Where Current Tools Fail — and What Comes Next
Honest assessment: the industrial tooling for multi-agent accountability is immature.
- LangChain / LangGraph: trace well, but the orchestration layer is too permissive — agents can mutate shared state without going through a checkpoint. Bad for regulated industries.
- CrewAI: handles collaboration nicely, but the lead agent's decisions aren't traced in a way that lets you reconstruct "why this particular path."
- OpenTelemetry: great for infrastructure tracing, but doesn't understand agent semantics — it can't tell you why an agent chose a specific tool over an alternative.
The gap I keep seeing: there's no standardized "decision ledger" format that spans model vendors, orchestration frameworks, and infrastructure layers. We'll get there — the EU's AI Act and the US's proposed transparency requirements are going to force this. But we're 18-24 months from a real standard.
Until then, you build it yourself. I know. I've heard the complaint a hundred times. But the alternative — shipping an unaccountable multi-agent system and hoping it doesn't fail spectacularly in a way you can't explain — is not a viable path.
FAQ: Accountability in Multi-Agent AI Systems
Q: Is accountability the same thing as logging?
No. Logging captures events. Accountability requires the ability to attribute, reproduce, and revoke decisions. You can have excellent logging and zero accountability (we see this all the time — teams log every request, but can't tell you which agent decided what).
Q: What's the minimum viable accountability architecture for a small team?
Start with three things: (1) trace context propagation through every agent call, (2) an append-only ledger for agent decisions, (3) a one-click query tool for reconstructing any task's decision path. That's about two weeks of work and it will cover 80% of your accountability needs.
Q: Do I need a blockchain or distributed ledger?
No. Regular infrastructure with append-only properties suffices (Kafka topic, immutable database, etc.). Blockchain is unnecessary overhead here unless your compliance team requires it — and they won't.
Q: How much latency overhead does this add?
In our testing at SIVARO, full trace instrumentation with hashing and ledger writing adds 30-80ms per decision. For the systems we build — where decisions are LLM calls taking 500ms-2s — that's negligible (a 5-15% overhead). If you're in a sub-100ms latency regime for agent decisions, you'd need to sample or offload ledger writes asynchronously.
Q: Can you be accountable for an agent's output if the model is non-deterministic?
You can be accountable for the process, even if outputs vary. You just need to log: the exact prompt, model version, temperature, seed, and the decision version. If the output varies in a way that violates policy, you can trace which input variations caused it. Deterministic decoding (temperature=0 with consistent seed) makes reproduction easier — for high-stakes systems, we recommend it.
Q: Should I make all agents deterministic?
This is the wrong frame. Deterministic outputs aren't the goal — reproducible decision processes are. You want to know why a decision happened, not necessarily that it's identical across runs. For exploration tasks, higher temperature gives better results. Just ensure you can reproduce the exact conditions.
**Q: What's the most common mistake teams make?
They design for success, not for failure. They instrument for the happy path — "here's how the agent completes a task" — and don't build the machinery for "here's what happens when it fails and we need to figure out why." Accountability is an incident response tool. Design it as such.
The Bottom Line
You can't build production AI systems in 2026 without taking accountability seriously — not if you care about customers, compliance, or your ability to sleep at night.
The infrastructure exists. The patterns are proven (I've list them above; we've been running this since early 2025). The hard part is committing to making accountability horizontal — not a checklist item, but something that underpins every single interaction your agents have.
The system I described — coordination, trace, verification — treats accountability as a first-class architectural concern. That's what makes the difference between a system that fails gracefully (and you can explain exactly how and why) and a system that fails unaccountably (and becomes a liability).
Every production AI system will need this. If you're running multi-agent architectures now, you need it today.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.