AI Agent Versioning in Production: The Guide We Wrote After Breaking Things
You've built an AI agent. It works. Then you update the prompt, and suddenly your customer support bot starts screaming at users in French. Or your code-generating agent swaps from Python 3.11 to Python 3.13 without telling anyone. Welcome to versioning hell.
This is about AI agent versioning in production — not the academic theory, but the practical mess of keeping agent behavior deterministic when every deployment feels like Russian roulette.
By the end of this, you'll know how to version prompts, tools, model configurations, and the agent's internal state machine. You'll see real patterns from teams at companies shipping agents to production. And you'll learn why most people approach this wrong.
Why Your Agent's Versioning Strategy Will Fail
Most teams treat AI agent versioning like API versioning — slap a number on it, call it v2, done. That's cargo cult thinking.
Here's the problem: an AI agent isn't a static function. It's a stochastic system with four moving parts:
- The prompt (system + user messages)
- The model (provider, temperature, max tokens, etc.)
- The tool definitions (functions, schemas, implementations)
- The orchestration logic (loops, memory, routing)
Change any one, and you might break the whole chain. I've seen teams at Brex in 2024 ship a "minor prompt tweak" that caused their document parser agent to hallucinate 30% more. They had no way to roll back cleanly.
The core insight: version the whole pipeline, not just the code. You need a single artifact that locks in all four components. That's what I'll show you.
What We Actually Mean by "Versioning an AI Agent"
Let's kill a myth: semantic versioning (2.1.3, etc.) is mostly useless for agents. Breaking changes in a stochastic system are rarely binary. A prompt change might degrade recall from 87% to 82% — that's not "broken," but it's worse. Semver can't express that.
Instead, think of versioning as a snapshot of the entire agent configuration at a point in time, tagged with a unique ID and metadata (author, date, commit hash, evaluation scores). This artifact should be deployable, testable, and comparable.
At SIVARO, we define an agent version as:
yaml
# agent_version.yaml - canonical version manifest
version: "2026-08-01.a3b2c"
created_at: "2026-08-01T14:30:00Z"
author: "[email protected]"
model:
provider: anthropic
name: claude-3-5-sonnet-20260801
config:
temperature: 0.2
max_tokens: 4096
prompt:
system: |
You are a financial analyst AI. You have access to tools for querying transaction data.
Rules: never share raw PII, always cite sources, use ISO currency codes.
user_template: "Analyze the following account: {{account_id}}"
tools:
- name: query_transactions
version: "2.4.0"
commit: "a1b2c3d"
- name: get_balance
version: "3.1.1"
commit: "e4f5g6h"
orchestration:
type: react_loop
max_iterations: 10
memory:
type: sliding_window
size: 20
evaluations:
precision: 0.94
recall: 0.91
rollback_from: "" # empty means first deploy
That YAML captures everything. When you deploy, you pin to this exact version. No surprises.
The CI/CD Pipeline for AI Agents: It's Different
A standard software CI/CD pipeline compiles code, runs unit tests, and deploys binaries. For AI agents, you need to also evaluate semantic behavior — not just whether the code compiles.
Here's the pipeline we've settled on after breaking our own agents about 40 times:
Git push → Lint configs → Validate YAML schemas →
Evaluate on held-out test set → Compare scores against current prod version →
If scores drop > 3% → block deployment →
Else → Deploy to staging → Canary 1% traffic for 1 hour →
Monitor metric drift → Full rollout → Tag version in registry
The evaluation step is the hardest. You need a representative test set with expected outputs (golden answers). For a customer support agent, that's 200+ conversation transcripts with annotated "correct" responses. You run the candidate version against these and measure exact match, semantic similarity, or human-rated correctness.
I've seen teams at Anthropic in 2024 use three separate evaluation sets: adversarial (edge cases), nominal (happy path), and regression (known failure modes). If the candidate version fails any one, it doesn't deploy. Simple rule, hard to enforce when your product team is shouting for a new feature.
Practical Pipeline Snippet (GitHub Actions + custom runner)
yaml
# .github/workflows/agent-deploy.yml
name: Agent CI/CD
on: [push]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Validate agent config
run: python validate_agent_version.py agent_version.yaml
- name: Run evaluations
run: |
python run_eval.py --test-set tests/golden.jsonl --agent-config agent_version.yaml --output eval_results.json
- name: Compare with production baseline
run: |
python compare_scores.py --candidate eval_results.json --baseline s3://agent-registry/production/scores.json
- name: Block if regression > 3%
if: failure()
run: echo "Agent version regressed. Blocking deploy." && exit 1
- name: Deploy to staging
run: python deploy_agent.py --environment staging
This isn't fancy. It works. We've caught dozens of regressions this way — silent failures that would have gone to production and made users angry.
Prompt Versioning: The Thing Everyone Gets Wrong
You'd think prompts would be easy to version: store them in Git, done. Wrong. The problem is prompt embeddedness — prompts are often baked into code strings, environment variables, or even directly copied from ChatGPT.
I've walked into codebases where the system prompt is a 200-line heredoc in a Python file with no date, no author, no commit message explaining why a change was made. That's a disaster waiting to happen.
Rule: every prompt lives in its own file, with its own version history, separate from the code.
python
# bad - prompt embedded in code
system_prompt = "You are a helpful assistant. ..." # no versioning
# good - prompt loaded from versioned file
from agent_config import VersionedPrompt
system_prompt = VersionedPrompt.load("system-prompt-v2.3.yaml")
prompt_version = system_prompt.version # "v2.3"
We store prompts as YAML files in a prompts/ directory with a manifest:
yaml
# prompts/system-prompt-v2.3.yaml
version: "v2.3"
previous_version: "v2.2"
changes:
- "Added rule about not generating code on production accounts"
- "Changed tone from 'polite' to 'direct' per customer feedback"
date: "2026-07-28"
author: "[email protected]"
content: |
You are a financial analyst AI...
When you change a prompt, you update the file, commit, and the CI/CD pipeline triggers a new evaluation. If the evaluation passes, the manifest gets a new version tag. You can always roll back by deploying the old prompt file.
Model Versioning: You Can't Assume the API Stays the Same
August 2026. Anthropic, OpenAI, Google — all have released new models in the last four months. Claude 3.5 Sonnet changed its behavior subtly in April. GPT-4 Turbo got a new prompt attention mechanism in June. Your agent using gpt-4-turbo-2026-04-09 might behave differently than one using gpt-4-turbo-2026-07-15. Even at the same temperature and top_p.
The only safe approach: pin your model to a specific snapshot or date. Most providers now offer versioned aliases:
claude-3-5-sonnet-20260601(pinned to a specific model weight snapshot)gpt-4-turbo-2026-07-15(pinned to a specific training checkpoint)
Use these. Never use claude-3-5-sonnet-latest in production. That's asking for a surprise on Tuesday morning.
We also store the model config in the version manifest (as shown earlier), including temperature. I've seen teams not version temperature — then someone changes it from 0.2 to 0.8 "for testing," forgets to change it back, and suddenly the agent starts generating long-winded, random outputs in production.
Tool Versioning: The Silent Killer
Your agent calls tools: APIs, databases, webhooks. Those tools have their own versions. When you update a tool's schema (add a new parameter, change the response format), the agent's reasoning breaks.
Example: your financial agent uses a tool get_transactions(start_date, end_date). In version 1, it returns a list of transactions. In version 2, it also returns a summary. The agent was prompted to process the raw list. Now it gets a summary it wasn't expecting. It might hallucinate or try to parse it incorrectly.
The fix: pin tool versions in the agent manifest, and use API versioning on the tool side. When you change a tool, you create a new endpoint version. The agent stays pinned to the old version until you explicitly update the manifest and re-evaluate.
python
# tool versioning example
tools = {
"get_transactions": ToolSpec(
version="2.4.0",
endpoint="https://api.company.com/v2.4/transactions",
schema="schemas/transactions_v2.yaml"
),
"submit_report": ToolSpec(
version="1.1.0",
endpoint="https://api.company.com/v1.1/reports",
schema="schemas/reports_v1.yaml"
)
}
We validate tool schemas on every agent deployment — the agent's tool call definitions must match the tool's actual API. We wrote a small library that uses JSON schema to validate at deploy time. It catches mismatches instantly.
Orchestration Versioning: Your Agent's Brain
The orchestration logic — how the agent loops, when it asks for clarification, how it handles errors — is the hardest thing to version. It's often scattered across code: a while loop here, a recursion limit there, a fallback path in another file.
Most people think the prompt is the agent's brain. It's not. The orchestration logic is.
I worked with a team at a B2B SaaS company in 2025. They updated their orchestration from a simple ReAct loop to a more complex tree-of-thought approach. The prompt didn't change. The model didn't change. But the agent suddenly started hallucinating because the new loop allowed it to revisit previous steps and override its own conclusions. The team had no idea the orchestration was versioned. They rolled back the code and lost a week.
Treat orchestration logic as a first-class versioned component. Store it as a configurable pipeline:
yaml
# orchestration_v3.yaml
version: "3"
loop_type: "react_with_verification"
max_iterations: 15
verification:
enabled: true
passes: 2
error_handling:
on_tool_failure: "retry_up_to_3_times"
on_context_overflow: "summarize_and_continue"
memory:
type: "sliding_window"
window_size: 30
summarization_frequency: "every_10_turns"
When you change the orchestration, you commit a new YAML file, create a new agent manifest, and run the full evaluation pipeline. No silent changes.
Rollbacks, Canaries, and the Horror of Tags
Let's talk about the moment you realize your new agent version is worse than the old one — and you need to go back.
If you've done versioning right, rolling back is trivial: redeploy the old manifest. The agent picks up the old prompt, tools, model config, and orchestration. Done.
But how do you know you need to roll back? You need monitoring that compares current performance to the previous version. We track three metrics in production:
- Success rate — did the agent complete its task without error?
- User feedback — thumbs up/down after each agent interaction
- Latency — is the new version slower? (Model changes often increase time)
We deploy to canary (1% of traffic for 1 hour), then ramp to 10% for 2 hours, then 50% for 1 hour. At any point, if metrics dip below thresholds, we auto-rollback. The version manifest makes this clean — we store all past manifests in a registry (S3 or artifact store), so we can revert instantly.
One trick: tag every deployment with the agent version hash. In logs, you'll see deployment_version="2026-08-01.a3b2c". That lets you correlate user complaints to a specific version. Without this, you're debugging in the dark.
The Registry: Your Single Source of Truth
You need a central place to store all agent versions — prompts, tool specs, model configs, evaluation results, deployment history. We use an S3 bucket with a simple JSON index:
python
# pseudo-code for agent registry
registry = S3Registry(bucket="agent-versions-prod")
index = registry.get_index()
# returns list of version metadata:
# [{"version":"2026-08-01.a3b2c","deployed_at":"2026-08-01T14:30:00Z","eval_scores":...}]
# deploy a specific version
def deploy_version(version_id: str):
manifest = registry.get_manifest(version_id)
# parse manifest, deploy to k8s/ecs/etc.
This registry is your source of truth. When you want to roll back, you find the previous successful version from the index and deploy it. When you want to know what's running in prod, you query the current deployment's manifest.
FAQ — Things People Actually Ask Me
Q: Do I need to version the model weights themselves?
A: No. You pin to a provider's snapshot. If you're running an open-weight model (Llama 3, Mistral), then yes — store the weights file hash in the manifest. We use sha256sum of the model weights directory.
Q: How do I handle prompt changes that aren't backward compatible?
A: Create a new prompt file with a new version. The old version stays in the registry. Your CI/CD evaluation will catch regressions. If the new prompt fails, you roll back to the old version by deploying its manifest.
Q: What if my agent calls external APIs that don't have versioning?
A: You can't control them. But you can log the actual API response and compare it to expected schemas. We add a validation step in the agent's tool call that checks the response against a stored schema — if it fails, the agent logs an error and handles gracefully.
Q: Should I version the training data for the underlying model?
A: Only if you're fine-tuning. Fine-tuning? Yes, version the dataset, the base model, and the training config. Store them in the manifest. This is another level of complexity — many teams don't need it yet.
Q: How do you test agent versions without a perfect golden dataset?
A: Use adversarial red-teaming. Have a team (or automated LLM) try to break the agent — ask it to bypass safety rules, give contradictory instructions, etc. Track how many adversarial cases it fails. If a new version fails more, block deployment.
Q: What happens when you deploy a new model version and it changes behavior on same prompt?
A: This has happened to us. We noticed Claude 3.5 Sonnet (April 2026 snapshot) stopped following a specific formatting rule. We had to revert to the March snapshot. That's why you pin to a dated snapshot, not a version alias.
Q: How do you version the agent's long-term memory / knowledge base?
A: That's a separate axis — data versioning. The knowledge base (vector store, database) has its own version (data update time, chunking method, embedding model). We include a knowledge_base field in the manifest with snapshot ID.
Q: What's the biggest mistake you see teams make?
A: Not versioning the orchestration logic. They treat it as "application code" and iterate freely. Then they wonder why the agent behavior changed. Version everything together — prompt, model, tools, orchestration. One artifact.
What We Actually Use at SIVARO (As of August 2026)
We run about 15 production agents — financial analysis, customer support, internal ops. Our stack:
- Version registry: S3 + DynamoDB index (for fast lookups)
- Prompt files: Git with YAML, separate repo from application code
- Tool specs: Protobuf definitions, stored in registry with version
- Model configs: Pinned to provider snapshots, updated monthly after eval
- Orchestration: Python classes, serialized as config (YAML)
- CI/CD: Custom GitHub Actions (the snippet above is close to reality)
- Deployments: Kubernetes, one pod per agent version, canary via service mesh
We do not use semantic versioning. We use date-based hashes (2026-08-01.a3b2c). It's easier to read, and the hash ensures uniqueness.
Conclusion: Ship Agents You Can Trust
AI agent versioning in production isn't about bureaucracy. It's about sleep. When you have a single manifest that locks in every behavior — prompt, model, tools, orchestration — you can deploy with confidence. Rollback in seconds, not hours.
The teams that succeed are the ones that treat agents as strict state machines, not magic black boxes. Version the whole system. Test on representative evaluations. Deploy in canaries. Monitor and compare to baseline.
This isn't the sexy part of AI. It's the necessary part. Ignore it, and your agent will eventually do something you don't expect. In production, that's more than a bug — it's a user trust bomb.
Build the pipeline. Store the manifests. Sleep better.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.