HomeFind your caseEvery deploy waits on one DevOps person
Case · Enterprise
A common situation · not a client storyEvery deploy waits on one DevOps person.
Engineering teams on Kubernetes where getting code from a repository to a running service depends on one or two people and a pipeline nobody fully trusts.
What you're probably seeing
If this sounds familiar, you're in the right place.
- Developers wait for one person to deploy, or to fix the pipeline.
- Kubernetes manifests are copied and edited by hand for each service.
- Nobody is sure what is running in production, or who changed it.
- A failed deploy is fixed on the cluster, not in the code.
- Adding a new service takes days of setup.
The usual ways out
What teams usually try. Each works somewhere. None works everywhere.
| The way | Works when | The catch |
|---|---|---|
| Hire another DevOps engineer | The work is mostly operations, not repetition | Moves the bottleneck to two people |
| Adopt a managed platform (a PaaS) | Your services fit its model | Less control, and costs that grow with you |
| GitOps with Argo CD or Flux | You want the cluster to follow the repository | Needs clean manifests and discipline first |
| Helm charts or templates for every service | Services look alike | Templates sprawl, and still need someone to own them |
| A self-service deploy API with checks | Developers should deploy without asking | Needs policies, verification and an audit trail built in |
| Write it all down in a runbook | Deploys are rare | Runbooks drift, and still need the expert |
What other companies did. Their published results, not ours.
We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.
| Company | What they did | What they published | Source |
|---|---|---|---|
| RedditIn 2022: 20 production Kubernetes clusters, with 92 infrastructure engineers supporting 706 application engineers. Taken from a KubeCon North America talk. | Built declarative self-service APIs: custom resources backed by controllers, with Flux for syncing. Developers now request a namespace through one resource instead of copying Helm or Kustomize files. | New cluster setup from 30+ hours to about 2 hours. Cluster upgrades take about 1 hour. Production controllers grew from 4 to 12 in the year. Before the change, review added about 24 hours, and getting a namespace out could take almost a week because of time-zone differences. No figure is given for namespace turnaround after the change.They noted: They call the orchestration cluster a single point of failure. They say limiting the kinds of clusters made the fleet easier to manage, and they describe the rebuild as still in progress. | Reddit No Longer Haunted by Drifting Kubernetes ConfigurationsJan 2025 |
| AdobeFlex platform serving 3,000+ developers in 100+ teams, with 6,000+ services across nearly 50,000 environments. | Replaced a legacy queue-based deployment platform with GitOps on Argo CD and related Argo tools. Onboarding, secrets and deployment controls were built into a self-service platform. | 10,400 pipelines migrated or decommissioned. About 80% of previously identified services removed. Nearly 19,000 production environments now on the platform. 'Significant reductions in deployment queue times', with no figure given.They noted: Their biggest regret was relying on standalone migration scripts, which left manual work on early teams. Mandates created urgency but not trust. Teams adopted once they saw gains. | Enabling GitOps at Scale at Adobe with ArgoApr 2026 |
| LyftAbout 1,000 microservices, each in its own repository. | Built an automatic deployer that moves each change through standard gates (open alerts, bake time, deploy hours, known-bad commits) and only proceeds when every gate passes. Teams can add their own gates. | Commits per production deploy fell from about 3 to about 1.4. Merge to production takes about 45 minutes. 90+% of about 1,000 services deploy to production this way. No timeframe given.They noted: They say they could have rolled it out to other teams much earlier. Some engineers assumed the automation was smarter than it was, and documentation alone did not fix that. | Continuous Deployment at LyftJul 2022 |
| ChainguardTwo regional GKE clusters running its API, registry front end, webhooks and internal services. Headcount not stated. | Moved serving from self-run Kubernetes to Cloud Run, a managed platform, with shared Terraform modules for each service. Cloud Run follows the same container contract as Knative, which they ran on GKE, so services could move with largely the same code. They ran both stacks side by side and shifted traffic gradually. | No numbers given. GKE was decommissioned in April 2024. Idle dev environments now scale to zero, and the author says external users did not notice the move.They noted: Cloud Run had no multi-regional services, so they built their own layer for that. Standard modules made new services easier to start and secure by default. | Migrating Chainguard's Serving Infrastructure to Cloud RunDec 2024 |
Our read
Same call. These companies removed the human step from deploys with declarative config and automatic checks, which is where our tests usually lead at their scale.
Reddit's write-up shows copy-pasted manifests, review that added about 24 hours, namespace requests that could take almost a week, and drifting clusters. So it gave developers a self-service resource rather than GitOps alone, in line with the catch in the table above: GitOps needs clean manifests first. Lyft automated deploys behind standard gates, which is our self-service deploy with checks. Adobe moved to GitOps, but the migration surfaced stale services, about 80% of those identified were removed, and its biggest regret was relying on ad hoc scripts, so we would still map what is running and who can change production before choosing. Chainguard's move to a managed platform fit because its services already followed the same container model.
What we would test first: Trace every step from a merged change to a healthy service, and who does each one.
What the diagnostic would test
The tests that rule the wrong ways out.
On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.
- 1
Every step from a merged change to a healthy service, and who does each one.
- 2
How manifests are made, checked and applied.
- 3
What happens when a deploy fails, and how long recovery takes.
- 4
Who can change production today, and whether it's recorded.
- 5
What a new service needs before its first deploy.
This week, without us
Three things you can do yourself.
- 1
Time your next deploy from merge to a healthy service, and note every wait.
- 2
List who can change production, and how you'd know they did.
- 3
Pick one service and check whether its manifests in the repository match what runs.
Kubernetes cost and autoscaling
Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.
This page describes what teams in this situation commonly face. It is not a client story.