HomeArticles

Chapter 8 of Kubernetes cost and autoscaling

Karpenter Node Consolidation Not Working? Here's the Real Fix

Why Karpenter nodes will not consolidate and how to fix it: PodDisruptionBudgets, instance and zone limits, taints, legacy autoscaler config, logs.

Fix Karpenter consolidation: relax PDBs, broaden instance types, allow spot and arm64, relax topology spread, unify taints.

If you're running Karpenter and your nodes aren't consolidating, you're bleeding money. Not slowly — aggressively. In 2026, the gap between optimal and actual spend on Kubernetes can be 30-40% (Kubernetes Cost Optimization: A 2026 Guide to Reducing ...). And when Karpenter's consolidation engine stalls, that gap grows by the hour.

I'm Nishaant Veer Dixit, founder of SIVARO. We build data infrastructure and production AI systems.

Here's what we'll cover: why consolidation stops working, how to diagnose it without guessing, and the specific fixes. No fluff. No "monitor your metrics" nonsense. Concrete steps.

Why Your Nodes Aren't Consolidating — The Silent Cost Drain

Most people assume Karpenter consolidates aggressively by default. That's wrong. The default settings are cautious — intentionally. But if you've tweaked them (or not), consolidation can stall completely.

Consolidation is Karpenter's mechanism to replace multiple underutilized nodes with fewer, better-suited ones. When it's working, you save money and improve bin packing. When it's not, you're paying for half-empty nodes.

The most common reason? You haven't given Karpenter enough room to swap nodes. It needs to find a replacement node that can fit all the pods from the target nodes — and if that replacement is more expensive than current, it won't trigger. The logic is: only consolidate if the new node(s) cost less than the old ones. Fair enough. But your instance type selection might be limiting its options.

Here's a snippet from a Karpenter provisioning spec:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["c", "m", "r"]
        - key: "karpenter.k8s.aws/instance-generation"
          operator: Gt
          values: ["2"]
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 1m

Looks fine, right? But notice: no spot instances allowed. No t or i categories. And the generation constraint is Gt 2 — meaning generation 3 and above. That severely limits the cheaper instances Karpenter can consolidate into. If your current nodes are mostly c5.xlarge (generation 5), and Karpenter can consider only c6i or c7g, you're blocking a huge price range (Karpenter EC2 node selection cost efficiency).

Relaxing instance generation to include older (but cheaper) generations can cut compute costs — and get consolidation working again.

PodDisruption Budgets: The Number One Culprit

Consider Deployments with PDBs set to minAvailable: 3 and replicas of 3. That means a pod can never be evicted because you'd drop below 3. Karpenter can't consolidate a node if even one pod on it is protected by a PDB that blocks eviction.

The fix is simple — change PDBs to allow one replica down during voluntary disruptions:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

But here's the kicker — StatefulSets with PDBs set to maxUnavailable: 0. Stateless workloads can survive a node drain; stateful ones often can't. But if you set maxUnavailable: 0 on everything, Karpenter becomes useless.

Contrarian take: Most people think PDBs are a safety net. They are. But excessive PDBs are a cost trap. You need to audit every PDB and ask: "Can we tolerate one replica being unavailable for 2 minutes?" If yes, change it to maxUnavailable: 1. If not, you're paying premium for availability you might not need.

Run a script that checks all PDBs and flags any with maxUnavailable: 0 or minAvailable equal to replicas. That's your consolidation blocker list.

Instance Type Constraints and Availability Zone Bias

Karpenter chooses instances based on your requirements. But if you've hardcoded specific instance families or excluded cheap options — consolidation stalls because there's no valid replacement node that fits your constraints.

A common pattern: companies lock themselves into c5 and m5 because that's what they've always used. Meanwhile, in 2026, AWS has cheaper Graviton-based c7g instances that offer better price/performance for most workloads. Karpenter can't consolidate c5 nodes into c7g if you didn't include c7g in your requirements.

Here's a better approach:

spec:
  requirements:
    - key: "karpenter.k8s.aws/instance-category"
      operator: In
      values: ["c", "m", "r", "t", "i"]
    - key: "karpenter.k8s.aws/instance-hypervisor"
      operator: In
      values: ["nitro"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["amd64", "arm64"]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["on-demand", "spot"]

Notice arm64 and spot. Karpenter uses spot instances aggressively — they can be 60-70% cheaper. Without them, your consolidation options shrink dramatically. And if you have both AMD64 and ARM64 workloads, Karpenter can pick the cheapest architecture per node.

Availability zone bias is subtler. If your pods have topologySpreadConstraints that force even distribution across zones, Karpenter might not be able to consolidate because you need one node per zone to maintain spread. Take a cluster with 3 AZs, each with a single underutilized node — Karpenter can't combine them into one node because the spread constraint prevents it. Solution? Relax topology spreads or use whenUnsatisfiable: ScheduleAnyway.

Taints, Tolerations, and Node Templates Gone Wrong

A common issue: workloads split into separate node pools with incompatible taint/toleration setups. Karpenter sees these as isolated groups — it won't consolidate across node pools unless the tolerations match.

The fix: unify tainting strategy. Use the same core taints across all node pools, or better, avoid taints unless absolutely necessary. If you need GPU isolation, use node selectors and resource requests instead of taints. Karpenter handles scheduling more flexibly that way.

Bad node template labels cause consolidation failures too. Karpenter uses labels to match pods to nodes. If your node template has labels that don't match any pod selector, the node gets created but instantly becomes empty — and consolidation doesn't touch empty nodes (it's designed to reduce waste on underutilized, not empty).

Cluster Autoscaler Legacy Configs Interfering

Cluster Autoscaler next to Karpenter causes node flapping; remove CA, or keep separate node groups if CA must stay.

Are you still running Cluster Autoscaler alongside Karpenter? In 2026, that's like keeping a flip phone next to your iPhone.

Cluster Autoscaler and Karpenter fight over node management. Karpenter might consolidate a node; Cluster Autoscaler might scale it back up because of a misaligned scale-down delay. The result? Flapping. Nodes disappear and reappear every few minutes. Karpenter's consolidation logs show "node removed" but you also see "node created by CA" simultaneously.

The fix is clear: remove Cluster Autoscaler entirely. Karpenter does both horizontal scaling and consolidation better (Karpenter vs Cluster Autoscaler: Which to Use in 2026). The migration isn't hard — just delete the CA deployment and scale-down its node groups.

But if you can't remove CA (maybe compliance), at least coordinate them. Set CA's scale-down disabling annotation on the node groups that Karpenter manages. Or better, use separate node groups for CA and Karpenter. That's ugly but works.

Debugging Consolidation with Karpenter Logs

You don't need a PhD to figure out why consolidation didn't fire. Karpenter logs are surprisingly clear — if you know where to look.

First, increase log verbosity:

kubectl patch -n karpenter deploy karpenter -p '{"spec":{"template":{"spec":{"containers":[{"name":"controller","args":["--log-level=debug"]}]}}}}'

Then watch the disruption controller:

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -c controller | grep -i disruption

You'll see messages like:

  • "could not consolidate node, action was action=delete, reason=node removal would cost more" — means the replacement node would be pricier.
  • "cannot remove pod pod-name due to PDB" — PDB blocking.
  • "node cannot be consolidated because it has 0 pods" — empty node, ignored.
  • "consolidation skipped, insufficient candidate nodes" — not enough underutilized nodes to make it worthwhile (thresholds).

The cost comparison logic is key. Karpenter computes the price of current nodes vs proposed nodes using the AWS pricing API. But if your spot pricing data is stale or you've excluded spot, the comparison is skewed. A small script can cache spot prices daily and feed them into Karpenter via a configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: karpenter-global-settings
  namespace: karpenter
data:
  spot-to-ondemand-price-ratio: "0.4"

That's a fake ratio to use when spot prices aren't available — forces Karpenter to assume spot is cheaper. Adjust based on your region.

Another diagnostic trick: use the dry-run consolidation.

kubectl drain --dry-run=server node-name

Kubectl's dry run will show you which pods would be evicted and why. If it fails, you've found your blocker.

Real-World Fixes

Three issues often block consolidation on AI training nodes:

  1. Pod topology spread constraints — maxSkew: 1 across zones. With many zones, even a lightly used node can't be consolidated because dispersing its pods would violate skew.

  2. Node template requirement instance-family: "p4d" — locked to p4d.24xlarge. Those are $30/hr. Karpenter can't find a cheaper replacement if p5 or cheaper p3 aren't allowed.

  3. Stale pricing data — spot prices for p3.16xlarge can be far cheaper, but a stale Karpenter pricing cache shows on-demand prices.

The fixes:

  • Change maxSkew to 2 and use whenUnsatisfiable: ScheduleAnyway for training pods.
  • Broaden instance family to include p3, p4d, p5, and even g5 (for lighter training runs).
  • Clear Karpenter's pricing cache every hour via a cronjob.

Cost Impact: What Inefficient Consolidation Costs You

Let's put numbers on it. A typical cluster in 2026 with 100 nodes, average $2/hr per node, running at 50% utilization. If consolidation works, you can reduce to 60 nodes — $120/hr saved. That's $105K/year.

But if consolidation is broken, you're not just paying for empty nodes. You're paying for the pods that could've been packed tighter but aren't. The right word count for consolidation failures is "silent burn" — it doesn't crash anything, it just eats your budget.

Tools like ScaleOps, Cast AI, and Kubecost track this exact cost gap (Cast AI vs ScaleOps vs StormForge vs Kubecost).

And it's not just about current nodes. When Karpenter doesn't consolidate, new pods often trigger new node launches instead of filling existing ones. That doubles waste. The entire bin-packing advantage of Karpenter goes away.

FAQ

Karpenter consolidation FAQ: check PDBs first, fix causes rather than forcing it, and timing won't help a stuck setup.

Q: Karpenter node consolidation not working even with the default settings. What's the first thing to check?
A: PodDisruptionBudgets. I guarantee you have at least one PDB blocking evictions. Run kubectl get pdb -A and review every entry. Change maxUnavailable: 0 to maxUnavailable: 1 for non-critical workloads.

Q: Can I force consolidation?
A: Yes. kubectl annotate node <name> karpenter.sh/do-not-disrupt=false won't help — that's the inverse flag. Instead, use kubectl annotate node <name> karpenter.azure.com/consolidation-override="true" (if on Azure) or trigger manually by draining a node. But forcing bypasses safety checks. Better to fix the root cause.

Q: Does spot instance selection affect consolidation?
A: Absolutely. If you don't allow spot, Karpenter can't propose a cheaper replacement node. Include spot in your node pool requirements. Use karpenter.sh/capacity-type: "spot" if cost is your primary goal. Karpenter vs cluster autoscaler cost savings 2026 are largely driven by spot usage — CA doesn't handle spot as well.

Q: I see "node removal would cost more" in logs. What now?
A: Broaden instance types and categories. Your current nodes are probably expensive. Allow smaller instances, older generations, or spot. Also check if you have any pods requesting large amounts of memory or CPU that force Karpenter to propose equally expensive replacements.

Q: How often should consolidation run?
A: Karpenter's default is consolidateAfter: 5m. For cost-sensitive clusters, drop it to 1m. For stable clusters, keep 5m. But if consolidation never fires, timing doesn't matter. Focus on constraints.

Q: My cluster has multiple NodePools. Does consolidation work across them?
A: No. Each NodePool is independent. If you have a pool for GPU and another for CPU, Karpenter won't consolidate a GPU node into a CPU pool. You need to unify requirements or accept some over-provisioning.

Q: Should I use Karpenter or Cluster Autoscaler in 2026?
A: Karpenter, unless you have regulatory restrictions forcing CA's simpler model. Karpenter's consolidation and spot handling are superior. See Karpenter vs Cluster Autoscaler for a breakdown.

Q: How do I know if my cluster is even a candidate for consolidation?
A: If you have nodes below 60% utilization for CPU or memory. Use kubectl top nodes or a tool like KRR (Kubernetes Rightsizing in 2026). If every node is above 80%, consolidation won't help — you need vertical scaling instead.


Consolidation not working is a feature, not a bug. Karpenter is designed to err on the side of safety. Your job is to tell it when safety isn't needed. Remove PDB blockers, broaden instance ranges, include spot, and set realistic topology spreads. Do that, and your nodes will start disappearing — and your bill will follow.

If you're stuck, open Karpenter's logs and look for the exact reason. Nine times out of ten, it's a PDB or a restrictive requirement. Fix that, and you reclaim 20-30% of your cluster cost.

Want to talk this through?

Sivaro Enterprise →