Best Practices for Cost Efficient ML Deployment

Best Practices for Cost Efficient ML Deployment

I spent the first half of 2026 watching a client burn $180,000 a month on ML infrastructure. The worst part? Their models weren't even in production yet. They were paying for training runs that crashed at hour 40, idle GPU nodes, and a data pipeline that re-processed the same corpus three times a day.

Here's what I've learned running SIVARO: the difference between teams that pay $50K a month for ML and teams that pay $15K isn't their model. It's their infrastructure decisions.

"Best practices for cost efficient ml deployment" sounds like a checklist. It's not. It's a series of trade-offs where the right answer depends on your workload, your team, and how much sleep you need.

Let me show you what actually works.

The Billing Reality Check

Most teams think GPU cost is the problem. It's not always. The Spendark analysis of ML cloud costs for 2026 shows that storage and data transfer can account for 30-40% of total ML spend in production systems. Storage. Not silicon.

You know what that means? You can optimize your GPU allocation perfectly and still bleed money through S3 GET requests and ECR image pulls.

The first thing I do with any new client is pull their AWS Cost Explorer data and break it down by service. Not by tag — by service. The results are usually embarrassing. A fintech client in March 2025 discovered they were spending $11,000 a month on NAT gateway data transfer fees because their training pods were pulling the same 40GB model checkpoint from EFS every single time they restarted.

Fix that, and you've saved more than any GPU rightsizing.

Spot Instances vs On Demand for ML Training Cost

Here's the question everyone asks, and the answer is more nuanced than the internet suggests.

Spot instances are 60-90% cheaper than on-demand for the same GPU. An p4d.24xlarge (8x A100) runs about $32.77/hour on-demand. Spot pricing typically lands between $8 and $13. But you lose the instance when AWS reclaims it. That's the trade.

Most people think this makes spot unusable for training. They're wrong. The Lyceum spot instance training guide demonstrates that modern checkpointing strategies make spot viable for most training workloads — you just need to design for interruption from day one.

The trick is knowing when spot is safe:

Use spot when:

• Your training run has frequent checkpoints (every 15-30 minutes)
• You have idempotent data loading
• Your job can resume from a checkpoint without manual intervention
• You're running hyperparameter sweeps or experiments

Don't use spot when:

• You have a hard SLA on training completion time
• Your model has a single critical training run with no fallback
• Your data loading isn't idempotent and re-runs cost more than the savings

At SIVARO, we run a training platform that mixes both. The default is spot. On-demand is the exception, not the rule. One of our healthcare clients runs 80% of their training on spot and has a 97% job completion rate. The 3% that fail get retried on on-demand automatically.

Here's a Kubernetes node group config that does this:

yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: training-spot
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p4d.24xlarge", "p5.48xlarge"]
      nodeClassRef:
        name: gpu-node-class
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: training-on-demand
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p4d.24xlarge"]
      nodeClassRef:
        name: gpu-node-class

Then in your training job, use node selectors and a retry mechanism. The AWS blog on cost-optimizing AI workloads shows exactly this pattern — spot with checkpointing plus on-demand fallback.

Inference Is Where You Actually Bleed

Training gets all the attention. Inference is where the money disappears.

A 2025 client of ours ran a recommendation model serving 40 million requests a day. Their training spend was $60K a month. Their inference spend was $210K. Nobody noticed because training costs showed up as big scary line items while inference costs were spread across dozens of Lambda functions and ECS services.

The AWS cost optimization guide notes that inference typically dominates total ML cost once a model reaches production. My experience confirms it. The ratio is usually 3:1 or 4:1 inference-to-training for deployed systems.

Here's the harsh truth: most inference workloads don't need a GPU. Not even an L4.

We tested this. A natural language processing pipeline processing 5,000 requests/minute — classification, not generation — ran 2.1x faster on a CPU-only c6i.4xlarge with optimized batching than it did on a GPU instance. The GPU added latency from data transfer and kernel launch overhead. The CPU instance cost $0.68/hour. The GPU cost $4.20/hour.

The rule I use: if your inference workload is simple transformations, embeddings, or small model inference, start with CPU. Scale to GPU only when you have data proving you need it.

When you do need GPU inference, the Comparative Study of Cloud GPU Offerings published in 2026 provides something rare: an apples-to-apples benchmark of GPU inference costs across providers. Their finding — T4 and L4 instances remain the most cost-effective for inference-heavy workloads, while A100s only make sense for large batch sizes or multi-tenant serving.

The Autoscaling Myth

Everyone says "just autoscale." But Kubernetes HPA on CPU metrics will destroy your inference costs.

CPU-based autoscaling for ML inference is like using a thermometer to control your water pressure. It's the wrong signal. Your model's latency and request queue depth are what matter.

Here's what we use for a document processing service at SIVARO:

yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: inference-scaler
spec:
  scaleTargetRef:
    name: inference-service
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        query: |
          sum(rate(http_requests_in_flight{app="inference-service"}[30s]))
        threshold: "50"
        activationThreshold: "10"

That scales on in-flight requests, not CPU. The difference: we cut our idle replica count from 14 to 2, saving roughly $9,000 a month on a workload that never scaled down because CPU was always at 40% from background garbage collection.

But here's the counterintuitive part — aggressive autoscaling can cost you more than it saves.

The EaseCloud analysis of AWS ML cost reduction shows that 70% savings are achievable, but the biggest wins come from eliminating idle resources, not from micro-optimizing scaling behavior. A replica that scales up, processes 10 requests, and scales down still burns money on the scale-up time, container image pulls, and cold model loads.

For inference, I recommend a minimum of 2 replicas per service. One is a single point of failure. Zero means you're paying cold-start latency on every request. Two is the sweet spot between availability and cost.

Quantization Is Free Money

This is the least sexy advice in this article. It's also the most reliable.

Quantizing your model from FP16 to INT8 typically cuts inference cost by 50-70% with minimal quality loss. For some models, the quality loss is literally zero. The Spendark cost breakdown projects that quantization and model distillation will be the dominant cost optimization techniques by 2027 — and from what I'm seeing in production, that projection is conservative.

We quantized a legal document summarization model for a client in Q2 2026. The model went from FP16 to INT8 with a 0.3% drop in ROUGE score. The GPU cost dropped 63% because we could fit 3x more requests per batch on the same hardware.

Here's the PyTorch pattern:

python
import torch
from torch.ao.quantization import quantize_dynamic

model = load_model("legal-summarizer-v3.pt")
model.eval()

quantized_model = quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.Embedding},
    dtype=torch.qint8
)

torch.save(quantized_model.state_dict(), "legal-summarizer-v3-int8.pt")
print(f"Model size: {original_size_mb:.1f}MB -> {quantized_size_mb:.1f}MB")

That's it. Five lines of code and you've cut your inference bill in half.

If you're using vLLM or TensorRT-LLM for serving, quantization is even easier — those frameworks have native INT8 and FP8 support built in. There's no excuse not to do it.

The Data Pipeline Tax

Nobody talks about this because it's not glamorous. But the arxiv analysis of cloud AI infrastructure costs identifies data movement as one of the most overlooked cost drivers in ML systems.

The problem is simple: data pipelines re-run when they shouldn't. A pipeline that processes 10TB of training data every day will cost more than the GPU training run itself if the data doesn't change.

I worked with a media company in early 2026 that had a nightly job re-embedding their entire content library. The embedding model was re-run on every video every night, even though only 2% of the content was new. They were spending $28,000 a month on embedding compute for content that hadn't changed.

The fix was incremental processing. Only process data that's new or modified. We implemented this with a simple checksum-based system:

python
import hashlib
import boto3

s3 = boto3.client("s3")

def needs_processing(bucket, key):
    response = s3.head_object(Bucket=bucket, Key=key)
    etag = response["ETag"].strip('"')
    checksum = hashlib.sha256(key.encode()).hexdigest()
    
    # Store processed checksums in DynamoDB
    processed = dynamodb.get_item(
        TableName="processed_content",
        Key={"content_key": {"S": key}}
    )
    return processed.get("Item", {}).get("checksum", {}).get("S") != etag

Incremental processing cut their embedding cost by 94%. The pipeline now runs in 15 minutes instead of 3 hours.

Multi-Tenancy: The Uncomfortable Solution

Here's the contrarian take: the biggest cost efficiency win in ML deployment isn't technical. It's organizational.

Most companies run separate GPU clusters for each team. Data science team A has their own cluster. Data science team B has their own cluster. Both are underutilized because workloads are bursty. Team A trains at night. Team B trains in the morning. Both pay for 24-hour GPU capacity.

The AWS GPU cost optimization guidance explicitly calls out cluster consolidation as a top recommendation. I've seen the numbers play out in real life. A logistics company consolidated 4 GPU clusters into one shared platform in December 2025. Their GPU utilization went from 32% to 71%. Their cost per training run dropped 58%.

The catch? It requires political capital. Teams don't want to share infrastructure. They want their own sandbox. You have to build guardrails — namespace quotas, priority classes, and preemption policies — to make sharing safe.

Here's a Kubernetes ResourceQuota that makes sharing palatable:

yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-b-quota
  namespace: team-b
spec:
  hard:
    requests.nvidia.com/gpu: "8"
    limits.nvidia.com/gpu: "8"
  scopeSelector:
    matchExpressions:
      - operator: In
        key: priority-class
        values: ["high", "medium"]

Teams get guaranteed capacity for critical work and burst capacity when the cluster has free GPUs. Everyone saves money. Nobody feels starved.

Monitoring Cost, Not Just Usage

You can't optimize what you don't measure. But most cost monitoring in ML is broken because it measures utilization, not cost per unit of value.

Track these three metrics:

1. Cost per training run — total spend divided by number of completed runs
2. Cost per inference request — total inference spend divided by request count
3. GPU idle time — percentage of time allocated GPUs are doing useful work

The EaseCloud study found that idle GPU time accounts for an average of 35% of ML cloud spend. Thirty-five percent. You're paying for GPUs that are doing nothing.

We built a simple cost attribution tool for one client that tags every pod with its owning team and project:

python
import boto3
from datetime import datetime, timedelta

ce = boto3.client("ce")

response = ce.get_cost_and_usage(
    TimePeriod={
        "Start": (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d"),
        "End": datetime.now().strftime("%Y-%m-%d")
    },
    Granularity="DAILY",
    Filter={
        "Dimensions": {
            "Key": "LINKED_ACCOUNT",
            "Values": ["prod-account"]
        }
    },
    GroupBy=[
        {"Type": "TAG", "Key": "team"},
        {"Type": "TAG", "Key": "project"}
    ],
    Metrics=["UnblendedCost"]
)

for item in response["ResultsByTime"]:
    for group in item["Groups"]:
        keys = group["Keys"]
        amount = float(group["Metrics"]["UnblendedCost"]["Amount"])
        if amount > 100:
            print(f"{keys[0]} / {keys[1]}: ${amount:.2f}")

Run that weekly. Send it to the team leads. Watch costs drop — not because anyone is forced, but because visibility alone changes behavior.

The 70% Question

Can you actually reduce AWS ML costs by 70%? Based on what I've seen, yes — but only if you haven't already optimized.

The EaseCloud breakdown outlines a methodology that mirrors what I've implemented for clients:

1. Right-size instances (40% savings)
2. Use spot for training (60-90% savings on those workloads)
3. Autoscale inference (50% savings on idle capacity)
4. Quantize models (50-70% savings on inference compute)
5. Consolidate clusters (30-50% savings through utilization)

These compound. A client that does all five doesn't get a 70% reduction. They get a 85% reduction, because the savings stack.

But here's the honest caveat: the first 20% is easy. It's cleaning up idle resources and removing zombie instances. The next 30% requires architectural changes — moving to spot, implementing autoscaling, quantizing models. The last 20% requires organizational change — consolidating teams, enforcing quotas, changing how engineers think about cost.

Most teams stop at the easy 20%. The teams that go all the way save real money.

What I'd Do Differently

If I were starting an ML platform from scratch today, here's my stack:

• Compute: Spot-first with on-demand fallback, managed by Karpenter
• Serving: vLLM for LLM inference, ONNX Runtime for traditional models
• Scaling: KEDA with custom Prometheus metrics
• Storage: S3 with lifecycle policies, no EFS for checkpoints
• Monitoring: Cost attribution by team and project, weekly reviews

The comparative GPU study makes clear that there's no single best GPU provider — the right choice depends on your workload mix, and the gap between providers narrows when you account for spot pricing and regional variations. Don't lock yourself into one provider. Design for portability.

FAQ

Q: What's the fastest way to cut ML costs?
A: Find and kill idle GPU instances. The EaseCloud analysis found 35% of ML cloud spend goes to idle GPUs. A one-time cleanup typically saves 20-30% within a week.

Q: Spot instances vs on demand for ml training cost — which is better?
A: Spot, for most workloads, if you implement checkpointing and automatic retry. You'll save 60-90% on GPU compute. The Lyceum guide has a solid framework for deciding. Just don't use spot for jobs with hard completion deadlines.

Q: Is serverless inference cheaper than Kubernetes?
A: Sometimes, for low and spiky traffic. But once you exceed a certain volume, serverless becomes more expensive than a properly autoscaled Kubernetes deployment. The crossover point for most workloads is around 1-2 million requests per day.

Q: How much can quantization save?
A: 50-70% on inference cost with minimal quality loss. It's the highest ROI optimization I know of. The Spendark cost projections expect quantization to become the default practice by 2027.

Q: Should I use multiple cloud providers?
A: For cost arbitrage, it's rarely worth it. The operational overhead of managing multi-cloud exceeds the savings for most teams. Instead, pick one primary provider and design your infrastructure to be portable, so you can move if the pricing changes.

Q: How do I convince my team to consolidate clusters?
A: Show them the numbers. Run a utilization report for a week. The AWS guidance has templates for this. When teams see their own utilization is under 40%, they stop fighting consolidation.

Q: What's the biggest mistake teams make with ML cost optimization?
A: Optimizing training when inference dominates the bill. I've seen this repeatedly — teams spend months perfecting spot instance strategy for training while their inference spend balloons unchecked)Skip.

---

Here's my final position: cost-efficient ML deployment is not about finding the cheapest GPU. It's about building systems that don't waste the resources they already have. Spot instances, quantization, autoscaling, consolidation — these are all tools. The real skill is knowing which ones to apply and when.

And the best practices for cost efficient ml deployment change as your system grows. What works for a startup with one model doesn't work for an enterprise with fifty. Revisit your cost architecture quarterly. Measure. Adjust. Repeat.

The teams that do this — the ones that treat cost as an engineering problem rather than a finance problem — are the ones shipping models that actually make money. The rest are just renting GPUs and hoping.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.