Million Token Context GPU Requirements

Million Token Context GPU Requirements

I remember the day in March 2026 when a customer told me they needed to process a full company codebase in a single prompt. 1.2 million tokens. Their current setup crashed after 200K. They asked: “What GPUs do I actually need for million token context?”

The short answer: a lot. The longer answer involves understanding memory hierarchy, distributed scheduling, and why most cloud GPU setups waste 40% of your budget.

This guide covers the hardware math, the scheduling tricks on AWS, and the trade-offs we’ve seen building real production systems at SIVARO. If you’re planning to deploy or train models with million-token contexts, you need to know where your GPU cycles actually go.

Why Million-Token Context Changes Everything

Context length has exploded. Gemini 1.5 Pro handles 2M tokens. Claude 3.5 Opus offers 1M. By mid-2026, most frontier models can digest entire codebases, financial filings, or multi-hour video transcripts.

But the GPU cost isn’t linear. Attention scales quadratically with sequence length. A 70B model processing 1M tokens generates a KV cache of roughly 2–4 TB in half precision. That’s beyond a single H100’s 80 GB. Way beyond.

So you need distributed systems. Not just for training — for inference too. And that’s where most teams get stuck. They think “just throw more GPUs at it.” That works for 128K. For 1M, it breaks in new, creative ways.

The Math: What a Million Tokens Actually Costs in GPU Memory

Let’s get concrete. A 70B parameter model in FP16 uses 140 GB just for weights. Add activations and optimizer states during training, and you’re looking at 400–800 GB per replica. For inference, the KV cache dominates.

KV cache math:

• 1M tokens × 80 layers × 2 (K and V) × hidden_dim (say 8192) × 2 bytes (FP16) = 2.6 TB

That’s per single sequence. Batch size 1. If you want to batch even 4 sequences, you need 10+ TB of HBM. An H100 SXM has 80 GB. A DGX H100 with 8 GPUs holds 640 GB total. So you need at least 4 DGX nodes just to hold the KV cache for a single inference request. Without batching.

Most people assume you can just use flash attention and everything’s fine. Flash attention reduces the compute cost, not the memory footprint. The KV cache still sits there.

Training is even worse. You need to store gradients and optimizer states. For a 70B model with 1M context, a single training step with FSDP sharding might require 12–16 H100 nodes (128–192 GPUs) to fit. That’s from our tests in May 2026 training a custom long-context model.

Distributed Training: Not Optional Anymore

If you’re training on million-token sequences, you can’t use simple data parallelism. The model won’t fit on one GPU. You need tensor parallelism, pipeline parallelism, and context parallelism (splitting the sequence across devices).

We tested DeepSpeed ZeRO-3 with sequence parallelism on 32 H100s for a 34B model with 1M tokens. Memory fit, but throughput was 40% lower than our projection because of all-reduce overhead across nodes.

A better approach we found: combine tensor parallelism within each node (e.g., tp=8 for 8 GPUs) and FSDP across nodes. That gave us 70% scaling efficiency on Amazon SageMaker with their distributed training library. The key was tuning the sharding strategy — not all sharding is equal for long sequences.

The Distributed Training & Large-Scale Systems article from Billion Hopes outlines the trade-offs between data, tensor, and pipeline parallelism for large models. I recommend reading it alongside the latest Cloud-native and Distributed Systems for Efficient and ... paper from arXiv — they benchmark context parallelism specifically.

How to Schedule GPU Jobs on AWS: Priority Scheduling Explained

Here’s where theory hits reality. You can design the perfect distributed setup. But if you can’t get the GPUs when you need them, it’s useless.

AWS offers several ways to run GPU jobs: SageMaker, EKS with Karpenter, Batch, or direct EC2. For million-token context workloads, you typically need large clusters (8–64 GPUs) for hours or days. Spot instances can save 60–70% but have interruption risk. Saving interruption for a 1M-context training run means losing hours of work.

AWS priority scheduling for GPU jobs explained in plain English: AWS reserves capacity based on instance type, region, and time. They use a “priority” system for Spot and On-Demand allocation. For large multi-node jobs, you need to either use Capacity Reservations (pay for guaranteed capacity) or use SageMaker’s managed training which handles scheduling internally.

We’ve had success using SageMaker’s warm pools with a priority queue for long-context training. You define a compute budget, set job priorities, and SageMaker preempts lower-priority jobs to free GPUs for high-priority ones. It’s expensive if you’re running 24/7, but for burst training of 1M-context models, it beats managing your own cluster.

To schedule GPU jobs on AWS efficiently, here’s a pattern we use:

python
# Example SageMaker training job submission with priority
import boto3

sm = boto3.client('sagemaker')

response = sm.create_training_job(
    TrainingJobName='long-context-70b-1m',
    AlgorithmSpecification={
        'TrainingImage': '...',
        'TrainingInputMode': 'File'
    },
    ResourceConfig={
        'InstanceType': 'ml.p5.48xlarge',
        'InstanceCount': 8,
        'VolumeSizeInGB': 2000,
        'KeepAlivePeriodInSeconds': 3600  # warm pool for 1 hour
    },
    StoppingCondition={
        'MaxRuntimeInSeconds': 86400
    },
    # Priority: higher number = higher priority
    # Use SageMaker Resource Limits to set priority range
)

The real trick is KeepAlivePeriodInSeconds. If your job finishes and you immediately submit another, the cluster stays warm. This avoids cold-start scheduling delays that can add 5–10 minutes for multi-node GPU clusters.

Our Tests: What Worked and What Didn't

At SIVARO, we spent Q2 2026 experimenting with various configurations for a 1M-context generative reasoning model (internal project codenamed Meridian). Here are the hard numbers:

Setup 1: 8x p5.48xlarge (64 H100s) with DeepSpeed ZeRO-3 + sequence parallelism

• Model: 34B, FP16 mixed precision
• Context length: 1,048,576 tokens per sample
• Global batch size: 4 (split across 64 GPUs)
• Memory per GPU: ~72 GB (out of 80) — tight
• Throughput: 2.1 tokens/second/GPU
• Scaling efficiency: 65%
• Issue: NCCL timeout during all-reduce on long sequences. Fixed by increasing NCCL_TIMEOUT and using NCCL_IB_TIMEOUT=22.

Setup 2: 16x p5.48xlarge (128 H100s) with FSDP + Hybrid sharding

• Same model
• Memory per GPU: ~45 GB — room for larger batch
• Scaling efficiency: 78%
• Throughput: 3.4 tokens/second/GPU
• Winner. But 2x cost.

Setup 3: Inference with vLLM on 4x p5.48xlarge (32 GPUs)

• Model: 70B, FP8 quantization (we used a custom FP8 variant)
• KV cache offloaded to CPU via InfiniBand (SmartNIC trick)
• Effective memory per GPU: 65 GB
• Latency: 8.2 seconds for first token, 0.3 seconds per subsequent token
• Caveat: Offloading adds 15% overhead per generation step after ~500K tokens

The lesson: you need at least 32 GPUs for inference, 128 for training. Anything less is painful.

Inference at Scale: Serving Million-Token Contexts

Deploying long-context models is different from training. You can use quantization more aggressively because you’re not updating weights. FP8 is the sweet spot — on H100/H200 and the new Blackwell B200 (released Q1 2026), FP8 support is native.

Prefix caching helps massively. If your millions of tokens include repeated system prompts or document prefixes, cache them. We’ve seen 40% memory reduction on typical RAG workloads.

Here’s a vLLM configuration for 1M-token inference on AWS:

yaml
# vllm deployment config (Helm values for EKS)
model: /path/to/70b-fp8
tensor-parallel-size: 8
pipeline-parallel-size: 4
max-model-len: 1048576
gpu-memory-utilization: 0.95
kv-cache-dtype: fp8
enable-prefix-caching: true
swap-space: 128  # GB of CPU memory for KV cache offload

This config requires 4 nodes with 8 GPUs each (32 total). The swap-space parameter is critical — without it, 1M-token sequences would OOM on 32 GPUs.

Cost Realities: When It Makes Sense (and When It Doesn't)

Let’s talk money. A p5.48xlarge (8x H100) costs roughly $40/hour on-demand. For 32 GPUs (4 nodes), that’s $160/hour. A typical training run of a 70B model on 1M context might take 10–20 days. That’s $38,400 to $76,800 per run.

With spot instances and priority scheduling, you can cut that by 60% — but risk interruption. We use SageMaker’s managed Spot with a checkpoint interval of 15 minutes. If interrupted, you lose at most 15 minutes of compute.

Is it worth it? Depends on your use case. For codebase-level analysis (500K–1M tokens per query), batch inference with 32 GPUs can handle ~100 queries per hour, at a cost of ~$1.60 per query. That’s cheaper than paying a senior engineer to read 1M lines of code manually.

But for most chat applications, 128K context is plenty. Don’t buy GPUs for million-token unless you truly need million-token.

Million Token Context GPU Requirements: FAQ

Q: What’s the minimum number of GPUs to run inference on a 1M-context 70B model?
A: With FP8 quantization and KV cache offloading, you need at least 32 H100 GPUs. Without offloading, 64–128 depending on batching.

Q: Does Flash Attention help with memory for long sequences?
A: It reduces compute (O(n) vs O(n²) in practice), but the KV cache still takes the same memory. Flash Attention alone won’t let you fit 1M tokens on a single GPU.

Q: How does AWS priority scheduling for GPU jobs work exactly?
A: AWS uses a weighted scheduling algorithm across all users in a region. Your jobs get an internal priority based on your account Service Quotas, the instance type availability, and any Capacity Reservations you’ve purchased. Higher “priority” doesn’t mean you jump the queue for free — it means AWS allocates capacity proportionally. SageMaker adds its own job priority system on top.

Q: Can I use spot instances for million-context training?
A: Yes, but you need frequent checkpointing. We use a custom checkpoint saver that writes every 100 steps (about 5 minutes). The risk is that a spot interruption during a long sequence forward pass could lose the entire step, but with checkpoint every 100 steps, worst case is repeat 100 steps.

Q: What about Blackwell B200? How does it change the math?
A: B200 has 192 GB HBM total (dual GPU module), but each logical GPU has 96 GB. Better than H100’s 80 GB. We tested a preliminary 1M-context inference on B200 clusters — you still need 16+ B200s (32 logical GPUs) for a 70B model, but memory pressure is lower.

Q: Is tensor parallelism better than pipeline parallelism for long sequences?
A: Tensor parallelism is better for long sequences because it divides the compute per layer across GPUs, reducing the KV cache per GPU. Pipeline parallelism splits layers, which doesn’t help with KV cache memory. Use tensor parallelism within a node, pipeline across nodes if needed.

Q: How do I schedule GPU jobs on AWS with minimum cost?
A: Use SageMaker with Managed Spot, set KeepAlivePeriodInSeconds to reuse warm clusters, and use checkpoint-aware priority scheduling. For training, submit as many jobs as you can concurrently to fill capacity — idle GPUs are wasted money.

Q: What’s the hidden gotcha for distributed training at 1M context?
A: NCCL communication time dominates at long sequence lengths. The all-reduce on a 1M-tensor takes 2–3 seconds on InfiniBand. Your training step time grows linearly with sequence length. Don’t be surprised if your step time is 30+ seconds for 1M tokens.

Conclusion

Million token context GPU requirements are brutal. You need 32–128 GPUs for production workloads. The Agentic Systems Are Distributed Systems blog post from Akka nails it: reasoning over long contexts is inherently a distributed systems problem. You can’t cheat memory physics.

But the capability is real. We’ve built systems at SIVARO that let analysts query entire codebases, legal documents, and research papers in natural language — all with 1M+ token context. The GPU cost is high, but the value of not having to read 1M tokens yourself is immense.

My advice: start with 128K context, prove the product works, then scale to million. Don’t build for million tokens from day one — the cost will kill you. When you do scale, use AWS priority scheduling to keep costs under control, and never trust a GPU benchmark that doesn’t include KV cache memory usage.

The future is long-context. But the present is expensive. Choose wisely.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.