What GPUs do you actually need for million token context?
The short answer: a lot. The longer answer involves understanding memory hierarchy, distributed scheduling, and why most cloud GPU setups waste 40% of your budget.
This guide covers the hardware math, the scheduling tricks on AWS, and the trade-offs. If you’re planning to deploy or train models with million-token contexts, you need to know where your GPU cycles actually go.
Why Million-Token Context Changes Everything
Context length has exploded. Gemini 1.5 Pro handles 2M tokens. Claude 3.5 Opus offers 1M. By mid-2026, most frontier models can digest entire codebases, financial filings, or multi-hour video transcripts.
But the GPU cost isn’t linear. Attention scales quadratically with sequence length. A 70B model processing 1M tokens generates a KV cache of roughly 2–4 TB in half precision. That’s beyond a single H100’s 80 GB. Way beyond.
So you need distributed systems. Not just for training — for inference too. And that’s where most teams get stuck. They think “just throw more GPUs at it.” That works for 128K. For 1M, it breaks in new, creative ways.
The Math: What a Million Tokens Actually Costs in GPU Memory
Let’s get concrete. A 70B parameter model in FP16 uses 140 GB just for weights. Add activations and optimizer states during training, and you’re looking at 400–800 GB per replica. For inference, the KV cache dominates.
KV cache math:
- 1M tokens × 80 layers × 2 (K and V) × hidden_dim (say 8192) × 2 bytes (FP16) = 2.6 TB
That’s per single sequence. Batch size 1. If you want to batch even 4 sequences, you need 10+ TB of HBM. An H100 SXM has 80 GB. A DGX H100 with 8 GPUs holds 640 GB total. So you need at least 4 DGX nodes just to hold the KV cache for a single inference request. Without batching.
Most people assume you can just use flash attention and everything’s fine. Flash attention reduces the compute cost, not the memory footprint. The KV cache still sits there.
Training is even worse. You need to store gradients and optimizer states. For a 70B model with 1M context, a single training step with FSDP sharding might require 12–16 H100 nodes (128–192 GPUs) to fit.
Distributed Training: Not Optional Anymore
If you’re training on million-token sequences, you can’t use simple data parallelism. The model won’t fit on one GPU. You need tensor parallelism, pipeline parallelism, and context parallelism (splitting the sequence across devices).
A common approach: combine tensor parallelism within each node (e.g., tp=8 for 8 GPUs) and FSDP across nodes, for example with the Amazon SageMaker distributed training library. The key is tuning the sharding strategy — not all sharding is equal for long sequences.
The Distributed Training & Large-Scale Systems article from Billion Hopes outlines the trade-offs between data, tensor, and pipeline parallelism for large models. I recommend reading it alongside the latest Cloud-native and Distributed Systems for Efficient and ... paper from arXiv — they benchmark context parallelism specifically.
How to Schedule GPU Jobs on AWS: Priority Scheduling Explained
Here’s where theory hits reality. You can design the perfect distributed setup. But if you can’t get the GPUs when you need them, it’s useless.
AWS offers several ways to run GPU jobs: SageMaker, EKS with Karpenter, Batch, or direct EC2. For million-token context workloads, you typically need large clusters (8–64 GPUs) for hours or days. Spot instances can save 60–70% but have interruption risk. Saving interruption for a 1M-context training run means losing hours of work.
AWS priority scheduling for GPU jobs explained in plain English: AWS reserves capacity based on instance type, region, and time. They use a “priority” system for Spot and On-Demand allocation. For large multi-node jobs, you need to either use Capacity Reservations (pay for guaranteed capacity) or use SageMaker’s managed training which handles scheduling internally.
One option is SageMaker’s warm pools with a priority queue for long-context training. You define a compute budget, set job priorities, and SageMaker preempts lower-priority jobs to free GPUs for high-priority ones. It’s expensive if you’re running 24/7, but for burst training of 1M-context models, it beats managing your own cluster.
To schedule GPU jobs on AWS efficiently, here’s a pattern:
# Example SageMaker training job submission with priority
import boto3
sm = boto3.client('sagemaker')
response = sm.create_training_job(
TrainingJobName='long-context-70b-1m',
AlgorithmSpecification={
'TrainingImage': '...',
'TrainingInputMode': 'File'
},
ResourceConfig={
'InstanceType': 'ml.p5.48xlarge',
'InstanceCount': 8,
'VolumeSizeInGB': 2000,
'KeepAlivePeriodInSeconds': 3600 # warm pool for 1 hour
},
StoppingCondition={
'MaxRuntimeInSeconds': 86400
},
# Priority: higher number = higher priority
# Use SageMaker Resource Limits to set priority range
)
The real trick is KeepAlivePeriodInSeconds. If your job finishes and you immediately submit another, the cluster stays warm. This avoids cold-start scheduling delays that can add 5–10 minutes for multi-node GPU clusters.
Inference at Scale: Serving Million-Token Contexts
Deploying long-context models is different from training. You can use quantization more aggressively because you’re not updating weights. FP8 is the sweet spot — on H100/H200 and the new Blackwell B200 (released Q1 2026), FP8 support is native.
Prefix caching helps massively. If your millions of tokens include repeated system prompts or document prefixes, cache them.
Here’s a vLLM configuration for 1M-token inference on AWS:
# vllm deployment config (Helm values for EKS)
model: /path/to/70b-fp8
tensor-parallel-size: 8
pipeline-parallel-size: 4
max-model-len: 1048576
gpu-memory-utilization: 0.95
kv-cache-dtype: fp8
enable-prefix-caching: true
swap-space: 128 # GB of CPU memory for KV cache offload
This config requires 4 nodes with 8 GPUs each (32 total). The swap-space parameter is critical — without it, 1M-token sequences would OOM on 32 GPUs.
Cost Realities: When It Makes Sense (and When It Doesn't)
Let’s talk money. A p5.48xlarge (8x H100) costs roughly $40/hour on-demand. For 32 GPUs (4 nodes), that’s $160/hour. A typical training run of a 70B model on 1M context might take 10–20 days. That’s $38,400 to $76,800 per run.
With spot instances and priority scheduling, you can cut that by 60% — but risk interruption. With SageMaker’s managed Spot and a checkpoint interval of 15 minutes, an interruption loses at most 15 minutes of compute.
Is it worth it? Depends on your use case. For codebase-level analysis (500K–1M tokens per query), batch inference with 32 GPUs can handle ~100 queries per hour, at a cost of ~$1.60 per query. That’s cheaper than paying a senior engineer to read 1M lines of code manually.
But for most chat applications, 128K context is plenty. Don’t buy GPUs for million-token unless you truly need million-token.
Million Token Context GPU Requirements: FAQ
Q: What’s the minimum number of GPUs to run inference on a 1M-context 70B model?
A: With FP8 quantization and KV cache offloading, you need at least 32 H100 GPUs. Without offloading, 64–128 depending on batching.
Q: Does Flash Attention help with memory for long sequences?
A: It reduces compute (O(n) vs O(n²) in practice), but the KV cache still takes the same memory. Flash Attention alone won’t let you fit 1M tokens on a single GPU.
Q: How does AWS priority scheduling for GPU jobs work exactly?
A: AWS uses a weighted scheduling algorithm across all users in a region. Your jobs get an internal priority based on your account Service Quotas, the instance type availability, and any Capacity Reservations you’ve purchased. Higher “priority” doesn’t mean you jump the queue for free — it means AWS allocates capacity proportionally. SageMaker adds its own job priority system on top.
Q: Can I use spot instances for million-context training?
A: Yes, but you need frequent checkpointing. The risk is that a spot interruption during a long sequence forward pass could lose the entire step, but with checkpoint every 100 steps, worst case is repeat 100 steps.
Q: What about Blackwell B200? How does it change the math?
A: B200 has 192 GB HBM total (dual GPU module), but each logical GPU has 96 GB. Better than H100’s 80 GB.
Q: Is tensor parallelism better than pipeline parallelism for long sequences?
A: Tensor parallelism is better for long sequences because it divides the compute per layer across GPUs, reducing the KV cache per GPU. Pipeline parallelism splits layers, which doesn’t help with KV cache memory. Use tensor parallelism within a node, pipeline across nodes if needed.
Q: How do I schedule GPU jobs on AWS with minimum cost?
A: Use SageMaker with Managed Spot, set KeepAlivePeriodInSeconds to reuse warm clusters, and use checkpoint-aware priority scheduling. For training, submit as many jobs as you can concurrently to fill capacity — idle GPUs are wasted money.
Q: What’s the hidden gotcha for distributed training at 1M context?
A: NCCL communication time dominates at long sequence lengths. The all-reduce on a 1M-tensor takes 2–3 seconds on InfiniBand. Your training step time grows linearly with sequence length. Don’t be surprised if your step time is 30+ seconds for 1M tokens.
Conclusion
Million token context GPU requirements are brutal. You need 32–128 GPUs for production workloads. The Agentic Systems Are Distributed Systems blog post from Akka nails it: reasoning over long contexts is inherently a distributed systems problem. You can’t cheat memory physics.
But the capability is real. The GPU cost is high, but the value of not having to read 1M tokens yourself is immense.
My advice: start with 128K context, prove the product works, then scale to million. Don’t build for million tokens from day one — the cost will kill you. When you do scale, use AWS priority scheduling to keep costs under control, and never trust a GPU benchmark that doesn’t include KV cache memory usage.
The future is long-context. But the present is expensive. Choose wisely.