HomeArticles

Chapter 19 of LLMs in production: choosing a model, speed and cost

FP8 vs FP16 Inference Cost Efficiency: A Practitioner's Buying Guide

FP8 versus FP16 for LLM inference: what each saves, when FP8 is a bad idea, quantization tooling, hardware support, and a framework to choose between them.

When to choose FP8, FP16 or a hybrid for LLM inference, by model size, GPU generation and workload

When a GPU bill grows faster than your model's accuracy gains, the obvious lever is quantization. But the choice between FP8 and FP16 isn't obvious at all.

Most people think FP8 is just "FP16 but cheaper." They're wrong. FP8 is a different beast with different failure modes, different hardware requirements, and — critically — a different cost curve that only pays off if your architecture is built for it.

This guide isn't a textbook comparison. It covers the numbers, the hidden costs, and how to decide which format your inference stack should use, plus the architectural moves that make FP8 worth the migration effort.


The Straight Numbers: What FP8 Actually Saves You

Let me hit you with the math first. On AWS, an p4d.24xlarge instance runs 8x A100 80GB GPUs at roughly $32.77/hour on-demand. An p5.48xlarge with 8x H100s costs around $98.32/hour. With FP16, you're fitting a 70B parameter model on a single node with tensor parallelism. Switch to FP8, and your memory footprint drops by half — same model, half the VRAM.

Here's the 2026 reality: NVIDIA's H100 and B200 both have native FP8 support, and AWS offers these on P5 and newer P6 instances. The B200 (available on AWS as of March 2026) takes FP8 further with FP8 tensor core throughput that's double FP16.


What FP8 Really Is (And Why It's Not Just Half of FP16)

FP8 uses 8 bits per number instead of 16. But it's not a simple truncation. FP8 comes in two flavors: E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits). E4M3 has better precision for values near 1.0, E5M2 has better dynamic range.

The problem: transformers have activation outliers — values that spike to 10x or 100x the median. Under FP8, those outliers get clipped or rounded to garbage. This isn't a theoretical concern.

FP16 doesn't have this problem. It uses 1 sign bit, 5 exponent bits, and 10 mantissa bits. It handles a range from roughly 6x10^-8 to 6.5x10^4. Plenty for transformer weights and activations.

So why bother with FP8? Because you can often recover that accuracy drop with:

  1. Activation-aware quantization — calibrating scale factors per-tensor or per-channel
  2. KV cache quantization — storing keys and values in FP8 while keeping weights in FP16
  3. Selective FP16 fallback — keeping outlier layers in FP16

Doing all three is more engineering work.


The AWS Cost Reality Check

When we talk about "fp8 vs fp16 inference cost efficiency," most people just compare GPU hours. That's like comparing cars by engine size alone. Real cost depends on your serving architecture.

The FP8 promise only pays off with the right serving architecture. FP8 serving requires:

  • A custom serving layer with dynamic batching
  • Quantization-aware fine-tuning (or loss-tolerant post-training quantization)
  • Careful monitoring for accuracy drift

Most teams skip the serving layer work and blame FP8. Don't be that team.


When FP8 Is a Bad Idea

Let me be direct. FP8 is not always the right choice. Here are cases where it fails:

Small models under 7B parameters. The accuracy loss from FP8 quantization is proportionally larger on smaller models. For a 3B model, FP8 can cost you 8-10% accuracy on reasoning tasks. That's often unacceptable.

Multi-modal models with vision encoders. Vision transformers are notoriously sensitive to quantization. Keeping the vision tower in FP16 avoids this, but it eats into the memory savings.

Real-time interactive services (latency < 100ms). FP8's compute advantage only shows up when you're batch-heavy. For single-stream inference, the overhead of FP8 dequantization sometimes negates the speedup.

When you're renting V100s or A100s. Wait, this is important. Older GPUs don't have native FP8 tensor cores. The A100 has FP16 tensor cores but not FP8. Running FP8 on A100 requires software emulation that's 2-3x slower than hardware FP16.

NVIDIA introduced FP8 tensor cores in the Hopper architecture (H100, H200). Anything older — A100, V100 — doesn't count.


How to Reduce AWS Inference Cost with Architecture (Beyond Just FP8)

FP8 is necessary but not sufficient. You need the full stack:

1. Prefill vs Decode Separation

The largest cost inefficiency in LLM inference is mixing prefill (processing input tokens) with decode (generating output tokens). These have different compute and memory profiles. One option is to split them onto different instance types:

Instance Type A: p5.48xlarge (H100) — prefill with FP8, high batch
Instance Type B: g5.48xlarge (A10G) — decode with FP16, low batch

The A10G (48GB) is dramatically cheaper than H100. But it's slow for prefill. By separating phases, prefill runs at high batch on H100s, and the KV cache streams to A10Gs for decode. The KV cache is FP8-compressed, so the transfer is small.

2. Dynamic Batching with PagedAttention

If you're not using continuous batching, you're wasting 50% of your GPU compute. But continuous batching + FP8 requires careful memory management. The vLLM project has FP8 kernel support. The key insight: FP8 reduces KV cache memory by half, which means you can keep 2x more concurrent requests in memory. This is where the throughput gains come from.

3. Autoscaling Based on Queue Depth, Not CPU

Most people set autoscaling on CPU utilization. That's wrong for LLM inference. Your GPU is the bottleneck. AWS's Application Auto Scaling doesn't natively track GPU utilization, so one option is a Lambda function that polls Amazon CloudWatch GPU metrics and adjusts target tracking policies.

Here's the CloudFormation snippet for a custom metric:

ScalingPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyName: !Sub "${Environment}-gpu-autoscale"
    PolicyType: TargetTrackingScaling
    ScalingTargetId: !Ref ScalingTarget
    TargetTrackingScalingPolicyConfiguration:
      TargetValue: 70
      CustomizedMetricSpecification:
        MetricName: GPUUtilization
        Namespace: Custom/LLM
        Dimensions:
          - Name: Model
            Value: Llama-70B-FP8
          - Name: Phase
            Value: Decode
        Statistic: Average
        Unit: Percent
      ScaleInCooldown: 180
      ScaleOutCooldown: 60

This lets the cluster scale down during off-peak hours.

4. Spot Instances for Prefill

This is contrarian, I know. But prefill workloads are idempotent and short-lived. If a spot instance gets reclaimated, you just re-run the prefill. AWS spot interruption rates for the p5.48xlarge have been under 2% per hour in us-east-1 (data from AWS Spot Instance Advisor).

This requires a checkpointing layer — if a spot node dies mid-prefill, you don't want to reprocess 10K tokens. One approach writes partial KV caches to FSx for Lustre during prefill.


The Quantization Tooling You'll Actually Use

Algorithm choice matters more than number format. The main options:

GPTQ-style Weight-Only Quantization

This keeps activations in FP16 but quantizes weights to FP8 (or even FP4). It works when your bottleneck is memory bandwidth, not compute. For small contexts (under 2K tokens), weight-only quantization gives up to 80% of the speedup of full FP8 with half the accuracy loss.

Full FP8 with Activation Quantization

This is what gives you the big compute gains. But activation quantization requires calibration data. You need to profile your production traffic, collect ~1000 representative inputs, and compute per-tensor scale factors.

A snippet using PyTorch's torchao:

import torch
from torchao.quantization import quantize_

# Load your model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B")

# Apply FP8 dynamic quantization (activations quantized on-the-fly)
model = quantize_(model, torchao.quantization.quant_api.fp8_dynamic_activation_quantize)

# Verify with calibration set
with torch.inference_mode():
    for batch in calibration_loader:
        outputs = model(**batch)
        collect_activation_stats(outputs)

# Save the quantized model
torch.save(model.state_dict(), "model_fp8.pth")

TensorRT-LLM is an alternative: it compiles the entire graph with kernel fusion.

KV Cache Quantization

This is the sleeper win. The KV cache grows linearly with sequence length and batch size. For long-context applications, the KV cache can consume 70% of GPU memory at high batch sizes.

Quantizing the KV cache to FP8 reduces memory by half with minimal accuracy impact — you're quantizing the cache, not the weights.

Selective Quantization

Some layers are more sensitive to quantization than others.

A common approach is to mix formats: FP8 for most layers, FP16 for the most sensitive ones. Frameworks like SGLang support per-layer precision.


Four quantization approaches: weight-only, full FP8 with activation scales, FP8 KV cache, and selective per-layer precision

Hardware Decisions: Where to Run FP8

Not all GPUs are created equal. AWS instance types compared:

H100 (P5 instances): Best FP8 support. Native FP8 tensor cores with no precision compromise.

B200 (P6 instances, available since March 2026 on AWS): FP8 throughput is 2x H100's FP8. But B200 instances cost ~$54/hour for p6.48xlarge.

A100 (P4 instances): No native FP8 support. Skip it. If you're on A100, stick with FP16. The software emulation costs more than you save.

Inferentia 2/3 (Inf2/Inf3 instances): AWS Inferentia now supports FP8 after a mid-2026 software update. Inferentia 3 is cheap — $2.19/hour for inf3.6xlarge — so it's interesting for high-volume, stable workloads. But the supporting ecosystem is still immature.

Also worth noting: if you're considering Groq, their LPU architecture natively operates in FP8.


The Decision Framework: FP8 vs FP16 vs Hybrid

Stop guessing. A decision framework:

Go FP8 from day one if:

  • Your model is 13B parameters or larger
  • You're running on H100, H200, B200, or Inferentia 3
  • Your workload is batch-heavy (more than 30 concurrent requests at peak)
  • You can tolerate up to 3% accuracy loss on generation tasks
  • You have engineering resources to handle calibration and monitoring

Stay FP16 if:

  • Your model is under 7B parameters
  • You're serving on A100 or A10G
  • Your latency targets are under 100ms with low batch
  • Accuracy on reasoning tasks is business-critical (e.g., medical or legal)
  • You don't have at least one engineer who understands numerical precision

Go hybrid if:

  • You serve both short and long contexts
  • Your traffic has unpredictable spikes (auto-scaling is challenging)
  • You're on mixed hardware (some H100, some A100)
  • You need to balance cost with risk tolerance

For hybrid, a simple rule: quantize everything except the embedding layer, the first 4 attention layers, and the LM head. Those go in FP16. With vLLM:

from vllm import LLM
from vllm.config import ModelConfig
from vllm.model_executor.layers.quantization import Fp8Config

# Hybrid config: FP8 for most layers, FP16 for sensitive ones
fp8_config = Fp8Config(
    ignored_layers=[
        "model.layers.0",
        "model.layers.1",
        "model.layers.2",
        "model.layers.3",
        "model.embed_tokens",
        "lm_head"
    ]
)

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    quantization="fp8",
    kv_cache_dtype="fp8",
    fp8_config=fp8_config,
    max_model_len=32768
)

The hybrid approach costs about 12% more memory than pure FP8 but eliminates almost all accuracy concerns on outlier-sensitive layers.


Hidden Costs Nobody Talks About

Real talk. The cost of FP8 isn't just GPU hours and accuracy. It's engineering time. The hidden costs:

Calibration runs: Collecting production traffic, fielding it through your model, and generating quantization scales.

Monitoring and alerting: FP8 introduces numerical drift over long sessions. One approach runs a "golden" FP16 model on a small fraction of traffic, compares its outputs with the FP8 version, and rolls back when they diverge. It costs compute and engineering attention.

Regression testing for every model update: Every time the model weights change, the full accuracy evaluation suite runs again.

In total, FP8 migration costs engineering weeks upfront and ongoing time every week. If your model is small enough that FP16 costs are tolerable, that engineering time might not be worth it.


How to Reduce AWS Inference Cost Using Architecture (The 5-Step Plan)

Since this is a buying guide, let me give you the buying plan. Whether you choose FP8 or FP16, these architecture decisions matter more:

Step 1: Profile your actual traffic patterns.

Grab a week of production logs. Use a BI tool or simple Python script to find out: average request rate, peak concurrency, context lengths, arrival patterns. No inference cost discussion is valid without this.

Step 2: Start with FP16 and dynamic batching.

Get your serving layer optimized before touching quantization. If your serving layer is naive — static batching, no request coalescing — you'll see 2-3x cost reduction just from adding vLLM or TRON serving.

Step 3: Measure cost per token under FP16.

Set a baseline. Use tags in AWS Cost Explorer to label instance types by model and format. Then you can see where your money goes weekly.

Step 4: Add FP8 as a second environment.

Don't kill FP16. Run FP8 side-by-side on different instances for 2 weeks. Compare latency, accuracy, and cost. Use the CloudWatch code I showed earlier for autoscaling both.

Step 5: Go hybrid based on region and traffic.

For example: FP16 where compliance-heavy workloads need strict accuracy SLAs, and FP8 where you can tune accuracy and monitor drift.


The FP8 Alternatives That Fly Under the Radar

If you've read this far and still feel uncomfortable with FP8, let me mention two other levers that cost the same engineering effort:

INT8 Quantization: Older but battle-tested. ONNX Runtime and TensorRT both have mature INT8 kernels. The precision is lower than FP8 (INT8 has no exponent bits), but for pure throughput on dense layers, INT8 kernels are often faster than FP8 because they've been optimized longer.

Model distillation: Instead of quantizing a huge model, train a smaller FP16 model on outputs of the huge model. For classification and extraction tasks, a distilled 7B model can match a 70B teacher at 1/10th the inference cost. This isn't possible for open-ended generation, but if your workload is structured outputs, distillation is the biggest lever of all.

Speculative decoding: This is a serving trick, not a quantization trick. Run a small "draft" model (FP16 or FP8) that predicts tokens, and have the large model verify them in parallel. The draft model can be 1/10th the size and the accuracy loss is zero — the larger model always makes the final decision.


FAQ

Is FP8 always half the memory cost of FP16?

Theoretically yes — 8 bits vs 16 bits. In practice, no. Frameworks often store scale factors per tensor, which adds overhead. You'll typically see 40-50% memory reduction, not exactly half. Also, FP8 inference often requires activations in higher precision for attention operations, so peak memory may exceed your expectations.

What is the difference between FP8 and INT8?

FP8 has 3-4 exponent bits, which gives it a wide dynamic range. INT8 is stored as integers with no implicit exponent, so values are typically scaled to a fixed range. FP8 is better suited to transformers because it can naturally handle weight distributions that vary by orders of magnitude. INT8 works fine for CNN-style models but is riskier for attention-based architectures.

Can I run FP8 on older AWS GPUs like A100 or V100?

No. FP8 requires hardware tensor cores that support the format. NVIDIA introduced this in the Hopper architecture (H100, released 2022). The A100 uses Ampere, which has FP16 tensor cores but not FP8. Running FP8 on A100 means software emulation in PyTorch, which is typically slower than native FP16. Don't use FP8 on A100.

Does FP8 reduce accuracy in production LLMs?

It can, but the impact depends on your workload. For chat and generation, 1-3% metric degradation is typical. For mathematical reasoning, coding, or data extraction, losses can be 5-10% if you don't calibrate properly. The mitigation is activation-aware quantization with calibration data drawn from your production traffic.

How does FP8 affect auto-scaling on AWS?

FP8 lowers per-token cost, which directly improves your auto-scaling economics. You'll need fewer instances to handle the same peak load. But FP8 also changes the relationship between concurrency and memory — you can hold more requests per GPU, which means autoscaling triggers need to account for the new memory ceiling.

Which serving frameworks have the best FP8 support in 2026?

TensorRT-LLM has the most mature FP8 kernels. vLLM works well if you use their quantization-aware fork. SGLang has good support but older versions had memory leaks with FP8 KV cache.

Is FP8 worth it for models under 7B parameters?

Usually no. The cost savings are proportionally smaller (smaller memory footprint in absolute terms), and the accuracy risk is proportionally higher. For small models, distillation, speculative decoding, and dynamic batching are bigger cost levers.


Final Verdict

FP8 vs FP16 total cost: FP8 wins on compute (25-45% cheaper per token); FP16 wins on engineering and risk cost.

FP8 is not the default choice. It's the informed choice.

A reasonable split: FP8 for large models, FP16 for anything under 13B and for customer-facing analytics where data integrity matters more than speed.

The "fp8 vs fp16 inference cost efficiency" question really comes down to the total cost of ownership across three dimensions:

  1. Compute cost: FP8 wins (typically 25-45% cheaper per token)
  2. Engineering cost: FP16 wins (no calibration, no monitoring)
  3. Risk cost: FP16 wins (predictable accuracy, no silent failures)

If you're running one model in a stable workload with no compliance requirements, FP8 is a no-brainer. If you're iterating weekly on model changes for a high-stakes application, FP16 is the safer bet — and the engineering time you save can fund a bigger model.

The best architecture starts with FP16, profiles its bottlenecks, and migrates specific components to FP8 where the cost data justifies it. Watch the numbers and move one workload at a time.

And remember: how to reduce AWS inference cost with architecture isn't a one-time decision. It's a process. Your traffic evolves. New instance types arrive. Frameworks get better kernels. Re-evaluate your format choice every quarter. The optimal answer changes faster than you think.

Want to talk this through?

Sivaro AI →