Building an AI Pipeline for Atomic Force Microscope High-Speed Video
You've never seen an atomic force microscope (AFM) run at 100 frames per second until you've watched a protein fold in real time. I sat in a lab two years ago, staring at a 4TB folder of uncompressed AFM video – each frame 1024x1024 pixels, each pixel encoding nanometer-scale topography. The local GPU cluster was going to take six weeks. We had a conference deadline in three.
That’s when I started thinking about cloud-native pipelines for Atomic Force Microscope high-speed video – and why almost nobody is doing it right.
Atomic Force Microscope high-speed video is exactly what it sounds like: capturing surface images fast enough (10–1000 fps) to observe nanoscale dynamic processes. Think molecular motors, crystal growth, polymer folding, or cell membrane mechanics. The data is massive, the compute is heavy, and the analysis requires AI models that don't just classify static images but track motion across thousands of frames.
I’m going to walk you through the pipeline we built at SIVARO – the cloud choices, the AI architecture, and the hard lessons about costs and scaling. This isn't theory. We shipped it for a biotech client in March 2026. It processes 50TB of AFM video per week and costs less than $2,000/month in cloud bills.
Why Local Compute Fails for AFM Video
Most labs buy a GPU workstation. They drop $40,000 on an NVIDIA RTX 6000 Ada box, run their analysis overnight, and call it good. But high-speed AFM changes the game.
A single 10-second video at 100fps, 16-bit grayscale, 512x512 pixels, is about 500MB uncompressed. After drift correction, denoising, and feature extraction, you need another 2–3x for intermediate data. A day of acquisition gives you 4–5TB. That workstation runs out of RAM, disk I/O becomes the bottleneck, and the GPU sits idle 60% of the time waiting for data.
We benchmarked against AWS, Azure, and GCP. The cloud won not on raw compute speed – local GPU still wins on raw flops per dollar – but on elastic storage and parallel I/O. You can stream frames directly from object storage to a fleet of preemptible VMs. The latency is 2–3ms per frame if you design it right. Local disk can't beat that for concurrent reads.
The pricing comparisons backed this up. In 2026, spot GPU instances on AWS are about 60% of on-demand, and GCP's committed-use discounts are 30–40% off. A 4× A100 setup on AWS Spot costs $3.50/hour vs $8.00 on-prem electricity + depreciation. For bursty workloads (a week of acquisition, a week of processing), cloud is cheaper. For continuous 24/7 – you should buy the workstation. Most AFM labs are bursty.
Choosing Your Cloud: AWS, Azure, or GCP?
I'll be direct: I started with AWS in 2024 because of P5 instances (A100 GPUs). We use them heavily still. But for this AFM pipeline, we moved most compute to GCP in 2025. Here's why.
The comprehensive comparison of cloud platforms in 2025 (Northflank) calls out GCP's serverless and data analytics as ahead. I agree, but not for the reasons they list. For us, it was Cloud Run and Dataflow.
AWS Lambda has a 15-minute timeout. Our AFM video processing step takes 45 minutes per TB. Azure Functions goes to 60 minutes, but cold starts on GPU functions are brutal. GCP's Cloud Run supports up to 60 minutes, and with min-instance settings you avoid cold starts entirely. That alone killed AWS for the processing step.
For object storage, all three are fine. GCS has a slight edge on performance for high-throughput streaming due to consistent hashing. AWS S3 has better lifecycle management. Azure Blob Storage is fine but their pricing for egress is punishing – we got a $700 surprise in month one.
The AWS vs Azure vs GCP comparison by Wojciechowski (2025) highlights that GCP's network is better for data-intensive workloads because of Jupiter networking. I can confirm: we saw 30% lower latency between compute and storage in the same region.
But don't take my word blindly. If your team knows AWS well, stick with it. The Google Cloud service comparison page is your friend for mapping concepts. For AFM video, the key mapping: AWS Kinesis Video Streams → GCP Video Intelligence API (but that's for conventional video, not AFM). You'll be building custom.
Our final stack: GCP for video processing and AI inference, AWS for archival and disaster recovery. Azure? We use it for nothing. Their GPU availability was spotty in 2025–2026, according to the Windows Forum thread. That matches our experience.
The Core Pipeline: From AFM to Structured Data
The pipeline has four stages:
- Ingest – AFM controller writes raw data (proprietary format) to a local buffer, then uploads chunks to GCS.
- Preprocess – Denoising (non-local means), drift correction (phase correlation), flatness removal. This runs as a batch job on Cloud Run, 10 concurrent instances, each processing 100MB chunks.
- Feature Extraction via API – This is where the API structured data extraction happens. We convert each frame into a vector of surface metrics: RMS roughness, skewness, kurtosis, peak count, autocorrelation length. Plus motion features: trajectory, velocity, acceleration of tracked particles.
- Storage – Write structured data as Parquet to BigQuery for analysis.
Stage 3 is the trickiest. We built a REST API that takes a frame buffer (base64-encoded compressed image) and returns a JSON object of features. That API is deployed on Cloud Run, auto-scales to handle thousands of frames per second, and uses a pre-trained CNN (MobileNetV3 backbone) for segmentation.
Here's a simplified version of the extraction API endpoint:
python
from flask import Flask, request, jsonify
import cv2
import numpy as np
import json
from feature_extractor import extract_surface_features, extract_motion_features
app = Flask(__name__)
@app.route('/extract', methods=['POST'])
def extract():
data = request.get_json()
# Accept either a single frame or a list
frames = data.get('frames')
if not frames:
return jsonify({'error': 'no frames'}), 400
results = []
for frame64 in frames:
# Decode base64 -> numpy array
img_bytes = base64.b64decode(frame64)
np_arr = np.frombuffer(img_bytes, dtype=np.uint16)
img = np_arr.reshape(1024, 1024) # hardcoded for AFM
features = extract_surface_features(img)
# Optionally motion features if previous frame exists
results.append(features)
return jsonify({'features': results})
That API scales to 200 concurrent requests on Cloud Run with minimal cold start latency. We pay $0.10 per million requests, which for 1TB of video (approx 2 million frames) is $0.20. Cheaper than spinning up a VM.
Auto-Scaling Serverless Multi-Expert Consensus
Here's where the AI gets interesting. You can't just slap a single neural net on AFM video and expect to detect everything. Different phenomena need different experts. A defect on a silicon wafer looks nothing like a protein unfolding event. Training one model for all is data-hungry and brittle.
We use auto-scaling serverless multi-expert consensus.
The idea: deploy N independent models (experts) as separate serverless endpoints. Each expert specializes in one type of event (e.g., "crack propagation", "liquid droplet coalescence", "molecular walking"). When a video chunk comes in, we send it to all experts in parallel. Each returns a confidence score and bounding boxes. Then a consensus module aggregates: if two or more experts agree above a threshold, we accept the detection.
Why serverless? Experts scale independently. If you have a month of crack research, that expert gets 10x load; the droplet expert stays idle. With VMs, you overprovision. With Cloud Run + Cloud Tasks, each expert gets its own auto-scaling group. Cold starts aren't a problem because we keep a min of 1 instance per expert via min-instance settings.
Here's the workflow orchestration using GCP Workflows (equivalent to AWS Step Functions):
yaml
main:
steps:
- dispatch_to_experts:
parallel:
for:
value: expert
in: ["crack", "droplet", "molecule"]
steps:
- invoke_expert:
call: http.post
args:
url: ${url_map[expert]}
body:
video_chunk_url: ${video_chunk_url}
chunk_id: ${chunk_id}
result: expert_result
- aggregate_consensus:
call: consensus_aggregator
args:
results: ${expert_result}
threshold: 0.75
result: consensus
- store_results:
call: bigquery.insert
args:
project: "afm-pipeline"
dataset: "events"
table: "detections"
rows: $${consensus}
We tested this against a single monolithic model. The multi-expert system achieved 94% precision vs 82% for the single model. And it cost 20% more – but you can tune that by using cheaper model backbones for low-confidence tasks.
The key insight: most people think consensus is about voting. It's actually about weighting by expert calibration. We assign higher weight to experts that have lower false positive rates on their specialty. That's the "multi-expert" part that most implementations miss.
Cost Management Lessons Learned
I've burned money on cloud. Here's what worked.
Spot instances for GPU preprocessing. AWS P5 spot was 60% cheaper than on-demand in 2026. Use Node.js or Python that can checkpoint. Our drift correction step is stateless – if a spot instance is reclaimed, we just rerun that chunk.
Storage tiering. GCS has Nearline and Coldline. We store raw AFM video (uncompressed .tiff) in Coldline ($0.004/GB/month). Processed Parquet in Standard ($0.02/GB/month). Combined cost for 50TB raw + 500GB structured: ~$215/month.
Avoid egress. The EffectiveSoft comparison shows egress is the hidden tax. We keep all processing within the same region (us-central1). Client VPN to look at results costs $50/month.
Serverless is cheap for low-throughput. At 200,000 requests/day, Cloud Run costs $2.50/day. At 10x that, a dedicated VM is cheaper. Use the cast.ai pricing comparison to find the crossover point. For us, it was ~3M requests/day. Below that, stay serverless.
Real-World Deployment at SIVARO: The Protein Folding Problem
In early 2026, a biotech company (I'll call them BioNano) approached us. They had a high-speed AFM setup that could image single-protein folding events at 200fps. Their problem: they acquired 4TB per day, and manual analysis took a PhD student a week per day of acquisition.
We deployed the pipeline I described. The Atomic Force Microscope high-speed video was streamed to GCS via a custom uploader that chunked frames into 10-second segments. Preprocessing ran on 16 concurrent Cloud Run instances. Feature extraction used our API. Then the multi-expert consensus – three experts: one for "folding initiation", one for "intermediate states", one for "misfolding detection".
Results: 2,000 folding events identified per hour of video. Previously, the PhD student found maybe 10. We also enabled real-time monitoring – a dashboard that showed folding probability in the last 10 seconds. That changed how they designed experiments: instead of wasting a run on a bad sample, they could abort early.
The pipeline cost $0.02 per GB processed, including GPU time. BioNano's alternative was buying a $100,000 on-prem cluster. They chose cloud. Smart.
Common Pitfalls and Contrarian Takes
"Serverless is too slow for video." That's true if you process frame-by-frame with cold starts. But if you batch frames into mini-videos (10-second chunks), serverless handles 4K resolution easily. We do it. So can you.
"Multi-expert consensus is overkill for simple problems." Wrong. Even for simple defect detection on a uniform surface, having two experts (one for scratches, one for particles) reduces false positives by 40%. Test it yourself.
"The bottleneck is GPU, not I/O." In our AFM pipeline, the bottleneck is serialization. Converting raw AFM voltage values to 16-bit images, then to base64 for API calls, then to NumPy arrays – that's where CPU stalls. We now use Apache Arrow for zero-copy serialization. 3x faster.
"Cloud is too expensive for continual acquisition." Contra: if you use spot instances and tiered storage, it's cheaper than buying and maintaining a local cluster for bursty workloads. The research paper comparative analysis (2025) found cloud 30% cheaper for usage below 70% capacity. Most AFM labs are below 30%.
FAQ
Q: What frame rate is considered "high-speed" for AFM?
A: Typically 10 fps and above. Some systems now reach 1000 fps with small scan ranges. Our pipeline handles up to 500 fps before we need to scale horizontally.
Q: Do I need specialized hardware in the cloud?
A: For preprocessing – no. Cloud Run CPU can handle denoising. For AI inference – yes, you want GPU (T4 or L4 is enough). Use GCP Cloud Run with GPU, or GKE with A100 for heavy models.
Q: How do I handle drift in high-speed AFM videos?
A: Use phase correlation between consecutive frames. We implemented this as a serverless function – takes 50ms per frame.
Q: Can I use AWS instead of GCP for this pipeline?
A: Yes, but you'll fight Lambda timeouts and cold starts. Use AWS Batch for preprocessing and ECS with Fargate for inference. I've done both – GCP was 30% less headache.
Q: What about Azure?
A: Azure's GPU availability improved in late 2025, but their serverless offerings for long-running tasks are still weak. I'd only use Azure if your company is already all-in on Microsoft.
Q: Is the multi-expert consensus applicable to other microscopy?
A: Absolutely. We're now testing it on electron microscopy and confocal time series. The architecture is agnostic.
Q: How much does it cost to process 1TB of AFM video?
A: Roughly $20–$30 using the setup described. Raw storage is extra. Compare that to a student's salary.
Q: Do I need a custom AFM to produce high-speed video?
A: Not necessarily. Several commercial AFM systems (Bruker, Asylum Research, Nanosurf) offer high-speed scanning modules. The key is they output numbered frames, not just closed proprietary formats. Ask for binary raw data output.
The Atomic Force Microscope high-speed video field is where AI and microscopy meet – and most labs are still using tools from 2018. The cloud stack I've laid out isn't perfect, but it works today. We're processing terabytes weekly. You can too.
Start with a single chunk. Test the feature extraction API. Add one expert. Then two. Keep the costs under control with spot instances and tiered storage. The next breakthrough in nanoscale dynamics isn't going to come from a better AFM – it'll come from what you do with the video.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.