NVIDIA shows how agent-assisted Spark rewrites avoid CPU fallback

NVIDIA reports reducing one Spark operator’s time from 274.55 to 12.78 hours on a 600-billion-row workload by rewriting its custom function for GPUs. Its agent workflow generates tests, converts the function, benchmarks it and iterates. Rewrites use SQL expressions, cuDF operations or CUDA to avoid CPU fallback. The reported result concerns one operator, not a guaranteed whole-pipeline speedup.

NVIDIA · Oct 13, 2026

Why it matters

Custom functions can erase GPU benefits by forcing row-wise CPU work and data conversions.

More context

Try the published cuDF skills on an isolated UDF with correctness tests. The workflow needs the cuDF Spark plugin and suitable GPU infrastructure; choose the simplest supported rewrite before custom CUDA.

Keep in mind

This is a vendor-reported production example, with results dependent on the function and workload.

Original read