Jane Street reduces message filtering with targeted indexes

Jane Street’s Aria message bus was scanning too much data when subscribers needed only selected topics. Engineers added indexes for topic partitions and merged the matching positions in message order. For older disk-backed data, they developed an adaptive splitting algorithm that separates busy topics while grouping smaller ones, reducing the irrelevant data each reader must process.

Jane Street · Oct 5, 2026

Why it matters

The account shows how workload-aware indexing can address overload without requiring users to design topics around storage internals.

More context

Read the benchmark methodology, pooled index storage and randomized consistency tests. Tip indexing has reached production; adaptive splitting is still in staging in the report. Try measuring bytes scanned versus bytes delivered in your own stream recovery path before changing its layout.

Keep in mind

Aria is an internal system, not a newly released message broker. The detailed results describe preliminary staging measurements on real scenarios; do not read the headline CPU reduction as a universal production guarantee. We did not reproduce those measurements.

Original read