Strata spreads local model inference across GPU and system memory

Strata runs quantized Qwen3.8-Flash-Next models by sharing inference across a consumer GPU, system memory and the CPU. Frequently used model experts stay on the GPU while other work runs on the CPU. It exposes a local chat interface and compatible API endpoints for coding tools, making the hardware and memory tradeoffs part of the setup.

Niko1221 / Strata

Why it matters

Engineers considering local inference can assess a concrete setup rather than rely on a headline token rate.

More context

Try only after checking the documented Windows or Linux requirements: a supported GPU with at least 12 GB VRAM, 32 GB RAM and roughly 80 GB free disk. Model variants need different memory budgets. Publication date is unverified.

Keep in mind

This is an actively evolving community project. Quantization and pruned variants trade quality for resources, and initial loading can stall the machine. Reported speeds are author measurements; 1min has not installed or benchmarked it.

Original read