feat: add -ngl to cap diffusion layers resident in VRAM

[mosaic-devel]

Problem

Residency is currently expressed only as a byte budget (--max-vram). That works, but it is awkward when the question is simply "how much of this model fits on my card", and there is no equivalent to the layer count users of similar projects already reach for.

Solution

-ngl / --gpu-layers caps how many diffusion layers stay resident in VRAM; the remainder stream from RAM per step.

sd-cli -m model.safetensors -p "a cat" -ngl 20

This reuses the existing graph-cut segmentation and streaming machinery. The substantive change is that annotate_residency can now be driven by a count as well as by bytes; segment merging is disabled when a count is given, so one segment corresponds to one layer.

The count is an upper bound, not a demand. Residency is also capped by free VRAM, reusing the same safety margin and worst-streamed-segment reserve the byte path already applies. Asking for more layers than fit fills the device and streams the rest, rather than failing part-way through a denoise pass — so -ngl 999 is a reasonable way to say "use as much VRAM as there is". This differs deliberately from llama.cpp, where the number is absolute and over-asking OOMs.

Since streaming reads weights out of host memory, -ngl implies --offload-to-cpu and --stream-layers. With no count given (-1, the default), byte-budget behaviour is unchanged — remaining stays negative and never gates the loop.

Verification

Vulkan (RX 6600 XT, 8 GB, RADV), Krea 2 Turbo, 29 segments:

-ngl 8 — honoured exactly:

RESIDENT=8  STREAMED=21

-ngl 999 — clamped to fit instead of OOMing:

krea2 streaming budget = 4204.47 MB     (free VRAM - 512MB margin)
RESIDENT=7  STREAMED=22

(The budget is 4.2 GB rather than the full 8 GB because the text encoder is resident at plan time; moving it to CPU frees more.)

Both produce correct images. Without -ngl the byte-budget path is untouched and behaves as before.

Caveats

• Only the Vulkan backend was built and tested here.
• Inherits the existing --stream-layers limitations: single-device only, and the streamed tail re-reads per segment, so it trades speed for headroom.

Note

This was written with AI assistance. Errors are mine; happy to rework or drop any part of it.

[leejet]

Thanks for the contribution and for exploring GPU layer offloading support. I understand the motivation of adding -ngl for consistency with llama.cpp, but I don't think layer-based offloading maps well to diffusion models. Unlike transformer models, diffusion architectures don't expose a stable concept of "layer" to users. The same -ngl value has very different VRAM requirements across SD1.5/SDXL/Flux/etc.

This makes the option difficult to discover and configure. Users usually know their available VRAM, not the number of diffusion layers they can fit.

I would prefer keeping VRAM-based control as the user-facing API and making layer-based partitioning an internal/debug option.