jitllm moves running model layers between GPUs mid-reply

jitllm is an inference engine that JIT-generates kernels per hardware and treats layers like movable processes; a demo shifts layers across T4 GPUs mid-reply in 8 ms without re-prefill.

Read it at Reddit →

Coverage: 1 sources, in order

1. 14 Oct 04:40 Reddit: Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill)

← Front page