Liang Su
A new checkpoint and restore mechanism called 'execution-state capsules' is proposed for low-latency, on-device AI serving, which snapshots and restores the entire execution state bound to a graph boundary.
Mainstream LLM serving systems reuse prefix work via KV caches, managing only one positional fragment of execution state. This is ineffective for the low-latency, small-batch, on-device serving regime required by interactive agents, speech systems, and robot policies, which involve frequent branching, resets, interrupts, and re-entries.
The mechanism defines a capsule as the complete execution state at a committed graph boundary. Using a white-box kernel runtime (FlashRT), it runs captured graph plans over contiguous static buffers. This allows snapshotting, restoring, forking, or rolling back the entire execution boundary, including KV, recurrent, convolution, MTP states, and metadata.
On an RTX 5090, capsule restore is byte-exact and token-identical under greedy decode. GPU-resident snapshot and restore are sub-millisecond, with TTFT speedup over cold prefill growing from 3.9x at 2k tokens to 27x at 16k tokens. The approach defines a complementary, latency-first serving point for explicit execution-state reuse, validated on Jetson AGX Thor and DGX Spark.