CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
Published in arXiv preprint (in submission), 2026
Designed CATS, a self-speculative decoding structure applicable to different auto-regressive LLM architectures, that outperforms prior SOTA self-speculative decoding structures in inference speed under edge memory constraints. Evaluated across five benchmarks on real edge devices, CATS achieves a wall-clock speedup of up to 5.08x with no degradation in generation quality, outperforming the prior SOTA method by up to 1.45x.
Recommended citation: Han, Y., Jin, Y., Zhao, D., & Sun, J. (2026). "CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration." arXiv preprint arXiv:2605.11186.
Download Paper