Shallow Verification

How it works

Benefits

Highlighted claims

• CATS uses a shallow verification pass that loads intermediate layers once per decoding cycle to verify draft tokens in parallel. — CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
• The shallow verifier produces correction candidates when its tokens differ from corresponding draft tokens. — CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
• Draft tokens and correction tokens are assembled into a verification tree for target verification. — CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
• The final target pass verifies the main branch and correction branches in one batched forward pass using tree-masked attention. — CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
• CATS improves verification capacity by keeping the drafter shallow and loading the shallow verifier once per cycle. — CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration