Moatasemmofadal/ssd
Accelerate LLM inference by running speculative decoding in parallel, improving speed without sacrificing exactness or output quality.
What's novel
Accelerate LLM inference by running speculative decoding in parallel, improving speed without sacrificing exactness or output quality.
Code Analysis
10 files read · 3 roundsA multi-GPU async speculative decoding inference engine that runs a draft model on a separate GPU in parallel with the target model, using tensor-backed tree caching, EAGLE3 conditioning, CUDA graphs, and FlashInfer paged KV cache for high-throughput LLM generation.
Strengths
Genuinely deep implementation of the full speculative decoding pipeline — multi-process NCCL communication, async draft/target overlap, proper p/q ratio verification with recovery sampling, EAGLE3 hidden-state conditioning, CUDA graph capture, and tensor-parallel target inference. The code shows real systems engineering (pre-allocated buffers, fused NCCL sends, state save/restore patterns) rather than toy examples.
Weaknesses
The README is completely misleading — it describes a Windows .exe download for non-technical users, while the code is a research-grade multi-GPU CUDA inference engine requiring PyTorch, Triton, FlashInfer, and NCCL. There are essentially no unit tests (only a single async_helpers/tests.py file), and some code paths have debug prints left in production code with __debug__ guards that are fragile.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
16
Contributors
2
Files
54
Active weeks
1
Repository
Language
Python
Stars
1
Forks
0
License
MIT