PyTorch Speculative Decoding Implementation
Budget: $1,500 – $3,000 USD
I’m preparing an academic paper on efficient language-model inference and need the speculative decoding variant I’ve designed turned into production-quality Python. I already have detailed pseudocode, mathematical notes, and reference benchmarks; what’s missing is a clean, modular implementation that sits on top of PyTorch so I can drop it into HuggingFace-style generation pipelines for experiments.
You’ll translate the algorithm into idiomatic Python 3.x, leveraging PyTorch tensors and autograd where appropriate, and write a concise test harness that demonstrates parity with greedy sampling as well as the speed-up expected from speculative execution. Please keep external dependencies light (PyTorch-stable, NumPy, and standard HuggingFace utilities are fine).
Acceptance criteria:
• A self-contained .py module exposing a SpeculativeDecoder class with clear docstrings
• Unit tests (pytest) covering edge cases and verifying deterministic output given a seeded model
• A short Jupyter notebook that loads a small pretrained model, runs baseline vs. speculative decoding, and reports latency & token-quality metrics
• Read-me with setup instructions and any implementation notes that clarify design decisions
If you’re comfortable profiling GPU kernels or suggesting minor architectural tweaks to squeeze out extra throughput, that’s a plus, but the primary goal is a correct, reproducible reference implementation that I can cite in the Methods section of my paper.
You’ll translate the algorithm into idiomatic Python 3.x, leveraging PyTorch tensors and autograd where appropriate, and write a concise test harness that demonstrates parity with greedy sampling as well as the speed-up expected from speculative execution. Please keep external dependencies light (PyTorch-stable, NumPy, and standard HuggingFace utilities are fine).
Acceptance criteria:
• A self-contained .py module exposing a SpeculativeDecoder class with clear docstrings
• Unit tests (pytest) covering edge cases and verifying deterministic output given a seeded model
• A short Jupyter notebook that loads a small pretrained model, runs baseline vs. speculative decoding, and reports latency & token-quality metrics
• Read-me with setup instructions and any implementation notes that clarify design decisions
If you’re comfortable profiling GPU kernels or suggesting minor architectural tweaks to squeeze out extra throughput, that’s a plus, but the primary goal is a correct, reproducible reference implementation that I can cite in the Methods section of my paper.