Speculative Decoding for LLMs
Speculative Decoding Speculative decoding is an innovative optimization technique designed to accelerate the inference process in large language models (LLMs) without compromising the quality of the output. It achieves this by generating multiple tokens in parallel and incorporating a verification mechanism to ensure the correctness of these speculated tokens, thereby guaranteeing that the overall output is identical to that of vanilla decoding. This approach significantly reduces the cost of generative AI and increases its adoption by optimizing the cost of inference of LLMs. ...