Parallel Prompting speeds up shared-context LLM inference by decoding multiple questions at once

Summary

The paper introduces Parallel Prompting, a method that decodes multiple independent questions about the same long document as parallel streams in a single request, reducing memory bandwidth overhead. Implemented in SGLang with about 80 lines of code, it achieves 3.7x speedup on A100 and 5.9-6.0x on RTX A6000 compared to stock SGLang across 144 runs. The approach uses a stream-id attention mask and fixed-shape CUDA graphs, with code available on GitHub.

This is a reviewed summary. Read the complete article at the original source.

Why it matters by industry

• Artificial intelligence

  This research directly improves inference efficiency for document QA, data labeling, and LLM-based evaluation, which are common AI workloads, by reducing latency and bandwidth costs.