The Vibe Father
NVIDIA says dense attention consumed 85 percent of DeepSeek-R1 prefill time at 128K context in its measured setup, up from 18 percent at 4K. Its July 31 engineering post argues that long-context performance must be co-designed at the model and GPU levels rather than fixed only with a faster kernel.
Prefill and decode are different problems
Prefill processes the input prompt in parallel and becomes compute-bound as context grows. Decode usually generates one token at a time and repeatedly reads the KV cache, making it memory-bandwidth-bound. The same architectural choice can therefore affect the two phases differently.
| Phase | Dominant behavior | Scaling with sequence length |
|---|---|---|
| Prefill | Compute and softmax | Approximately quadratic for dense attention |
| Decode | KV-cache memory traffic | Approximately linear per generated token |
NVIDIA’s four guidelines
- Choose larger query-to-KV group sizes for decode. More query heads sharing a KV head can reduce the data loaded per token and improve utilization.
- Use head dimensions of 128 or 256. NVIDIA identifies these as practical fits for GPU tiles, 128-byte transfers, and tensor-memory limits.
- Reduce effective KV state. Compression, sliding-window or sparse attention, and hybrid architectures can limit the global state carried through every layer.
- Match parallelism to KV-head count. Keep tensor parallelism at or below the number of KV heads or risk duplicating KV state across GPUs.
What application developers can use today
Most teams do not design foundation-model attention. They still control effective context. Avoid sending entire repositories on every call. Reuse stable prefixes, retrieve only relevant files, summarize finished work, and measure time to first token separately from output speed.
A million-token window is capacity, not a performance promise. Dense prefill cost grows quickly, and long KV state makes each decode step heavier. More context can reduce missing information while making the interaction slower and more expensive.
Bottom line
NVIDIA’s analysis explains why long-context model design is becoming a hardware problem. For model builders, group size, head dimension, KV state, and sharding must be planned together. For product teams, the immediate win is disciplined context, send what the task needs and prove that extra tokens improve the result.
