DeepSeek V4 Preview: Reducing the Cost of Million-Token Context

A million-token context window is useful only if a service can afford the compute, memory, and prefill needed to use it. That is the premise of DeepSeek-V4-Pro and DeepSeek-V4-Flash, announced on April 24, 2026.

Both open-weight preview models support a one-million-token window on DeepSeek’s services. The technical report focuses less on the size of that window than on reducing attention cost and KV-cache memory. V4 is an architecture-and-serving release built around the economics of long prompts.

The API exposes deepseek-v4-pro and deepseek-v4-flash in Thinking and Non-Thinking modes. The legacy aliases deepseek-chat and deepseek-reasoner were scheduled to disappear after July 24, 2026, 15:59 UTC. Open weights are available through the DeepSeek V4 Hugging Face collection.

The preview label matters. Most performance and efficiency numbers come from DeepSeek’s report. They describe the design and its initial results, not independently verified production performance.

Two Models, One Architecture

V4-Pro has 1.6 trillion total parameters and 49 billion active per token. V4-Flash has 284 billion total and 13 billion active. Both use a mixture-of-experts design, which routes each token through a subset of the weights. Active parameters are therefore more informative for per-token computation, while the full checkpoint still determines storage and distribution requirements.

Pro allocates more capacity to difficult reasoning, coding, agent, and long-context work. Flash is the lower-cost operating point. This is a deployment distinction rather than a simple quality ranking. Flash may be sufficient for frequent, structured work with reliable validation; Pro gives harder and less verifiable tasks more capacity.

V4 retains DeepSeekMoE feed-forward layers and Multi-Token Prediction from the V3 family. Its main changes concern long-context attention, residual-stream routing, optimization, quantization, and cache management. The public scope is text-first; this preview is not presented as a multimodal release.

The two sizes also clarify what MoE does and does not solve. Routing reduces the weights used for each token, but loading or distributing a 1.6-trillion-parameter checkpoint remains a large systems task. The smaller active count should not be read as the memory footprint of the whole model. For hosted use, price and latency will reflect more than parameter activation alone, especially when prompts approach the advertised limit.

Reducing the KV Cache

Long context has two important costs. Processing the prompt creates a substantial prefill workload, and generation retains keys and values from earlier tokens in the KV cache. At hundreds of thousands of tokens, that cache can dominate accelerator memory. Repeated requests with a shared prefix may also pay much of the same prefill cost unless the serving system can reuse prior work.

DeepSeek addresses these costs with three attention paths:

• Compressed Sparse Attention (CSA) compresses KV entries and selects relevant blocks.
• Heavily Compressed Attention (HCA) stores distant history at lower resolution.
• A sliding-window branch preserves recent tokens in greater detail.

The separation matches the different roles of recent and old context. Current instructions, code, and tool output often require exact access. Earlier decisions or document sections may tolerate a coarser representation if the model can still select the relevant block. Compression saves memory and computation, but it also creates the risk that a detail discarded earlier cannot be recovered later.

The report further describes reuse of compressed KV-cache entries from disk. Stable prefixes such as a repository snapshot, a documentation collection, or a policy corpus could then avoid repeated prefill. This is a serving claim, not an automatic benefit of the context window. Results will depend on request shape, cache hit rate, storage throughput, and the inference engine.

Training and Reasoning Modes

Hybrid attention is supported by changes elsewhere in the network. Manifold-Constrained Hyper-Connections (mHC) modify residual connections to stabilize signal flow through depth. DeepSeek uses the Muon optimizer for most parameters and AdamW for selected modules, including embeddings, output heads, mHC, and RMSNorm weights. The report also covers FP4-aware deployment work, another part of its effort to reduce serving cost.

The company reports 32 trillion pre-training tokens for Flash and 33 trillion for Pro. Sequence length increases in stages from 16K to 64K and finally 1M, rather than remaining at the maximum throughout training.

Post-training produces three operating modes: Non-Think for direct answers, High for deliberate reasoning, and Max for the largest reasoning budget. These settings alter output length, latency, and cost without changing the underlying parameter count.

A comparison must therefore identify both the model and the effort level. V4-Pro Max in a benchmark and V4-Flash Non-Think in an API product occupy very different points on the cost-quality curve. Context length, output budget, cache conditions, and tool setup also affect the result. This distinction is especially important when a benchmark reports only the maximum-effort configuration.

What the Benchmarks Show

The long-context tests bear most directly on the architectural claim. DeepSeek reports 83.5 on MRCR 1M and 62.0 on CorpusQA 1M for Pro Max, both above Flash Max. The results indicate that compressed attention can retain useful information across very long inputs in the reported setup. They also show that the larger active model remains stronger on difficult retrieval and synthesis.

Agentic evaluations give a mixed view of the gap between the two models. The report lists 67.9 for Pro Max and 56.9 for Flash Max on Terminal Bench 2.0. On SWE Verified, the difference narrows to 80.6 versus 79.0. Model capacity is only one variable in these tests: reasoning budget, prompts, tools, scaffolding, and evaluation rules can all change the score.

Benchmark scores also say little about the consistency of information retained across a long session. A model may retrieve a planted passage while still losing a constraint, confusing two versions of a file, or giving excessive weight to recent text. Those failure modes matter in the document and agent workloads that a million-token window is meant to support.

The frontier comparisons are vendor-reported, and many use DeepSeek’s evaluation environment. Independent tests are needed to confirm accuracy, latency, memory use, cache savings, and total cost under matched conditions. A successful synthetic million-token retrieval test also does not establish reliable reasoning over a disordered repository, log archive, or collection of PDFs.

Assessment

V4 Preview presents a coherent approach to long context: sparse compression for distant history, high-resolution access to recent tokens, a smaller KV cache, prefix reuse, and two MoE operating points. Its most important numbers are the reported reductions in inference FLOPs and cache size. The one-million-token limit is a consequence of that system, not proof that every task benefits from filling the window.

The model family therefore offers a capacity ceiling, not a recommendation to submit the largest possible prompt. Smaller, well-selected inputs remain cheaper and easier to inspect. Large windows are most useful when omission is risky or when an evolving task needs direct access to a broad record.

The design does not eliminate retrieval or summarization. Structured external memory can still improve relevance and reduce input size. Nor do 13B or 49B active parameters make the checkpoints small; total weights, runtime memory, and serving infrastructure remain substantial constraints for self-hosting.

The practical obligation for existing API users is narrow: replace the legacy model aliases before their stated retirement time. Broader adoption depends on measurements with representative prompts and serving conditions. In particular, teams need to separate the value of a larger working set from the cost of processing it and check whether prefix reuse occurs in their actual traffic.

If the reported savings hold, Flash is the more consequential production option, while Pro offers more capacity for difficult tasks where errors are costly. The report gives a plausible mechanism and initial measurements; independent tests must determine whether the FLOP and cache reductions survive realistic workloads without unacceptable retrieval loss.

Sources: DeepSeek release note, DeepSeek V4 technical report, DeepSeek V4 Hugging Face collection, and the supporting YouTube discussion.