The operator builds it. The model compresses it.
Congratulations.
You want to build a language model that can read one million tokens.
This sounds like a reading problem.
It is not.
Before the model reads anything, someone builds the room.
Someone selects the files. Someone writes the prompt. Someone decides which code, documents, logs, transcripts, papers, messages, screenshots, outputs, and prior mistakes are allowed inside the context window.
The model does not receive the world.
It receives a room.
Then it compresses it.
DeepSeek V4 is interesting because it turns this hidden machinery into the product. Its preview release presents two Mixture-of-Experts models: DeepSeek-V4-Pro with 1.6 trillion total parameters, and DeepSeek-V4-Flash as the smaller variant. Both support a one-million-token context window, and the architecture is explicitly framed around reducing compute and memory costs for long-context use. Reuters reports the release as DeepSeek V4, while DeepSeek's own model cards use DeepSeek-V4-Pro and DeepSeek-V4-Flash. (Reuters; DeepSeek V4-Pro model card; DeepSeek V4-Flash model card)
That phrase matters.
Not perfect memory.
Not full understanding.
Not a machine patiently reading the whole archive.
Cost-effective context.
A million-token context window is not a window. It is a filing system. It is a way to make a very large room economically available to the model without forcing the model to inspect every object in that room at full resolution every time it speaks.
DeepSeek V4 does not show us a machine that remembers everything.
It shows us a machine learning what it can afford to forget.
Step 0 — Build the room
Before you build the model, build the room.
This is the part usually skipped in model reports.
A context window does not fill itself. Someone selects the documents. Someone decides whether the model receives the source file, the summary, the bug report, the benchmark prompt, the previous conversation, the legal clause, the financial table, the terminal log, the screenshot, the codebase, or the wrong folder entirely.
That person is not a footnote.
The operator decides what becomes visible before the model begins to reason.
This is the fourth parameter.
Scale is one parameter. Reasoning is another. Agency is another. But the human operator is still there, shaping the conditions under which the system operates.
A small prompt looks authored.
A million-token context looks objective.
But size does not remove framing. It hides framing under volume.
The model does not receive reality. It receives selected territory.
And selected territory still has an author.
A longer context window does not abolish authorship.
It disguises it as capacity.
DeepSeek V4 does not remove the fourth parameter.
It makes it harder to see.
Now the model enters the room.
And immediately discovers that it cannot afford to read it.
Step 1 — Build a transformer, then panic
Start with the transformer.
The original Attention Is All You Need paper proposed the Transformer as an architecture based solely on attention mechanisms, removing recurrence and convolution from the sequence model itself. (arXiv)
The simple version is this:
A transformer processes text by letting tokens look at other tokens.
That matters because language is relational. A word depends on the words around it. A name refers to something earlier. A legal clause modifies a previous definition. A function call depends on a variable declared far above it. A joke depends on the setup.
Attention gives the model a way to ask:
What parts of the previous text matter for this next token?
For ordinary context sizes, this is elegant.
Then the room becomes a warehouse.
At long context lengths, the system is no longer glancing across a paragraph. It is operating inside an archive. Every new token may need to relate to an enormous amount of prior material.
That is where the fantasy breaks.
The transformer is elegant until the room becomes a warehouse.
Step 2 — Make it huge, but do not wake the whole thing
The next trick is scale without full activation.
DeepSeek V4 is a Mixture-of-Experts model. The idea is not difficult:
Build a very large factory.
Then open only the relevant workshops for each token.
A dense model asks most of the model to participate. A sparse MoE model routes each token through selected experts. This lets the total model become enormous while the active computation per token stays much smaller.
DeepSeek had already developed this direction before V4. The DeepSeekMoE paper frames MoE as a way to manage computational cost while scaling parameters, using fine-grained expert segmentation and shared expert isolation to improve specialization and reduce redundancy. (arXiv)
DeepSeek-V3 continued the same logic: 671 billion total parameters, 37 billion activated per token, plus Multi-head Latent Attention, DeepSeekMoE, auxiliary-loss-free load balancing, and multi-token prediction. (arXiv)
V4 pushes that pattern further.
V4-Pro looks monstrous from the outside: 1.6 trillion total parameters. But per token, it activates only part of the model. V4-Flash is smaller, but follows the same basic logic: large total capacity, smaller active computation. (DeepSeek V4-Pro model card; DeepSeek V4-Flash model card)
So the second rule is simple:
Do not make the whole brain think every time.
The model looks enormous from the outside.
Inside, only part of the factory is open.
Step 3 — Discover that memory is the bill
Now give the model a long room.
A million tokens.
A large codebase. A court record. A research archive. A financial report stack. A month of agent logs. A book-length conversation. A company's internal documentation.
The user sees magic.
The server sees rent.
When a model generates text, it does not simply "remember" the conversation in the human sense. It stores reusable traces of prior tokens in what is usually called the KV cache — key/value representations used by attention. The longer the context, the larger the cache pressure.
The cache becomes the bill.
This is not just a DeepSeek problem. Long-context inference makes memory management central because large KV caches put increasing pressure on available hardware memory; recent systems papers explicitly treat KV-cache compression and persistence as production bottlenecks rather than philosophical questions about memory. (arXiv)
That is not how people talk about memory.
That is how infrastructure talks about cost.
A human remembers badly, emotionally, selectively, and continuously.
A serving system remembers through cache hits, prefix units, disk persistence, reuse rules, token intervals, and bills.
So the third rule is:
Stop pretending memory is free.
Step 4 — Make index cards
Now we reach the heart of V4.
DeepSeek V4's long-context story is not simply that the model can accept more text. The interesting part is how it makes that text cheaper to use. Reuters describes V4's architecture as designed to reduce compute and memory costs for long-context use, while follow-on technical work identifies DeepSeek-V3.2 and V4 as introducing Compressed Sparse Attention. (Reuters)
That is the machine's real sentence.
Not "we read more."
"We made reading cheaper."
Compressed Sparse Attention, or CSA, is the index-card move.
The simplest way to understand CSA is this: the model does not treat every old token as an equally expensive object. It compresses pieces of the past, scores which compressed pieces matter, and reads only the selected ones.
StreamIndex describes the mechanism clearly: DeepSeek-V3.2 and V4 introduce CSA where a learned indexer scores compressed keys, top-k entries are selected per query, and a sparse attention kernel reads only those selected entries. (arXiv)
This is not human memory.
This is archive management.
The model is not remembering the room.
It is maintaining a cheaper representation of it.
Imagine a library where the model does not pull every book from every shelf. Instead, it builds cards. Each card represents a compressed piece of the archive. When the model needs to answer, it searches the cards and chooses which compressed entries are worth reading.
CSA gives the model index cards.
That is the fourth rule:
Do not read the archive.
Build a catalogue.
Step 5 — Draw a bad map of the whole room
Index cards help with selected details.
But a million-token room also needs orientation.
That is where the second half of the long-context idea enters: a more heavily compressed view of the room.
The human version:
CSA is the card catalogue.
HCA is the bad map.
A bad map is not useless. It is not a full satellite image. It does not show every chair, wire, crack, label, and footprint. But it tells you where the rooms are. It tells you that the kitchen is not the basement. It tells you that the north wing exists.
For long-context models, that matters.
Sometimes the model needs detail.
Sometimes it needs bearing.
Sometimes it needs to know that something relevant happened far back in the room without carrying the entire passage in full resolution.
HCA gives the model a compressed world map.
It is crude.
But crude is cheaper than lost.
Step 6 — Keep the present sharp
The past can be compressed.
The present cannot.
This is the part of the architecture that feels almost human.
The model can afford to blur the distant past.
It cannot afford to blur the sentence happening now.
When you are in a conversation, the latest words matter intensely. A correction, a negation, a constraint, a new instruction, a changed variable — these can alter the entire answer.
If the model compresses the immediate present too aggressively, it becomes clumsy. It misses the user's last turn. It responds to the old instruction. It keeps following the wrong frame.
So V4's long-context system is not one flat memory.
It is a hierarchy.
The distant past becomes compressed.
The wider room becomes mapped.
The selected archive becomes searchable.
The present stays sharp.
That sentence should be taken literally:
The past is compressed.
The present is kept sharp.
Step 7 — Compress the rest of the machine
Once the room has been compressed, the machine starts compressing itself.
Not metaphorically.
Numerically.
DeepSeek's recent models already sit inside a lineage of low-level efficiency work: V3 used MLA, MoE, multi-token prediction, and a strong emphasis on efficient inference and cost-effective training. (arXiv)
The point is not that lower precision, multi-token prediction, or post-training consolidation are magic.
The point is that modern intelligence, in production, keeps asking the same question:
What can be lowered without breaking the illusion?
Some values need to stay sharp. Others can be stored at lower resolution. Some parts of the model need full attention. Others can be routed, compressed, cached, or approximated.
The same logic appears in training. DeepSeek-V3's report presents multi-token prediction as part of its training objective for stronger performance. DeepSeekMath introduced GRPO, a PPO variant designed to improve reasoning while reducing PPO memory usage. (arXiv)
But these are not separate stories.
They are the same story at different layers.
The room is compressed.
The cache is compressed.
The weights are compressed.
The training signal is shaped.
The active model is routed.
DeepSeek V4 is not one trick.
It is cost pressure applied everywhere.
At production scale, intelligence often becomes a compression format.
Step 8 — Turn reasoning into a budget setting
Now add a switch.
This is one of the most important product facts hiding inside modern AI systems: reasoning is not only a capability. It is increasingly a budget setting.
A model may not have one fixed level of "reasoning." It may have modes. It may spend more tokens, more time, more money, and more context on one request than another. It may be shallow because the user chose shallow. It may be deeper because the serving layer allowed deeper.
Reuters describes V4-Pro as aimed at complex tasks such as agentic coding and competitive programming, while V4-Flash is positioned as faster and more cost-effective but weaker on more demanding agent-based tasks. (Reuters)
That split matters.
It means "intelligence" is not just an abstract property of the model.
It is also a product configuration.
So when someone asks, "Does DeepSeek seek deep?" the answer is not simply yes or no.
The better answer is:
At what budget?
Under what mode?
With what context?
Selected by which operator?
Measured by which benchmark?
Step 9 — Benchmark it and start the fight
Now test the machine.
This is where everyone gets loud.
The benchmark says the model scored well. The company cites the score. The community repeats the chart. The chart becomes a sentence.
"DeepSeek is back."
"DeepSeek beat X."
"DeepSeek is behind Y."
"DeepSeek is cheap."
"DeepSeek is overhyped."
The benchmark becomes the room.
But the benchmark is not the machine.
Independent testing complicates the story. Artificial Analysis places DeepSeek V4-Pro Max among the leading open-weight reasoning models, but its public evaluation page also shows the trade-offs around output tokens, speed, price, and non-hallucination rate. (Artificial Analysis)
That is the benchmark problem in miniature.
The model is stronger.
The model is useful.
The model is expensive in output tokens.
The model still hallucinates.
All of these can be true.
A benchmark is not nothing. It is evidence. But it is evidence under a frame. It tells us what the system did when measured. It does not tell us what the system is.
For DeepSeek V4, the architecture tells us something the benchmark cannot:
The performance is being made affordable by compression.
The benchmark shows the output.
The architecture shows the bill.
Step 10 — What comes after the room
At this point, you have not built a machine that remembers everything.
You have built a reading machine that survives the cost of pretending to.
That distinction matters because the next bottleneck is already visible.
Once the model learns to skim, the problem becomes the machinery of skimming.
A paper like StreamIndex points directly at this next layer. It argues that public CSA implementations can materialize enormous score tensors before top-k selection. For V4-Flash-shaped inputs, the paper says that intermediate can reach 256 GB at sequence length 65,536, while its streaming top-k implementation runs the same indexer to sequence length 1,048,576 with 6.21 GB peak HBM on synthetic-but-realistic V4-shaped inputs. (arXiv)
That is the direction.
Not just better attention.
Better indexing.
Better cache behavior.
Better memory layout.
Better serving.
Better machinery for deciding what not to read.
The future model may not simply be a larger reader.
It may be a better librarian.
Not just better attention.
Better indexing.
Final frame
So does DeepSeek seek deep?
Not in the romantic sense.
It does not stare harder into the text until meaning appears.
It does not read the whole room like a patient human with infinite attention.
It does not make the operator disappear.
It does not turn benchmarks into truth.
It seeks deep by making depth affordable.
It builds index cards.
It draws bad maps.
It keeps the present sharp.
It lowers the resolution where it can.
It wakes only part of the factory.
It turns reasoning into a budget setting.
It compresses the past and charges for the future.
The operator arranged the room.
The model compressed the room.
The benchmark scored the performance.
Then everyone argued about intelligence.
DeepSeek V4 does not read the room.
It compresses it.
And once you understand that, the million-token claim becomes less magical and more interesting.
Because the real breakthrough is not that the machine remembers everything.
The real breakthrough is how much it can forget before we still call it memory.
This article is built from DeepSeek V4 release reporting, DeepSeek's V4-Pro and V4-Flash model cards, the Transformer and DeepSeekMoE paper lineage, DeepSeek-V3 and DeepSeekMath technical papers, Artificial Analysis' independent V4 assessment, and StreamIndex as an early example of follow-on work around compressed sparse attention implementation.