Think about how you write an important email or essay.
You don’t blindly type the final polished words from left to right without knowing how the sentence ends. Instead, your brain naturally works from coarse to fine:
- First, you decide the big-picture concept (“Politely decline the meeting, suggest Friday instead”).
- Next, you sketch the paragraph structure.
- Finally, you fill in the exact words and grammar.
Yet for the past decade, almost every Large Language Model (LLM)—from GPT-4 to Claude and Llama—has been forced to write like a blindfolded typewriter: one single word (token) at a time, strictly from left to right.
In October 2026, researchers at Stanford University’s Hazy Research lab and Together AI (Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini, and Christopher Ré) published a landmark paper (arXiv:2610.14162v1: “Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation”) that teaches AI to write the way humans actually think: from rough concepts down to fine details.
The result? Next-Concept Modeling (NCM) generates text 10× to 31× faster than state-of-the-art Diffusion Language Models while matching or beating their quality—and even outperforms standard LLMs on complex logic puzzles like Sudoku.
In this post, we will break down everything inside the NCM paper in plain English so anyone can understand how it works, why it is so fast, and what every component of the architecture does.
Level 1: The Three Ways an AI Can Write a Sentence
To appreciate why Next-Concept Modeling (NCM) is such a breakthrough, let’s compare the three ways a language model can generate a 100-word paragraph:
flowchart TD
subgraph AR["1. Autoregressive (Standard LLMs: GPT, Llama)"]
A1["Word 1"] --> A2["Word 2"] --> A3["Word 3"] --> A4["... (100 sequential steps!)"]
end
subgraph DLM["2. Diffusion Language Models (MDLM, SEDD)"]
D1["[Noise, Noise, Noise, ... 100 slots]"] --> D2["Step 1: Slightly clearer (All 100 words)"]
D2 --> D3["Step 50: Mostly clear (All 100 words)"]
D3 --> D4["Step 128: Final 100 words (No KV Cache!)"]
end
subgraph NCM["3. Next-Concept Modeling (NCM — Coarse-to-Fine)"]
N1["Level 1 (Tiny — 4 Concept Codes): Global Topic Sketch"] --> N2["Level 2 (16 Codes in Parallel): Section Outline"]
N2 --> N3["Level 3 (64 Codes in Parallel): Sentence Details"]
N3 --> N4["Decode -> Final 100 Words in Just a Few Fast Steps!"]
end
1. The Typewriter Way: Autoregressive (Next-Token Prediction)
Standard LLMs predict one token at a time.
- The Problem: To write 1,000 tokens, the model must run 1,000 separate sequential steps. Even if you have a massive GPU with thousands of parallel cores, step #500 cannot start until step #499 finishes. Also, because the model commits to each word left-to-right, it can easily paint itself into a logical corner on planning tasks.
2. The Fog-Clearing Way: Diffusion Language Models (DLMs)
To break the one-word-at-a-time bottleneck, researchers built Diffusion Language Models (like MDLM and SEDD). They start with a full page of 1,000 [MASK] / noise tokens and repeatedly un-mask a few words at a time over 100+ steps.
- The Problem: On every single step, the Transformer has to look at all 1,000 tokens all over again. Because every token can change at any time, diffusion models cannot use KV Caching (the memory shortcut that makes standard LLMs efficient). Processing all 1,000 tokens 128 times in a row requires massive compute!
3. The Artist’s Sketch Way: Next-Concept Modeling (NCM)
Think of how a digital artist paints a portrait or how a progressive JPEG loads on a slow website:
- First, you paint a tiny 4×4 blurry sketch of the big shapes (Coarse Concepts).
- Next, you add medium brushstrokes to turn the shapes into faces and trees (Mid-Level Structure).
- Finally, you add sharp eyelashes and leaf textures (Fine Details).
Next-Concept Modeling (NCM) brings this exact coarse-to-fine pyramid (inspired by Visual Autoregressive Modeling / VAR in images) to human language!
Level 2: Inside the NCM Architecture (The Two Stages)
The paper builds Next-Concept Modeling in two distinct stages:
- Stage 1 — The Multiscale Residual VQ-VAE: A “Concept Pyramid Tokenizer” that learns how to compress any piece of text into a multi-level hierarchy of discrete codes (from rough concepts down to fine details) and reconstruct the text back again.
- Stage 2 — The Level-Wise Transformer: A generator that predicts the pyramid one level at a time, outputting all codes within each level in parallel!
Let’s unpack both stages without getting lost in math.
Stage 1: The Multiscale Residual VQ-VAE (Building the “Concept Pyramid”)
Standard LLMs chop text into a flat list of word pieces (tokens) of length L:
["The", " cat", " sat", " on", " the", " warm", " mat", "."] (Length L = 8)
NCM says: “Instead of a flat list, let’s turn that sentence into a pyramid of K levels with lengths s₁ < s₂ < ⋯ < s_K!”
To build this pyramid, the authors train a Multiscale Residual VQ-VAE (Vector-Quantized Variational Autoencoder). Here is how it works step-by-step:
ORIGINAL TEXT (Length L = 256 tokens)
|
v
[ VQ-VAE Encoder ] --> Continuous Representation (z)
|
+---> Level 1 (Coarse, e.g. s₁ = 8 codes)
| 1. Downsample z to length 8
| 2. Snap to closest IDs in Level-1 Codebook (C₁) -> Captures Global Gist!
| 3. Upsample back to 256 and SUBTRACT from z to get what's still missing ("Residual r₁")
|
+---> Level 2 (Medium, e.g. s₂ = 32 codes)
| 1. Downsample the leftover Residual r₁ to length 32
| 2. Snap to closest IDs in Level-2 Codebook (C₂) -> Captures Paragraph/Clause Structure!
| 3. Upsample back to 256 and SUBTRACT to get leftover Residual r₂
|
+---> ... Repeats down to Level K (Fine Details, e.g. s_K = 256 codes)
| Captures exact words, punctuation, and grammar!
|
v
[ VQ-VAE Decoder ]
Sums all upsampled levels together -> Reconstructs the Exact Original Text!
Every word in the name “Multiscale Residual VQ-VAE” has a simple job:
- Multiscale (
s₁ < s₂ < ⋯ < s_K): Early levels have very few slots (short sequence lengths₁), forcing the model to summarize broad, global concepts. Later levels have more slots (s_K), allowing fine word-by-word detail. - Residual (“What’s Still Missing?”): Just like subtracting your rough sketch from the final photo to see what details you still need to draw, each level
konly encodes the residual error left over by levels1 … k−1. - Vector-Quantized (
VQ) with Level-Specific Codebooks (C₁, C₂, …, C_K): Instead of keeping messy floating-point numbers, each level snaps its vectors to discrete dictionary IDs (like"Concept #412"). Crucially, the paper uses separate codebooks for each level so Level 1’s dictionary specializes in high-level topics while LevelK’s dictionary specializes in fine surface wording. - Variational Autoencoder (
VAEEncoder + Decoder): Trained with a text reconstruction cross-entropy loss plus a VQ commitment loss so that summing allKlevels reconstructs the original text faithfully.
Once Stage 1 finishes training, the VQ-VAE is frozen. It now serves as a fast translator between raw text and “Concept Pyramids.”
Stage 2: The Level-Wise Transformer (Generating Text Coarse-to-Fine)
Now comes the fun part: generating new text!
Instead of training a Transformer to predict Token 1 → Token 2 → Token 3 → ⋯ → Token 256, NCM trains a single shared Transformer to predict Level 1 → Level 2 → ⋯ → Level K:
flowchart LR
L1["Step 1: Predict ALL of Level 1 in Parallel\n(e.g., 8 Coarse Concept Codes)"] --> L2["Step 2: Given Level 1 (KV-Cached),\nPredict ALL of Level 2 in Parallel\n(e.g., 32 Mid-Level Codes)"]
L2 --> L3["Step 3: Given Levels 1..2 (KV-Cached),\nPredict ALL of Level 3 in Parallel\n(e.g., 64 Phrase Codes)"]
L3 --> LK["Step K: Predict Fine Level K in Parallel\n-> VQ-VAE Decoder Outputs Final Text!"]
Look at the two architectural breakthroughs that make Stage 2 work so well:
1. Hybrid “Block-Causal” Attention (Causal Across Levels, Bidirectional Within a Level!)
How should tokens look at each other inside the NCM Transformer?
- Across Different Levels (
Level 1 → Level 2 → Level 3): Attention is strictly causal. Level 3 can look back at Level 1 and Level 2, but Level 1 cannot peek into the future at Level 3.- Why this matters: Because earlier levels never change once generated, NCM supports full KV Caching across levels! Once Level 1 and Level 2 are generated, their Key/Value tensors are locked in memory and never recomputed.
- Inside the Same Level (All codes within
Level k): Attention is bidirectional (full self-attention)! Every slot inside Levelkcan see every other slot in Levelksimultaneously, allowing the model to coordinate the entire sentence structure in parallel.
Block-Causal Attention Matrix Across 3 Levels:
(1 = Can Attend, 0 = Masked Out)
[Lvl 1] [ Lvl 2 ] [ Lvl 3 ]
[Lvl 1] 1 0 0
[Lvl 2] 1 1 0
[Lvl 3] 1 1 1
2. Optional Within-Level Iterative Refinement
When jumping to a large, fine-grained level near the bottom of the pyramid, predicting hundreds of fine codes in a single shot can occasionally leave minor local inconsistencies.
- NCM supports lightweight iterative refinement within a level (similar to MaskGIT / discrete masked refinement for just a couple of steps inside Level
k). - Because all coarser levels
1 … k−1are already KV-cached and fixed, refining a level is vastly cheaper than running a full-sequence Diffusion Language Model!
Level 3: Why Is NCM 10× to 31× Faster? (Counting the Work)
Let’s compare the actual computational work (Number of Function Evaluations / Forward Passes and Attention Cost) needed to generate a passage of text:
| Generation Paradigm | How It Generates Text | Supports KV Caching? | Why It’s Slow or Fast |
|---|---|---|---|
| Autoregressive (AR) (GPT, Llama) | 1 token at a time (L sequential steps) |
✅ Yes | Must run L separate serial passes (e.g., 512 or 1,024 steps). Memory-bandwidth bound on GPUs. |
| Diffusion Language Models (DLM) (MDLM, SEDD) | Un-masks all L tokens over T steps (e.g., 128–1,024 steps) |
❌ No! | Every single denoising step runs full bidirectional attention over the entire max length L. |
| Next-Concept Modeling (NCM) (Stanford / Together AI) | 1 hierarchy level at a time (K levels, all tokens in a level predicted in parallel) |
✅ Yes (Across Levels!) | 1. Only K hierarchy levels (far fewer steps than L or T).2. Early levels are tiny sequences ( s_k ≪ L), costing almost zero compute!3. Completed levels are KV-cached. |
Think about that second point: in a Diffusion Language Model, Step 1, Step 2, and Step 50 all pay the full price of processing 1,024 tokens. In Next-Concept Modeling, the early steps operate on tiny coarse sequences (like 8, 16, or 32 concept codes), so the first 70% of the hierarchy finishes in a fraction of a millisecond!
Level 4: Complete Benchmark Results from the Paper
The authors evaluated NCM across four major benchmarks spanning unconditional text generation, large-scale language modeling, abstractive summarization, and logical reasoning. Here is every key finding from the paper:
1. Text Generation Speed & Quality (OpenWebText & LM1B)
How do you measure if generated text is both fluent and diverse? Researchers measure Generative Perplexity (Gen PPL) (scored by an external evaluator LLM like GPT-2 Large—lower is more fluent) alongside Entropy / Diversity (to ensure the model isn’t just repeating dull phrases) and Wall-Clock Speedup:
| Benchmark Dataset | Task Type | NCM Quality vs. Strongest Diffusion Baseline (MDLM / SEDD) |
NCM Inference Speedup |
|---|---|---|---|
| OpenWebText | Open-domain web text generation | Competitive / Matches strong DLM baselines on Gen PPL and diversity | 10× Faster |
| LM1B (One Billion Word Benchmark) | Standard language modeling & generation | Beats (Lower Gen PPL) than the strongest DLM baseline | 21× Faster |
| XSum (Extreme Summarization) | Conditional document-to-summary generation | Matches ROUGE-L score of the strongest DLM baseline | 31× Faster |
Look at that XSum summarization result: NCM produces summaries with the exact same ROUGE-L accuracy as the best Diffusion Language Model while running 31 times faster!
2. Logical Reasoning & Global Planning: The Sudoku Benchmark
One of the most fascinating experiments in the paper tests NCM on Sudoku puzzles across Easy, Medium, and Hard difficulty levels.
Why is Sudoku a torture test for standard Autoregressive LLMs?
- In Sudoku, the number you place in the top-left corner (Row 1, Col 1) depends on constraints in Row 9 and Column 9.
- A standard left-to-right Autoregressive model is forced to guess Row 1, Column 1 on its very first step—before it has resolved the rest of the board! Once it makes a wrong guess early on, the entire board becomes unsolvable.
What happened when NCM played Sudoku?
- NCM beat both Autoregressive (AR) and Diffusion (DLM) baselines across all three difficulty levels (Easy, Medium, and Hard) while using significantly fewer function evaluations (NFEs)!
- Why? Because NCM’s multiscale VQ-VAE naturally encodes the global board constraints in the coarse levels first, and then resolves the individual cell digits in parallel in the finer levels.
3. Training Efficiency & Ablation Studies in the Paper
The paper also includes detailed ablation studies showing why each piece of NCM is necessary:
- Sample & Token Efficiency During Training:
Flat diffusion language models must learn to denoise millions of random masking permutations, requiring massive numbers of training tokens to converge. Because NCM’s frozen VQ-VAE provides a structured coarse-to-fine supervision signal at every scale
1 … K, the Stage-2 Transformer converges using significantly fewer training tokens. - Level-Specific Codebooks vs. Shared Codebook:
Using separate VQ codebooks per scale (
C₁ … C_K) outperforms a single shared codebook because coarse concepts (paragraph-level semantics) have very different statistical distributions from fine-grained residuals (local word endings and punctuation). - Residual Quantization vs. Independent Scale Encoding: Having each level predict only the residual difference from coarser levels prevents redundant information across scales and sharply improves reconstruction accuracy in Stage 1.
- Effect of Scale Schedule & Within-Level Refinement: Adding a small number of iterative refinement steps on the finest scale boosts generation fluency (lowering Gen PPL) while preserving the massive 10×–31× wall-clock speedup over full-sequence diffusion.
Summary: Why Next-Concept Modeling Matters for the Future of LLMs
Let’s recap what Next-Concept Modeling (arXiv:2610.14162) brings to the LLM landscape:
- Models Should Think in Concepts Before Words: Compressing text into a multiscale hierarchy (
Multiscale Residual VQ-VAE) lets a language model plan global structure before committing to individual words. - Parallel Within Levels + Causal Across Levels = Extreme Speed: By predicting all codes in a level simultaneously while keeping KV caching across levels, NCM achieves 10× to 31× speedups over Diffusion Language Models.
- Better Reasoning Where Left-to-Right Fails: On global constraint problems like Sudoku, planning coarse-to-fine beats both token-by-token autoregressive models and flat diffusion models across every difficulty tier.
References & Official Links
- Primary Research Paper (October 2026): Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini, and Christopher Ré, “Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation”, arXiv:2610.14162v1 (Stanford University Hazy Research & Together AI).
- Official Open-Source Codebase: HazyResearch/ncm on GitHub — Official PyTorch implementation of the Multiscale Residual VQ-VAE tokenizer and Level-Wise NCM Transformer.
- Related Visual Autoregressive Inspiration: Keyu Tian et al. (2024), “Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (VAR)”, NeurIPS 2024.