Corruption
How the discrete state is noised: masking, substitution, transition matrices, or continuous-time processes.
Algorithm · From tokenization to generation
Discrete Diffusion
A tokenization-centric survey that treats discrete diffusion as one design space, from how the state space is built through training, sampling, systems, and evaluation.
Denoiser
How the reverse model is parameterized to recover clean tokens from a corrupted categorical input.
Objective
Which training loss matches the intended reverse process, from likelihood to score and ratio matching.
Sampler
Ancestral steps, remasking, block updates, guidance, and the systems tricks that make parallel decoding usable.
Where the framework lands
- Text and codeDiffusion language models for open-ended generation, infilling, and reasoning.
- Tokenized multimodal mediaImages, audio, and video after vector quantization, often jointly with text.
- Scientific alphabetsProteins, genomics, molecules, graphs, and structured planning.
Abstract
Discrete denoising diffusion models have become a compelling alternative to autoregressive modeling for discrete data, offering parallel generation and iterative global refinement. Unlike continuous diffusion, they are shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets.
This survey treats that construction as the primary design axis. Existing formulations — transition-matrix, masking, and score- or ratio-based approaches — become different instantiations of one space, with shared trade-offs in training, inference, scaling, and evaluation.