Something quiet happens when a language model gets fine-tuned to be helpful and safe — it starts to sound like everyone else. That homogenization effect, known as mode collapse, has long been treated as a byproduct of imperfect training algorithms. But new research challenges that assumption at its root, tracing the problem not to the algorithm, but to the data itself — and specifically to a deeply human cognitive quirk baked into every preference label.
Summary
Key takeaways
- Post-training alignment reduces diversity in large language models, producing mode collapse — a tendency to generate repetitive, predictable outputs.
- The root cause is typicality bias in preference data: human annotators consistently favor familiar-sounding text, which shapes model behavior at scale.
- Typicality bias is grounded in cognitive psychology and has now been formalized theoretically and verified empirically on preference datasets.
- Verbalized Sampling (VS) is a training-free prompting method that counteracts mode collapse by asking the model to generate multiple responses alongside probability estimates.
- In creative writing tasks, VS increases output diversity by 1.6 to 2.1 times compared to direct prompting, with gains across dialogue, open-ended QA, and synthetic data generation.
Mode Collapse Driven by Post-Training Alignment and Typicality Bias
Mode collapse mitigation has become one of the more pressing challenges in modern AI development, precisely because the problem is so easy to miss. A fine-tuned model still answers questions. It still writes poems. It just writes the same kind of poem, over and over, in slightly different words.
Impact of Post-Training Alignment on LLM Diversity
Post-training alignment — the process by which a base language model is shaped through human feedback to be more helpful, harmless, and honest — consistently reduces the generative diversity of large language models. The result is a narrowing of the output space: models converge on a smaller set of “acceptable” responses, trimming the tails of their distribution in ways that sacrifice originality and variety.
This is not a minor stylistic concern. For applications like synthetic data generation, creative writing, or dialogue simulation, diversity is a functional requirement. A model that collapses toward the mean produces training data that reinforces the same biases, dialogue that feels scripted, and creative output that feels derivative.
Role of Typicality Bias in Preference Data
Typicality bias is the mechanism at the heart of the problem. When human annotators evaluate model outputs and label which responses they prefer, they systematically favor text that feels familiar — responses that match their intuitive sense of what a “typical” good answer looks like. This preference for the prototypical over the novel is not a flaw unique to AI labelers; it reflects well-established findings from cognitive psychology about how humans categorize and evaluate information.
The consequence, at scale, is significant. Preference datasets built from thousands of such annotations encode a structural bias against unusual but valid responses. Models trained on this data learn, implicitly, that unfamiliar outputs are less desirable — even when those outputs are correct, creative, or meaningfully different from the median. Typicality bias in AI systems, in other words, is an inheritance from human cognition passed through the training pipeline.
Theoretical and Empirical Analysis of Typicality Bias
The research behind these findings does not stop at identifying the problem. The authors formalize typicality bias theoretically, constructing a rigorous framework that explains how annotator preferences distort the learned distribution of aligned models. They then verify this effect empirically, testing it against real preference datasets to confirm that the bias is not incidental but pervasive and central to the mode collapse phenomenon.
This dual approach — theoretical formalization followed by empirical validation — matters because it shifts mode collapse from a vague observation into a tractable, well-defined problem. It also opens the door to principled solutions, rather than ad hoc engineering fixes. By understanding why alignment narrows output diversity, researchers can design methods that address the cause rather than mask the symptom.
Verbalized Sampling: A Training-Free Approach to Mitigate Mode Collapse
The proposed remedy is called Verbalized Sampling, and its core insight is elegant in its simplicity. Rather than retraining the model or modifying the alignment pipeline — both expensive and technically demanding — Verbalized Sampling works entirely at inference time, through a change in how the model is prompted.
Mechanics of Verbalized Sampling Prompting
Instead of asking a model to produce a single response, Verbalized Sampling instructs the model to generate a set of candidate responses and assign explicit probability estimates to each. A prompt might read: “Generate 5 jokes about coffee and their corresponding probabilities.” By forcing the model to reason over a distribution of possible outputs rather than committing to one, VS bypasses the mode-collapsing tendency that alignment has instilled. The model’s pre-trained generative diversity, which was suppressed but not erased by fine-tuning, gets reactivated through this probabilistic framing.
The practical appeal here is real. No retraining. No new datasets. No changes to the model architecture. The method is applicable to any aligned language model, and it introduces no additional infrastructure burden.
Performance Improvements Across Multiple Tasks
Experiments testing Verbalized Sampling across a range of tasks confirm the approach works — and the gains are not marginal. In creative writing tasks (poems, stories, jokes), VS increases output diversity by a factor of 1.6 to 2.1 times compared to direct prompting. Similar improvements appear in dialogue simulation, open-ended question answering, and synthetic data generation. Critically, these diversity gains do not come at the cost of factual accuracy or safety — both remain intact.
An additional pattern emerges in the data: more capable models appear to benefit more from Verbalized Sampling than less capable ones. This suggests that stronger base models have more suppressed diversity to unlock — their alignment training has constrained a richer underlying distribution, and VS provides a more effective key to open it.
Why This Changes the Conversation Around LLM Diversity
What makes this work analytically important is the reframing it offers. Most prior research treated mode collapse as an algorithmic problem — something to fix by improving RLHF methods, modifying reward models, or adjusting training objectives. This research repositions the diagnosis: the bottleneck is in the data, not the algorithm. Preference datasets, built by humans applying human cognitive patterns, carry structural biases that no amount of algorithmic refinement will fully remove if the underlying labels remain unchanged.
This data-centric perspective has broader implications. It suggests that the quality of human feedback — not just its quantity — is a fundamental constraint on how diverse and generative aligned models can be. For researchers and practitioners building preference datasets, the typicality bias finding is a concrete warning: annotator tendencies shape model personality in ways that are systematic, measurable, and consequential.
Verbalized Sampling, meanwhile, represents a practical inference-time answer to a training-time problem. Its value lies not just in the diversity improvements it delivers today, but in what it demonstrates: that the generative breadth of pre-trained models is not lost through alignment — it is merely suppressed, and addressable without starting from scratch.
FAQ
What causes mode collapse in large language models?
Mode collapse is primarily caused by typicality bias in preference data used during post-training alignment, where annotators systematically favor familiar-sounding text. This bias, rooted in cognitive psychology, trains models to converge on predictable outputs and avoid unusual but valid responses.
How does Verbalized Sampling mitigate mode collapse?
Verbalized Sampling prompts the model to generate a set of candidate responses and verbalize a probability distribution over them — for example, producing five versions of an answer alongside likelihood estimates. This approach reactivates the model’s pre-trained generative diversity without requiring any additional training or architectural changes.
In which applications does Verbalized Sampling improve diversity?
Verbalized Sampling improves output diversity across creative writing (poems, stories, jokes), dialogue simulation, open-ended question answering, and synthetic data generation — increasing diversity by 1.6 to 2.1 times over direct prompting in creative writing tasks alone.
Does Verbalized Sampling affect factual accuracy or safety?
No. According to the research, Verbalized Sampling improves diversity without sacrificing factual accuracy or safety, making it a viable option for production use cases where both variety and reliability are required.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
