Large Language Models Are Bad Dice Players: The Illusion of Native Sampling

TL;DR

A new large-scale audit of 11 frontier LLMs (including GPT-4o, Llama-3, and Gemini) reveals a fundamental flaw: LLMs cannot actually sample from probability distributions. While they might look "random" in a single long response, their ability to generate statistically valid data collapses in independent calls. This flaw propagates into downstream tasks, creating severe biases in multiple-choice questions and demographic "fairness" in synthetic data.

Background: The Hidden Stochastic Requirement

We often treat LLMs as black-box engines for "diverse" content. Whether it's generating a balanced set of persona descriptions or ensuring a test's correct answers aren't always "C," we assume LLMs have an internal dice-shifter. This paper argues that this is a dangerous assumption. As LLMs move from chatbots to "world simulators," their inability to handle basic statistical distributions (like Gaussian or Poisson) becomes a critical failure point.

The "Context-Fidelity" Dilemma

The researchers identified a fascinating asymmetry in how LLMs fail, categorized into two protocols:

1. Protocol A (Batch Generation): The model generates 1,000 numbers at once. Here, the model uses its "history" to self-correct, producing a result that looks more statistically valid.
2. Protocol B (Independent Requests): The model is asked for one number, 1,000 times. Without the "scaffolding" of previous numbers, the models' internal priors take over, leading to total statistical collapse.

Figure 1: The dual-protocol evaluation pipeline used to unmask the lack of an internal sampler.

Methodology: Tiered Complexity & Geometric Metrics

The study didn't just look at coin flips. It used a 15-distribution taxonomy ranging from Tier I (Uniform, Gaussian) to Tier III (Heavy-tailed Cauchy, Gamma). To measure "how wrong" the models were, they used Wasserstein-1 (W1) Distance—a metric that calculates the "cost" of transforming the LLM's distribution into the target mathematical distribution.

Key Insight: The Inverse Scaling Trend

Normally, in statistics, more samples (higher N) lead to better convergence. In LLMs, the opposite happens. As the "sampling horizon" N increases, the model's drift accumulates, and the statistical discrepancy becomes more undeniable.

Figure 2: Performance degrades monotonically as distributions become more mathematically complex (Tier I to III).

Real-World Impact: Bias in Action

The study's most alarming findings involve downstream applications:

• Educational Bias: When asked to create 1,000 medical MCQs with "evenly distributed" correct answers, models failed miserably. GPT-OSS-120B put 54.6% of answers in position C, while position A received only 4.5%. A student could easily "game" this model.
• Demographic Collapse: In generating prompts for a "person in a coat," models could not follow US Census distributions. One model (Llama-4) generated 97.2% females when asked for a 50/50 split.
• Variance Collapse: For a "Height" attribute following a Normal distribution, models' outputs were far too narrow—showing they can match the "mean" but have no concept of "spread" (variance).

Critical Analysis & Conclusion

This work confirms that LLMs are "linguistic mimics" rather than "mathematical reasoners." They have learned to describe randomness through patterns in their training data, but they lack the underlying functional competence to simulate it.

Takeaways for Engineers:

• Stop Native Sampling: If your pipeline requires a specific distribution (e.g., for A/B testing or synthetic data), do not ask the LLM to "be random."
• Use Tool-Use/Code-Gen: Force the LLM to write a Python script using numpy.random.
• Beware of Synthetic Bias: Data generated by LLMs to train other models will likely suffer from "mode collapse," where the variety of the data is much lower than in the real world.

Limitations

While the study is a robust "audit," it is not an impossibility proof. Future architectures (like those incorporating State Space Models or explicit stochastic layers) might overcome these hurdles. However, for the current generation of Transformers, the dice are definitely loaded.