Testing an RNG begins with a deceptively important question: what are you trying to learn? A statistical test of finished bits, an entropy estimate for a physical source, and a security review of a complete generator are three different jobs. Choosing the wrong tool can produce a reassuring answer to the wrong question.
This guide gives you a repeatable first-pass workflow using public tools from the US National Institute of Standards and Technology (NIST). It applies to classical TRNGs and QRNGs as well as software generators, but the evidence required is not the same for each.
Choose the right NIST resource
| Resource | Use it to answer | Feed it | It does not establish |
|---|---|---|---|
| SP 800-22 Rev. 1a and the NIST Statistical Test Suite | Does this output contain patterns detected by this battery? | Finished binary output, split into many streams | Where entropy came from, unpredictability, or cryptographic security |
| SP 800-90B and the official EntropyAssessment code | How much min-entropy can be conservatively estimated in a physical noise source? | Samples from the source, before conditioning, plus restart data where required | Security of the whole RNG or correct use by an application |
| SP 800-90A Rev. 1 | Is a deterministic random bit generator built from an approved construction? | Design and algorithm evidence, not just a sample file | Quality of the entropy source that seeds it |
| SP 800-90C | How are entropy sources and deterministic generators combined into complete RBG constructions? | Architecture and assurance evidence | A one-command pass or fail |
The short version is: SP 800-22 examines appearance; SP 800-90B examines a source; SP 800-90A and 90C examine construction.
NIST decided in 2022 to revise SP 800-22. Its current publication page also says that statistical testing is only a first step and cannot substitute for cryptanalysis. That limitation should appear beside every result you publish.
Before you collect data
Write down the exact path your samples take. A useful record looks like this:
- physical event or software state;
- analogue measurement, if there is one;
- digitisation and raw symbol width;
- health tests or rejection logic;
- conditioning or whitening;
- buffering, driver and API;
- file written for analysis.
Capture raw source samples and conditioned output as separate files. Record firmware, hardware revision, operating conditions, sample rate, bit ordering, discarded samples and every transformation. Hash each captured file so later reports can identify the exact evidence used.
Do not feed only whitened output into an entropy-source assessment. A hash or deterministic generator can make a weak input look uniform without creating the missing unpredictability.
Path A: test finished output with NIST SP 800-22
NIST STS contains fifteen test families, including frequency, runs, longest run, rank, spectral, templates, serial and approximate entropy tests. Different tests have different minimum lengths. NIST's own project page uses 1,000,000-bit streams for its reference samples and recommends that length for the spectral test.
1. Get the official package
Download the current Statistical Test Suite from NIST's Random Bit Generation documentation and software page. Use that page rather than an unlabelled mirror: NIST records corrections made to the source in 2014.
2. Build and check it
On a Unix-like system, unpack the archive, enter its directory, then run:
make
./assess 1000000
The number passed to assess is the length of each sequence in bits, not
the total file size. In the interactive prompts, choose your file, binary or ASCII
input as appropriate, all tests, and the default parameters for a baseline run.
Before testing your generator, reproduce NIST's supplied sample-data results.
This catches build, input-mode and interpretation mistakes. NIST says to use one
1,000,000-bit stream per sample file and inspect the individual
experiments/AlgorithmTesting/*/results.txt files for the Appendix B values.
3. Supply enough independent sequences
One long stream is not a strong campaign. The suite evaluates both the p-values from individual sequences and the proportion of sequences that pass. Collect many non-overlapping streams under a documented operating condition. Then repeat the campaign across relevant temperature, voltage, clock, startup and ageing conditions for a hardware source.
For raw binary input, confirm how bytes become bits. A reversed bit order or an
extra text header changes the sequence under test. For ASCII input, permit only
0 and 1 plus the separators accepted by the package.
4. Read the report correctly
Start with experiments/AlgorithmTesting/finalAnalysisReport.txt, then inspect
the individual results.txt files for failures.
- A p-value is not “the probability that the generator is random”. It measures how surprising the observed statistic would be under the test's null model.
- At the common significance level (alpha = 0.01), occasional failures are expected even from ideal data. Running many tests creates more opportunities for a low p-value.
- Repeated failure of the same test, a poor passing proportion, or a non-uniform distribution of p-values deserves investigation.
- Passing every test means only that this battery did not detect its listed patterns in this sample.
Path B: assess a physical source with SP 800-90B
For a TRNG or QRNG, the central question is not whether conditioned output looks uniform. It is how much min-entropy per raw sample the physical source can support under a defensible model.
1. Build NIST's EntropyAssessment tools
Follow the dependencies and build instructions in the official NIST repository. The current repository builds all programs and offers a self-test:
git clone https://github.com/usnistgov/SP800-90B_EntropyAssessment.git
cd SP800-90B_EntropyAssessment
make
cd selftest
./selftest
cd ..
The repository says a self-test delta below 1.0E-6 is a pass. Keep the tool
version, commit identifier, compiler and self-test output with your report.
2. Format the sample file
Each source symbol must fit in one byte. The optional bits_per_symbol argument
tells the program how many low-order bits represent one symbol. Use the raw symbol
alphabet produced by the measurement — do not silently convert it to a string of
ASCII digits or expand each symbol into unrelated bit samples.
SP 800-90B assessments normally require at least 1,000,000 consecutive samples. Restart testing needs its own row-dataset collection, described in Section 3.1.4.1 of the publication. Do not splice repeated captures together without recording the boundaries.
3. Run the non-IID path first
NIST's repository notes that most common entropy sources are not independent and identically distributed (IID). Unless you have evidence supporting an IID claim, use the conservative non-IID path:
make non_iid
./ea_non_iid -i raw-samples.bin 8
Replace 8 with the true bits per symbol. The -i option marks unconditioned
data and requests an initial entropy estimate.
If you have a justified IID model, run the IID tests as an additional track:
make iid
./ea_iid -i raw-samples.bin 8
An IID result is not permission to ignore the physical model. It is a statistical check of an assumption that still needs technical justification.
4. Run restart testing
Restart data looks for hidden dependencies on previous state or initial conditions. After creating the row dataset required by SP 800-90B, run:
make restart
./ea_restart -n restart-samples.bin 8 H_I
Use -i instead of -n only for an IID claim. Replace H_I with the
assessed entropy value required by the tool. Follow the repository README and the
standard for the exact dataset layout; this file is not interchangeable with the
ordinary consecutive-sample file.
5. Assess conditioning separately
Conditioning compresses an entropy-bearing input into a shorter output. It does
not manufacture entropy. NIST provides ea_conditioning for calculating the
bound after conditioning; its inputs include the conditioner dimensions and the
assessed input entropy. Treat this as a separate calculation, not as a rerun of
the raw estimator on polished output.
A practical campaign you can repeat
Use a matrix rather than one “golden” capture:
| Stage | Data | Baseline | Stress repeats | Main question |
|---|---|---|---|---|
| Raw source | Consecutive native symbols | Nominal temperature and supply | Temperature, supply, clock, startup, ageing where relevant | Does the entropy bound remain conservative? |
| Restarts | Required row dataset | Repeated clean starts | Fast cycling and boundary conditions | Does state or startup create repetition? |
| Conditioned output | Non-overlapping bitstreams | Normal operation | Same conditions as raw capture | Are obvious statistical defects visible? |
| Integration | Bytes read through the real API | Production software path | Reboots, concurrency, errors and fallback paths | Is the intended generator actually used? |
Archive the configuration, raw files, hashes, tool versions, complete output and your explanation of anomalies. A screenshot of green results is not a test record.
Common mistakes that invalidate the result
- Testing only conditioned output. It hides source bias and correlation.
- Claiming IID because the IID tool did not reject one dataset. IID is a model claim, not a convenience switch.
- Using too little data. Some STS tests become meaningless or unavailable; entropy estimates become unstable.
- Ignoring framing and bit order. The file analysed may not represent what the device emits.
- Regenerating one deterministic stream. Multiple slices from one seeded PRNG are not independent experimental repeats.
- Testing only at room temperature. Physical generators must be evaluated across their claimed operating envelope.
- Treating a pass as certification. Self-testing is useful engineering evidence. Official NIST validation is a separate process involving the Entropy Source Validation programme and accredited laboratories.
Recent failures: what the test plan must catch
These public incidents show why an output battery alone is not enough.
2023 — Libbitcoin Explorer “Milk Sad”
The bx seed command used Mersenne Twister seeded from 32 bits of system time.
Wallet recovery phrases could therefore have far less effective entropy than
their length suggested. Researchers documented thefts and assigned
CVE-2023-39910. Read the
researchers' summary and
full disclosure. The project disputed
how responsibility should be characterised, which is why the implementation and
its intended use should be assessed separately.
Test-plan lesson: inventory the seed source and its effective search space. A deterministic generator can produce output that passes statistical tests while remaining completely reproducible.
2024 — Chilkat used a non-cryptographic generator
Security firm X41 reported that versions before 9.5.0.98 used the R250 generator for security-sensitive random data. Its technical advisory describes state recovery as the risk; it does not claim a confirmed real-world breach.
Test-plan lesson: verify the algorithm and call path, not only the bytes it returns during a short test.
2026 — CryptoJS “Ill Bloom” wallet losses
The CryptoJS advisory
for CVE-2026-71851 describes WordArray.random() using a weak generator in
versions before 4.0.0. Coinspect's
investigation linked
wallet applications to exploitable seed generation and reported approximately
USD 5.7 million as a lower bound from its on-chain analysis, not an audited
total.
Test-plan lesson: dependency review belongs beside entropy testing. A safe hardware source cannot help an application that calls a different function.
2026 — COLDCARD's intended hardware path was bypassed
Coinkite reported a build and linking problem that caused a call intended for the hardware RNG to resolve to a software fallback. The vendor's technical backgrounder, security update and firmware changelog document the issue and remediation. Seeds generated on affected paths required migration; updating code cannot retroactively strengthen an existing secret.
Test-plan lesson: capture through the production API and verify which symbol the shipped build resolves to. Test source, conditioning and integration as one traceable chain.
How to report your result honestly
A useful conclusion is specific:
We tested 100 non-overlapping one-million-bit sequences from conditioned output using the NIST STS package downloaded on [date], with [version or source hash] and default parameters. The listed tests showed [result]. This is a statistical observation about these samples, not proof of unpredictability or NIST validation.
For an entropy source, report the raw symbol definition, collection conditions, IID or non-IID rationale, min-entropy bound, restart result, conditioning calculation, health-test design and tool version. Then state plainly what remains outside scope: physical modelling, adversarial analysis, integration review or formal validation.
The final rule is simple: test the bytes, test the source, and test the path between them. Each catches a different class of failure.