NumPy as the oracle

Here is a question with a genuinely uncomfortable answer. You have just written the function that multiplies two matrices. It runs. It returns a grid of numbers. How do you know they are the right numbers?

For most software, this is not a hard question. If you write a function that formats a date, you look at the output and you can see whether it says 2026-09-04. If you write a shopping cart, you add two items and check that the total is the sum of the prices. The correct answer is something you can recognize.

Numerical code is not like that. A 64×64 matrix multiplication produces four thousand numbers, each of which is a sum of sixty-four products. They are all plausible. There is no glance you can take at [-0.3129, 1.8871, 0.4410, ...] that tells you whether the third one should have been 0.4410 or 0.4409 or -1.2. And unlike a date formatter, a subtly wrong matrix multiply does not crash or produce garbage. It produces numbers that look completely reasonable and are quietly, slightly wrong.

That is the central problem of building a machine learning library, and it gets worse as you go up the stack. A tensor library with a subtle bug does not fail loudly. It trains a neural network that reaches 85% accuracy instead of 97%, and you will spend three weeks assuming you chose bad hyperparameters.

#The way out is to not be the source of truth

You cannot verify these numbers by inspection, and writing a second implementation to check the first only means you now have two implementations that might share the same misunderstanding.

But you do not have to be the source of truth. Somebody else already is.

NumPy is a numerical library for Python that has been in continuous use since 2006, is a dependency of essentially all scientific computing done in Python, and has had its arithmetic checked by millions of users over two decades. When NumPy says that a particular matrix multiplication produces a particular number, that number is right. Not axiomatically right, but right in the way that matters: any disagreement between Ferro and NumPy is overwhelmingly likely to be Ferro's fault.

So Ferro does not try to verify its own arithmetic. It compares against NumPy's. This technique has a name in testing literature — differential testing against a reference implementation, sometimes called an oracle — and for numerical work it is by far the highest-value testing you can do per hour invested.

#How it actually works

There is a Python script, tools/gen_fixtures.py. It uses NumPy to build input arrays and compute expected outputs, then writes both to disk in NumPy's own .npy file format. Those files are called fixtures, and they get committed to the repository like any other source file.

On the Rust side, tests load the fixtures and compare:

let expected = fixture("ops/permute_narrow_contiguous_f32.npy");
assert_eq!(expected.shape(), &[2, 2, 3]);
Tolerance::EXACT.assert_matches(&my_result, &expected);

Ferro starts with 23 of these, covering three categories. Some exercise the reader itself — every numeric type, both memory layouts, empty arrays, and the strange floating-point values. Some are reference tables for mathematical functions with no closed form. And some are the expectations for the tensor operations Ferro has not implemented yet, written down first so that each operation starts with a target instead of inventing one as it goes.

#Why the files are committed rather than generated

There is an obvious alternative: run the Python script as part of the test suite. It would mean fewer files in the repository and no possibility of the data going stale.

Ferro commits them instead, for one reason. Committed fixtures mean that running Ferro's tests requires only Rust. No Python, no NumPy, no SciPy, no virtual environment, no version pinning across two ecosystems. Someone who wants to build the project and run the tests needs one toolchain. The Python dependency exists only for the person changing the reference data, which is a much smaller group and a much rarer event.

This matters more than it sounds. A test suite that needs two language ecosystems configured correctly is a test suite that people stop running.

#The cost of committing them, and the fix

Committing generated data creates a specific failure mode: the data can drift out of sync with the generator. Someone edits the Python script — adds a fixture, fixes an input, changes a shape — and forgets to regenerate and commit the output. Now the Rust tests are asserting against the old expectations while the script describes the new ones. Everything passes. The tests are measuring nothing.

The fix is a check that catches exactly this:

cargo xtask check-fixtures

It regenerates every fixture into a temporary directory and compares byte for byte against what is committed. If a single byte differs, or a file is missing, or there is an extra file that the generator no longer produces, it fails and says which:

committed fixtures do not match the generator:
  stale: ops/gather_axis1_f32.npy

run `cargo xtask gen-fixtures` and commit the result

This runs as its own job in continuous integration — the only job that needs Python, kept separate precisely so the main test job does not.

#Testing the test

A drift check that cannot fail is worse than no drift check, because it produces confidence rather than merely lacking it. So before it was trusted, it had to be seen failing.

The method was crude: pick a fixture, overwrite four bytes in the middle of it, and see what happens. The first attempt was instructive in an embarrassing way. The four bytes chosen happened to land on an element whose value was already zero, and the bytes written were zeros. Nothing changed, the check passed, and for a moment it looked like the check was broken.

It was the test of the test that was broken. Writing genuinely different bytes produced the right outcome twice over. The drift check failed and named the file. And the Rust test that consumes that fixture also failed, with this:

1 of 8 elements differ (rtol 0e0, atol 0e0):
  [1] got 4.00000000000000000e0, expected 3.40282346638528860e38
      (abs 3.403e38, rel 1.000e0)

Which is the report the comparison helper was written to produce: how many elements are wrong out of how many, which one first, both values, and the error in absolute and relative terms. Restoring the file made both pass again.

That whole exercise took about ten minutes and it is the reason the drift check can be believed.

#Two rules about what may become a fixture

Everything must be deterministic. Any fixture involving randomness draws from one explicitly seeded generator and nothing else. If regenerating the fixtures produced different bytes each time, the drift check would fail constantly and would be turned off within a week — and then the staleness problem it exists to prevent would be back, silently.

Never make a fixture out of a random number generator. This one is less obvious and more important. Ferro has its own pseudorandom generator, and it will never produce the same bit patterns as NumPy's, because it is a different algorithm with a different internal state layout. That is not a bug in either one; there is no such thing as "the" correct sequence of random numbers.

A fixture comparing Ferro's random samples against NumPy's cannot ever pass. Write it anyway and it becomes a permanently failing test, which gets marked as ignored, which trains everyone to ignore failures in that file, which is how a real bug eventually hides there. Random number generation gets tested a different way: statistical properties (does the normal distribution have the right mean and variance?) and determinism (does the same seed produce the same sequence twice?). Both are real tests. Neither is a fixture.

The rule generalizes. A test that cannot pass is not a strict test, it is a disabled one with extra steps.

#What this buys

Twenty-three fixtures and roughly two hundred lines of Python, in exchange for the ability to write a new tensor operation and know within seconds whether it is correct. Not "does it run" — correct, to the last representable digit, against an implementation with two decades of scrutiny behind it.

The next post is about the other half of the mechanism: actually reading NumPy's file format from Rust, which turned out to be more interesting than expected.