Your software calculates something the business depends on: a payslip, a quote, a tax, a simulator on your website. The tests are green. But who decided what the right answer was in those tests? Often, nobody. Someone copied what the program produced on the day it "looked about right". That is exactly what a test oracle is there to fix.
I am writing about it because this is what we did at Moonworkers, my payroll software company for small UK businesses, and because I apply the same idea to my content machines. Here is how I go about it, including what the method does not see.
01What a test oracle is, in one sentence
The ISTQB, the body that certifies software testers, gives the shortest definition in its glossary: a test oracle is a source to determine an expected result, which you then compare with what the system under test actually produces.
The word suggests a fortune teller. It means something very down to earth. In a short video on the Test Automation TV channel, the author uses the example of a loan calculator: the requirements document says that 1,000 lent at 10% over one year gives 100 in interest. That document is the oracle. If the software says anything else, the test fails. He also points out that an oracle can be code, a text, a mathematical model or a specialist's judgement.
Researchers have a name for the hard part: the oracle problem. A review published on arXiv by Rui Li and co-authors puts it like this: you need a systematic mechanism to check the result of each test case, and without it many testing techniques lose their value. Generating a thousand cases is pointless if nobody can say which ones are right.
02Where to find an oracle when nobody has the right answer
For a business calculation, you rarely have a ready-made table of answers. These are the sources I use, from the most reliable to the most fragile.
The official text
For a regulated calculation, the reference exists: the law, the rate tables, the tax authority's manual. You build a separate implementation from it, written by someone else or at least without looking at the production code, and compare the two. This is the strongest kind of oracle, because it does not share the errors of the software under test.
The reference tool
When the authority or a public body publishes a calculator, it becomes a practical oracle. You enter the same cases on both sides and compare. It is slower, often manual, but it is a genuinely independent second opinion.
The previous version
For a rebuild, the current version acts as the oracle for the new one: same inputs, same expected outputs. Be careful, it passes on its errors. I use it to spot differences, then settle each one against the text.
Properties of the calculation
When you have no exact answer at all, you can still check relationships. That is the idea behind metamorphic testing, described in the same arXiv review: the shortest distance from a to b must equal the shortest distance from b to a. For payroll, a rise in gross pay should never lower net pay, and a contribution should not appear below its threshold.
03What an oracle found in our contribution calculations
At Moonworkers, an oracle written from HMRC's texts replayed 896 National Insurance cases. It found four errors of one penny. Every one of them came from a rounding rule.
A penny is not much. On a payslip, it is a wrong line, and in payroll software an error like that never stays isolated: it repeats for every employee, every month. Nobody would have spotted it by eye. The existing tests did not catch it either, since they expected whatever the code already returned.
Yet the texts are precise. HMRC's manual on the exact percentage method requires employee and employer contributions to be calculated separately, each rounded to the nearest penny, with amounts of less than £0.005 disregarded. One sentence like that, translated slightly wrongly into code, is enough to create a one-penny difference on some of the cases.
We applied the same method to the free calculators on the website, rebuilt in May 2026: each one was checked, one by one, against the official tools before going live. It is slow, on purpose.
04Building an oracle step by step
This is the method I would follow today on any calculation that involves money.
1. Write the rules from the source
List each rule with its exact reference: article, page, paragraph. Thresholds, rates, order of operations, rounding. For UK contributions, the government page on rates and thresholds for employers 2026 to 2027 gives, for example, a primary threshold of £242 a week and an employee rate of 8%, applying from 6 April 2026 to 5 April 2027. Note that period: the oracle needs to know which tax year it is checking.
2. Implement the oracle separately
As simply as possible, readable by someone who knows the business, with no optimisation. A spreadsheet can be enough to start with. What matters is that it does not reuse the production code.
3. Choose the cases
Ordinary cases, and above all the edges: just under a threshold, exactly on it, just over, zero, a very large amount, half pennies. Edges are where rounding and "less than or equal to" comparisons go wrong.
4. Compare and settle each difference
A difference does not tell you who is wrong. I go back to the text every time. Sometimes the oracle is the one that got it wrong, and that is good news: the rule was ambiguous and you have just found out.
5. Run it on every release
Once the differences are settled, the cases become regression tests, run on every deployment. It is the same logic as acceptance testing before go-live: you check before your users do it for you.
05When the oracle is wrong too
An oracle is not the truth. It is a second reading of the text, and it can be wrong. The Test Automation TV video says it well: an incomplete oracle lets errors through, and a wrong one flags correct results as errors. If the same person writes the code and the oracle with the same misreading, the two will agree, and be wrong together.
It also has a cost. It has to be updated every time a rule changes, or it becomes a source of errors itself. And it only covers the cases you gave it: several hundred cases is a lot, but it is not everything.
I have learnt to keep an eye on the check itself. In my content machines, deliberately bad test articles must always come out red at the quality gate. If one of them turns green after a change to the rules, the article has not improved: the check has regressed. An oracle deserves the same suspicion.
Finally, for a simple and stable calculation, a table of twenty cases checked by hand is often enough. I would not build a full oracle for a calculation that never changes and that nobody relies on.
06Where to start with your own calculation
Pick the calculation that would hurt most if it were wrong. Find the text that defines it and write its rules on one page, with references. Choose ten cases, five of them on the edges. Work out the expected answer without going through your software, then compare.
If you find a difference in that first batch, you know where you stand. If you find none, keep those ten cases: they are your first regression tests. And if the rule itself is unclear, now is the time to write it down as a specification before going further.
This is the kind of work I do on short engagements, hands-on with the team, if you would like to talk about it.