How Residue was tested, and what the numbers mean

01The objects

Forty identical parts, built in software to the physics of the real ones. Each object is a patch of surface, 512 pixels square at 10 microns per pixel, about 5 mm across, made of bumps at six sizes from 40 microns to 480 microns. The bump heights follow the same statistics real machined and moulded surfaces show.

Two things make the simulated objects realistic:

• They share a pattern. Half of every object's surface is a component all forty share, standing in for the mould, the lathe or the build plate. Only the other half is the object's own. A matcher that cannot remove the shared half confuses every object with every other.
• They wear in three dimensions. A real surface does not blur when it wears. Material is removed and what was underneath is exposed. The model treats the surface as a volume: removing d microns replaces fine bumps faster than coarse ones. The rate is set by one number, kappa, how deep a bump runs compared with how wide it is. Kappa is swept across the whole published range, so the result does not depend on one value.

The model is in pebbleid/surface.py in the private project and is restated in the browser inside every demo.

02The test

Every object is photographed once under clean conditions. That photo is its record, the same thing a plant would capture at enrolment. Then every object is photographed three more times, each with its own fresh wear, dust film, camera noise and a random light direction. Those are the reads.

Each read is shown to a matcher together with all forty records and the matcher is asked: which one is this? There are 40 objects and 3 reads each, so 120 answers per method per condition. Each read is also scored against the 39 records it does not belong to, which gives 4,680 "wrong pair" comparisons per method per condition. Those wrong pairs are what tell you how often a method would confuse two different objects.

One read is one 5 mm tile. A real camera view of a part covers about 78 such tiles, so the single-tile result is the hard version of the test.

03The 13 conditions

One condition would prove nothing, because we chose the difficulty. So the test is a grid. A centre point is held fixed and one thing is changed at a time, then one cell turns everything bad at once.

Twelve one-axis cells plus the corner is 13. Every method runs in every cell on identical data.

04The 28 methods

"28 methods" means 28 matcher variants ran the same test. They fall into five families. All but the last two families see a single photo under one oblique light at a random direction, which is what a competitor without three-light hardware has.

The PCA rows matter most. PCA is a textbook technique anyone can implement in an afternoon. Giving it our three-light photos and our band filters asks the fair question: once a rival has the hardware, is our matcher worth anything on top?

05Reading a score

d′, the separation

Each method gives a number for how alike two pictures are. Look at the numbers it gives for same-object pairs and for different-object pairs. d′ is the gap between those two averages, divided by their spread. Under 1 means the two piles overlap and the method is near a coin flip. In the tens, the piles are far apart. In the hundreds, the model is saying the margin is enormous, not that anyone would measure that exact figure.

Rank-1 errors

Simpler: of 120 reads, how many times was the top-scoring record the wrong object? Where a method makes zero errors we do not report zero. With 120 trials the true rate could still be as high as 3 in 120, so 2.5% is what is shown. That is the rule of three, and it is the right way to report a clean run.

Confidence intervals

Every d′ carries a 95% interval from a bootstrap that resamples whole objects, not individual reads, because three reads of one object are not independent of each other.

A statistic dropped during the run

Dividing the gap only by the spread of the different-object scores makes keypoint methods look superb: SIFT scores 26.7 that way while naming the wrong object 87% of the time, because almost all of its different-object scores are exactly zero. That number is kept in the full tables and nothing is ranked on it.

06Who wins where

Three findings, in the order we learned them.

1. The keypoint tools fail. ORB, SIFT, AKAZE and BRISK name the wrong object 8 or 9 times in 10 on nominally identical parts. Rank-1 accuracy at the centre: ORB 8%, SIFT 13%. Their d′ sits at 0.4. This is the toolkit the published literature leans on.
2. Our first matcher lost to PCA. Given the same three-light photos, the textbook PCA method beat our v1 in 10 of 13 conditions, by a small margin at the centre (d′ 41.9 against 40.9) and by a large one under heavy dust (24.2 against 19.5). In the corner cell v1 named 56 of 120 wrong; PCA named 32 wrong. A benchmark that only confirmed what we hoped would be worthless; this one sent us back to work.
3. v2 fixed the reason. v1 gave every scale of texture the same vote. Fine texture is reliable and coarse texture is almost useless, so equal votes watered down the good signal. v2 weights the bands by their measured separation and corrects for dust dimming the picture. Result: 0 of 120 wrong in every condition, including the corner, and d′ about 10 times higher than v1 at the centre.

d′ per 5 mm tile. One real camera view composes about 78 tiles.

The size of the lead. If the PCA rival is also given our band weighting, v2 is 10% to 30% ahead, and level in 3 of 13 conditions. That is the matcher's margin on its own, and it is why the moat is the system around the matcher: the capture rig, closed-inventory tracking, scan-again-never-guess, and the error budget.

07Ablation, what each ingredient is worth

"Ablation" means removing one part of the design at a time and rerunning the test, so you learn which part bought which share of the result. Our pipeline is three moves on the image, plus the decision rule. Here is each move taken out, at the centre condition.

Ratios are paired bootstrap intervals: the same objects go through both versions.

The third row is the one to notice. An ablation that only ever says "every part helps" is suspect. This one found a part that hurt, and the fix for it became v2.

08The worst-case physics

Kappa is the one input no literature gives us. So a second run holds everything else at the centre and sweeps kappa down to the pessimistic end while material is removed, using a full camera view (78 tiles) and realistic sensor noise: photon shot noise, read noise, 10-bit quantisation, a random sub-pixel shift on every read. The bar is d′ 9.8, which is what under one false match a year requires in the pebble-reactor case, the hardest application we have sized.

d′ per view, 24 objects per cell, all bands. Fine features die first: the 40 µm band alone falls from 2,556 to 153 over this sweep while the 160 µm band falls from 924 to 275.

Two corrections were made during this run and are recorded, because a test that never found a bug in itself is not being read carefully. A band-pass filter nominally tuned to 40 µm was mostly reporting 600 µm content, which made fine scales look far more wear-resistant than they are. And any shared pattern the subtraction failed to remove was being credited to the genuine score; separation is now measured from the impostor mean, not from zero.

What d′ in the hundreds means. The margin, not a headline accuracy. In a running system the binding event is a retaken capture, not a false match, which is why every demo shows its retake rate.

09Stress tests, the ways a read can go wrong

The grid varies the surface. A third run varies the capture, applied to the read only while enrolment stays clean, since that is what a plant sees.

• Exposure at 1.1, 1.2 and 1.4 times nominal, with the sensor clipping at 1.5 times the mean grey. 1.4 already saturates 95% of the window, which is why the severities were brought down from a first guess of 1.2 to 1.7.
• Dark frames, contrast loss, drift in the lamp brightness between enrolment and read.
• A thick dust patch that shows up as fake texture rather than as a blank, because each light sees the dust differently.
• Geometry: rotation, scale change and shifts up to 216 px, where only a quarter of the enrolled patch overlaps the read.
• Scratches and swaps in the market demos: a swapped unit, a never-enrolled unit, a re-machined face, a damaged ring.

Two mechanisms came out of it. A quality detector on the capture side, blind to the truth, flags saturated, dark and abnormally textured pixels, and the read is taken again when too much of the window is flagged. On clean reads it flags 0.00% of the window, so it does not send good captures back. And a registration step that uses the shared batch pattern as a landmark recovers rotation to about 0.01 degrees, scale to 0.1% and shift to 0.05 px, including the 216 px case, in about half a second per read.

The one inherited shortcut is stated in the log: the benchmark estimates the shared pattern while excluding the read's true object, which a plant cannot know. The effect shrinks as the batch grows and is listed as an open item rather than hidden.

10Reported to the industry standard

For any system that answers "is this the same one?", the standard is ISO/IEC 19795, biometric performance testing and reporting. It asks for the false-match rate and the false-non-match rate at a stated threshold, on data you did not tune on, with confidence intervals, and ideally run by someone who is not the vendor. The gold example is NIST's face-recognition vendor test: vendors submit, NIST runs it, everyone sees the same curve. In surface fingerprinting nobody publishes to that standard; the incumbent's public figure is a marketing line ("99.9%, zero false positives") with no test behind it that a reader can check.

Every rate on this site is reported in the standard's vocabulary and its arithmetic: benchmarked on a physical model, reported in ISO/IEC 19795 terms, reproducible by anyone who runs the command.