Evaluating labelling-challenge submissions with CheckMAITE

Left: real drone-swarm footage. Right: 16 synthetic clips built from it, with drones cut from the footage and procedural birds. Because a program placed every object, the ground truth is exact (with visibility), and every clip records its generator settings: size, speed, blur, noise, background.

1 · Four simulated submissions and the classic score

TeamMOTAID switchesClass accuracyDrone recallBird recall
A · careful0.97801.00.9790.976
B · fragments tracks0.8421991.00.9010.863
C · misses small/fast, flips class0.86700.9640.8760.855
Auto · fine-tune + ByteTrack0.34340.9870.870.017
B and C score almost the same MOTA, but they fail in completely different ways. Auto finds drones as well as the humans, but labels about a third of the birds as drones.
White corners = ground truth (empty corners = missed). Colour = track ID (a colour change = ID switch). U/B = class; red = wrong class.

2 · Same score, different failures: DataevalVideoQuality

LabelsTracksBoxesMean track length (frames)Mean gap fractionTracks that switch classTrack outliers
Ground truth19112236640.5%010
A · careful19011968632.2%05
B · fragments tracks31310775343.3%3343
C · misses small/fast, flips class191106295612.5%1924
Auto · fine-tune + ByteTrack18286244710.1%07
B: 122 extra track IDs and tracks half as long, because it restarts objects with a new ID after they pass behind the ridge; 43 of its tracks are outliers. C: about 1,600 fewer boxes than the ground truth, and its tracks are missing about 12% of their span, from skipped stretches; 19 tracks switch between drone and bird. B's 33 class switches are a side effect of ID swaps between a drone and a bird. Auto never switches class because it only outputs "uav".
Finding about the footage (same for every submission)Value
Videos / frames16 / 1536
Frames that are near-duplicates of a frame in another clip1152 of 1536
Clips that use the same stretch of footagedrone09 & drone11, drone09 & drone14
Frames that barely change from the frame before87%
Outlying frames (brightness, contrast, sharpness)50
VideoQuality also checks the footage itself. It spots that most clips reuse the same source footage, and that some clips use exactly the same stretch of background. That's correct (every clip is cut from one 16-second video), and it's exactly the check a challenge organiser wants before releasing data.

3 · Which boxes are wrong, and why: label errors and factor insights

Label errors: no ground truth needed

Labels checkedBoxesFlagged as likely wrong classReal class flipsReal flips caughtFalse alarms
Team C26551139191 (drone→bird 64, bird→drone 27)22
Ground truth (control)3046260–26
Each box is cut out and embedded; a box whose embedding sits closer to the other class than to its own is flagged. On Team C's labels alone it catches all 91 class flips, mostly small drones labelled "bird", with 22 false alarms. On the ground truth it flags 26 of 3046 boxes, all false alarms. So treat flags as a review list, not a verdict. Runs in about 35 seconds.

Factor insights: which generator settings explain each team's mistakes

TeamBoxes wrong (of 3046)Kind of errorStrongest factor
A · careful6363 missedvisibility (0.25)
B · fragments tracks432380 missed, 52 new/swapped IDvisibility (0.09)
C · misses small/fast, flips class487395 missed, 91 wrong class, 2 new/swapped IDspeed (0.12)
Auto · fine-tune + ByteTrack14771454 missed, 21 wrong class, 2 new/swapped IDclass (0.66)
This part uses the ground truth to find each team's wrong boxes, then the synthetic metadata to explain them. Scores run from 0 (no link) to 1 (the factor alone separates right from wrong). A's few misses go with low visibility. B's go with occlusion (low visibility). C's go with speed and box size. Auto's go with class: it doesn't label birds. Caveat: numeric factors were grouped into 5 bands, because otherwise DataEval treats them as categories and spurious factors win (an issue to raise on !695).

4 · How hard is the class distinction? Feasibility

LabelsBoxes labelledBoxes scored (≥ 8 px)Boxes the nearest-neighbour check confusesBayes error rate (lower–upper)AP ceiling (uAP)
Ground truth30462914210.5–0.7%0.999
Each box is cut out and embedded (the same ImageNet EfficientNet-B0 as section 3), and a nearest-neighbour vote estimates how often any classifier would confuse drone and bird. It confuses 21 of 2914 boxes: the classes are almost perfectly separable, so confusing them is a labelling failure, not an ambiguous task. So the drone/bird flips in section 3 are the labeller's fault.
Caveats: 42% of boxes are small (the report warns above 10%), and boxes under 8 px are skipped. About 9 minutes per run on CPU.