CheckMAITE labelling demo
Evaluating labelling-challenge submissions with CheckMAITE
Left: real drone-swarm footage. Right: 16 synthetic clips built from it, with drones cut from the footage and procedural birds. Because a program placed every object, the ground truth is exact (with visibility), and every clip records its generator settings: size, speed, blur, noise, background.
1 · Four simulated submissions and the classic score
| Team | MOTA | ID switches | Class accuracy | Drone recall | Bird recall |
|---|
| A · careful | 0.978 | 0 | 1.0 | 0.979 | 0.976 |
| B · fragments tracks | 0.842 | 199 | 1.0 | 0.901 | 0.863 |
| C · misses small/fast, flips class | 0.867 | 0 | 0.964 | 0.876 | 0.855 |
| Auto · fine-tune + ByteTrack | 0.343 | 4 | 0.987 | 0.87 | 0.017 |
B and C score almost the same MOTA, but they fail in completely different ways. Auto finds drones as well as the humans, but labels about a third of the birds as drones.
White corners = ground truth (empty corners = missed). Colour = track ID (a colour change = ID switch). U/B = class; red = wrong class.
2 · Same score, different failures: DataevalVideoQuality
| Labels | Tracks | Boxes | Mean track length (frames) | Mean gap fraction | Tracks that switch class | Track outliers |
|---|
| Ground truth | 191 | 12236 | 64 | 0.5% | 0 | 10 |
| A · careful | 190 | 11968 | 63 | 2.2% | 0 | 5 |
| B · fragments tracks | 313 | 10775 | 34 | 3.3% | 33 | 43 |
| C · misses small/fast, flips class | 191 | 10629 | 56 | 12.5% | 19 | 24 |
| Auto · fine-tune + ByteTrack | 182 | 8624 | 47 | 10.1% | 0 | 7 |
B: 122 extra track IDs and tracks half as long, because it restarts objects with a new ID after they pass behind the ridge; 43 of its tracks are outliers. C: about 1,600 fewer boxes than the ground truth, and its tracks are missing about 12% of their span, from skipped stretches; 19 tracks switch between drone and bird. B's 33 class switches are a side effect of ID swaps between a drone and a bird. Auto never switches class because it only outputs "uav".
| Finding about the footage (same for every submission) | Value |
|---|
| Videos / frames | 16 / 1536 |
| Frames that are near-duplicates of a frame in another clip | 1152 of 1536 |
| Clips that use the same stretch of footage | drone09 & drone11, drone09 & drone14 |
| Frames that barely change from the frame before | 87% |
| Outlying frames (brightness, contrast, sharpness) | 50 |
VideoQuality also checks the footage itself. It spots that most clips reuse the same source footage, and that some clips use exactly the same stretch of background. That's correct (every clip is cut from one 16-second video), and it's exactly the check a challenge organiser wants before releasing data.
3 · Which boxes are wrong, and why: label errors and factor insights
Label errors: no ground truth needed
| Labels checked | Boxes | Flagged as likely wrong class | Real class flips | Real flips caught | False alarms |
|---|
| Team C | 2655 | 113 | 91 | 91 (drone→bird 64, bird→drone 27) | 22 |
| Ground truth (control) | 3046 | 26 | 0 | – | 26 |
Each box is cut out and embedded; a box whose embedding sits closer to the other class than to its own is flagged. On Team C's labels alone it catches all 91 class flips, mostly small drones labelled "bird", with 22 false alarms. On the ground truth it flags 26 of 3046 boxes, all false alarms. So treat flags as a review list, not a verdict. Runs in about 35 seconds.
Factor insights: which generator settings explain each team's mistakes
| Team | Boxes wrong (of 3046) | Kind of error | Strongest factor |
|---|
| A · careful | 63 | 63 missed | visibility (0.25) |
| B · fragments tracks | 432 | 380 missed, 52 new/swapped ID | visibility (0.09) |
| C · misses small/fast, flips class | 487 | 395 missed, 91 wrong class, 2 new/swapped ID | speed (0.12) |
| Auto · fine-tune + ByteTrack | 1477 | 1454 missed, 21 wrong class, 2 new/swapped ID | class (0.66) |
This part uses the ground truth to find each team's wrong boxes, then the synthetic metadata to explain them. Scores run from 0 (no link) to 1 (the factor alone separates right from wrong). A's few misses go with low visibility. B's go with occlusion (low visibility). C's go with speed and box size. Auto's go with class: it doesn't label birds. Caveat: numeric factors were grouped into 5 bands, because otherwise DataEval treats them as categories and spurious factors win (an issue to raise on !695).
4 · How hard is the class distinction? Feasibility
| Labels | Boxes labelled | Boxes scored (≥ 8 px) | Boxes the nearest-neighbour check confuses | Bayes error rate (lower–upper) | AP ceiling (uAP) |
|---|
| Ground truth | 3046 | 2914 | 21 | 0.5–0.7% | 0.999 |
Each box is cut out and embedded (the same ImageNet EfficientNet-B0 as section 3), and a nearest-neighbour vote estimates how often any classifier would confuse drone and bird. It confuses 21 of 2914 boxes: the classes are almost perfectly separable, so confusing them is a labelling failure, not an ambiguous task. So the drone/bird flips in section 3 are the labeller's fault.
Caveats: 42% of boxes are small (the report warns above 10%), and boxes under 8 px are skipped. About 9 minutes per run on CPU.