Validation plan - 16 May 2026

Dataset validation plan

This is the public plan for eventually proving fire detection without hiding behind random frame splits, visible-flame shortcuts, or unmeasured false positives. No validation runs have happened yet.

Read the validation stack Read mission requirements Read detector path Read optical baseline Read detection pipeline Read calibration strategy Read survey model Read field-test plan

Evidence base

The plan is grounded in the public FLAME sequence and the current EmberScope mission requirements.

FLAME Pile-burn RGB and thermal-palette data with classification, segmentation, and a documented false-positive imbalance warning.

FLAME 2 Side-by-side RGB/IR prescribed-fire frames with expert fire/no-fire and smoke labels plus burn context records.

FLAME 3 Synchronized RGB and radiometric thermal TIFF imagery from multiple prescribed fires, with nadir thermal plots.

Stress tests Dataset choice is tied to small-target radiometry, false-positive rural scenes, and survey operations.

Validation principle

EmberScope should be tested as a radiometric survey payload, not as a generic fire-picture classifier.

A useful benchmark has to show detection of weak or small hot targets at the drone GSD and dwell time, rejection of rural no-fire clutter, and survival of held-out burns, sites, days, and sensor paths.

The plan keeps RGB-only, thermal-only, RGB/thermal, and RGB/radiometric-TIFF scores separate so visible smoke or flame cues cannot disguise weak thermal performance.

Validation stack

The headline split must hold out burns and backgrounds, not just shuffled frames.

Validation layer	What it tests	Required output
Burn-held-out public data	Generalization across complete fire events rather than adjacent video frames.	Precision, recall, specificity, sensitivity, F1, ROC/PR data, and confusion matrix by modality.
No-fire stress set	False alarms from sun-heated clutter, vehicles, people, structures, smoke, shadows, water, and residual heat.	False positives per flight minute and per surveyed hectare, with representative rejected examples.
Radiometric TIFF / raw thermal path	Whether the detector chain works from temperature-like data rather than palette color alone.	Thermal-only, RGB/thermal, and RGB/radiometric-TIFF results reported separately.
Small-hot-target simulation	EmberScope's centimetre-scale target after GSD, dwell, blur, noise, and calibration error are known.	Detection curves versus target size, target radiance or temperature, background, altitude, and threshold.
Local EmberScope field data	Same detector, calibration kit, optics, geotagging, and survey profile as the payload under test.	Field surrogate results and failed-case packet ready for engineering review.

False-positive policy

The negative set has to look like rural fire-service operating terrain, not a clean lab background.

Hot clutter

Sun-heated rocks, bare ground, roads, rooftops, metal gates, vehicles, and machinery.

Warm non-fire objects

People, livestock, buildings, camp equipment, and other warm objects that operators must not chase as fire.

Atmospheric and optical confusion

Smoke, dust, haze, shadows, clouds, water, reflective surfaces, and RGB/thermal alignment offsets.

Operational negatives

Pre-burn, post-burn, and same-terrain no-fire flights at comparable altitude, speed, time of day, and solar loading.

Acceptance gate

No detection claim should ship without provenance, held-out data, and failed examples.

A credible first report needs a burn-held-out public score, a no-fire stress score, a radiometric-TIFF or raw-radiometry score, an EmberScope-specific small-hot-target simulation, and a packet of missed detections and false positives for review.

Every run should record dataset source, version, checksum, split definition, model or rule version, threshold, detector assumptions, calibration inputs, GSD, altitude, frame rate, and reviewer label provenance. Raw datasets and bulky experiment outputs should stay outside routine public materials unless intentionally packaged for review.