Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree
Hongsen Cao, Mona Jaber, Shanxin Yuan, Ahmed Sayed
Abstract
Multi-modal fusion for 3D object detection assumes that cameras and LiDAR are broadly complementary. In
practice, how much each modality is trusted at a given location
is decided by learned weights that are neither characterized
nor auditable. We label every ground-truth object by which
modalities detected it, then mine the resulting 0.4M-object
database across nuScenes, KITTI, and Waymo. Because the
patterns shift with distance from the ego vehicle, we read the
mined model through Spatially Stratified Feature Attribution
(SSFA), which conditions Shapley attribution on distance zones
and exposes structure that global attribution averages away.
Three assumptions built into the current fusion design do not
hold. The modality-preference boundary tracks the object itself:
a traffic cone is nearly six times as likely as a truck to be seen
by the camera alone (16.8% vs. 2.9%), regardless of range.
Camera-only detections peak at 10–30 m, inside the band usually ceded to LiDAR, and fall away on either side. The drivers of
failure are not additive either: the distance×density interaction
alone outweighs the entire main effect of viewing direction.
We further find that modality dominance is directionally nonstationary, with objects behind the vehicle being markedly
more LiDAR-dominant than those in front. We compile the
mined structure into a Modality Reliability Map and inject
it into four pretrained fusion detectors through a lightweight
attention layer. This improves performance by up to 1.3 mean
Average Precision (mAP) while fine-tuning under 4% of their
parameters in 2–3 GPU-hours. Our work lifts reliability out of
the weights and treats it as a quantity that can be examined
before deployment. That is what a safety measure for multimodal perception must rest on.