One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble
LENORE MARIE MULLIN, Gaetan Hains
Abstract
companion MPI-extension paper validated the first real, communication-bearing ρmachine(d)
ensemble: one NVIDIA A100 partition and one AMD-class CPU partition of a single NCSA Delta
node, under one genuine Open MPI job, reassembled by a real MPI Gatherv. That result is still
single-vendor on the accelerator side — one GPU architecture, one NUMA-local CPU. This paper
merges that result with the heterogeneous multi-machine paper’s multi-vendor, no-communication
partition (NVIDIA, AMD, and Intel devices, split across institutions with no shared interconnect)
into one theoretical and experimental design: a single Slurm heterogeneous job spanning Delta’s
gpuA100x4 and gpuMI100x8 partitions, placing NVIDIA A100, AMD MI100, and CPU ranks under
one real MPI communicator for the first time, with Intel’s Data Center GPU Max 1550 (TACC
Stampede3) retained as the no-communication companion case its own institution’s lack of shared
interconnect still requires. We give the theoretical extension in full — the generalized wi ∝1/ti
weight split for three concurrent realizations, the admissibility of a Slurm het-job specification as a
Cd-class directive, and the hardware-placement mechanism itself — report the first real results —
a matching-shape AMD MI100 td; one working nine-rank communicator (4 A100, 4 CPU, 1 MI100)
across both het-job components, with its measured per-rank times and Gatherv cost; and an in-
dependent fp64 correctness check — and state plainly what the run exposed: the wi ∝1/ti split
assumes time scales linearly with rows, and that assumption fails for the MI100 rank (6 rows took
597 ms against a linear prediction of 1.6 ms), so the slowest device sets the ensemble’s time by a
factor of about 90. We then give a launch-geometry model and record its predictions before measur-
ing them. The A100 test succeeds: its time is flat from 128 to 3456 query rows and steps at exactly
3520 and 6976 rows, as predicted. The MI100 shows a floor of about 600 ms; floor-decomposition
runs place it in the key loop, and changing the kernel’s launch settings (64 workers per team, keys
split over 16 teams) cut it to 14.6 ms and the energy per call at 6 rows by 32×, though the MI100
remains about 95×slower than an A100 at 8192 rows. For the original configuration the measured
curves show that the CPU ranks (6.2 ms per-call floor) and the MI100 rank (about 600 ms) should
receive no rows at this problem size, while the four A100 ranks finish in about 2.2 ms; the tuned
MI100 would still receive none. we then ran that split as an ensemble and measured 2.21 ms (slow-
est compute plus Gatherv: 4.7–5.0 ms in two runs and 9.1 ms in a third, against 9.2–9.7 ms for the
1/t split). A controlled set of reruns also shows that the companion paper’s A100 result (8.1×the
linear prediction) and its ≈22 ms Gatherv were launch artifacts: under mpirun with Open MPI’s
default transport all four GPU ranks shared one A100 (Section 7). Finally we derive from the