Does synthetic pre-training pay off on a microcontroller? Accuracy, latency and energy across two MCUs
Achintya Yadav
Abstract
Rendered images are cheap to label, but a microcontroller classifier must survive 8-bit integer (INT8) quantisation and compute correctly on the chip. Five models, from MobileNetV2 at 224 px to a 61K-parameter convolutional neural network (CNN), were fine-tuned on a six-class scene dataset with and without a synthetic stage, against thresholds fixed in advance, and deployed on an ESP32-S3 and an RP2040. With full real data the stage gave no detectable gain. For the main deployable model the INT8 difference was +0.312 percentage points (pp), its interval below the +1.6 pp gain recomputed from the source study; quantisation did not detectably change it, nor did model size. With 10% of the real data the stage paid off for the CNN, by +3.438 pp on one seed and +2.715 pp over three; its cause is open. On the ESP32-S3 the main model took 152.7 ms and 16.3 mJ above idle per inference, and adding the stage changed neither beyond preset bounds. The model's INT8 outputs matched TensorFlow Lite's reference kernels exactly, but only after a memory overrun in a vendor kernel was patched. On the RP2040 the same model took 1.80 s and 27.3 mJ above idle, about twelve times slower, and the largest 96 px model exceeded the board's flash-capacity limit. Neither the RP2040 nor the host interpreter used for scoring reproduced the reference arithmetic. A synthetic stage is worth adding only when real data are scarce, and on-device INT8 outputs should be checked against reference kernels, not inferred from host scores.
Review reports
xPeerd (AI review)
AI Review (xPeerd — DAReview)
Summary
This manuscript investigates whether a synthetic pre-training stage improves scene-classification models that are small enough for microcontroller deployment after INT8 quantization. The study fine-tunes five models on a six-class dataset, compares training with and without a synthetic stage, and deploys key models on ESP32-S3 and RP2040 hardware. The central finding is narrow and carefully stated: with full real-data training, the synthetic stage does not produce a detectable gain for the main deployable model, while with only 10% real data the smallest CNN shows a replicated gain whose cause remains unresolved. The paper’s strengths are its preregistered design, explicit confirmatory/exploratory separation, sensitivity checks, and unusually strong attention to on-device verification, including the observation that “the ESP32-S3 computed exactly what the reference kernels compute once a vendor kernel was patched; the RP2040 and the host interpreter did not” . The main weaknesses are limits to external validity, some dependence of conclusions on recipe and fold choices, incomplete separation of confounded factors in the low-data gain, and several measurement/procedural gaps that the author openly acknowledges, including that “no capture measured the supply voltage” and one RP2040 meter check “was not done” . I do not see strong paper-mill or fabrication signals; if anything, the manuscript is unusually candid about nulls, departures, and unresolved technical issues. However, some inferential claims should be tightened where exploratory findings are discussed beside confirmatory ones.
Potential Major Revisions
- Clarify the scope of the main conclusion and reduce any risk of overgeneralization from a narrow case study.
The manuscript’s title asks, “Does synthetic pre-training pay off on a microcontroller?” yet the limitations section states plainly: “This is a case study: one dataset of six scene classes [4], one renderer, one recipe and one board of each kind. I make no claim about other tasks, newer image generators, or Cortex-M parts with DSP or vector extensions” . The conclusion is appropriately narrowed in several places, but the title and parts of the Discussion can still read more broadly than the evidence warrants. For example, the Discussion answers the title with “With all the real data and the frozen recipe, no” before later narrowing that answer . I think the work would be epistemically stronger if the abstract, title framing, and concluding claims aligned even more tightly with the actual design: one dataset, one synthetic source, one quantization pipeline, and two boards, one of which does not reproduce reference arithmetic reliably .
- Address the internal-validity concern created by tuning and decision use on folds that later serve as test folds.
The manuscript candidly states: “Three choices used data that later served as test folds” and further notes that “The synthetic-stage length and freezing were tuned on arm D on folds 0 to 2 with no matching budget for arm B” . Although the author argues that residual optimism would favor arm D and that the main row-3 result remained null when folds 0–2 were excluded, the same section also concedes that over folds 3–9 “the INT8 upper bound no longer excludes the +1.6 pp planning value” , echoed in the exploratory reanalysis where D−B becomes +0.128 pp with CI −1.455 to +1.710 . This does not invalidate the registered result, but it does mean the strongest inferential sentence about excluding the planning value depends on the full ten-fold analysis, including folds implicated in earlier choices. I recommend that the manuscript explicitly downgrade the strength of this exclusionary claim in the abstract and conclusion, and clearly mark it as sensitive to the full-fold design rather than a robust cross-validated exclusion.
- Better separate confirmatory findings from exploratory interpretation, especially around the low-data CNN gain.
The most interesting positive finding is the row-4 gain with 10% real data, but the paper itself states that “its cause is open” and that “the CNN’s 48 px input has not been separated from its size” . The Discussion says that “neither the weak Imagenette start alone nor the synthetic labels explain it” , but this wording risks sounding more resolved than the design supports. The same manuscript also says that only row 4 at 10% survives the Benjamini–Hochberg screen in exploratory analysis . I think the paper should more sharply distinguish: confirmatory null at full data; exploratory but replicated gain at row 4 with 10% data; mechanism unresolved due to confounding between model size and resolution, possible limited variety of synthetic scenes, and recipe dependence. Without that sharpening, readers may overread the low-data result as mechanistically explained when it is not.
- Strengthen the discussion of recipe sensitivity because it materially changes the sign of the main contrast.
The addendum reports that under heavier augmentation, “At row 3 the INT8 difference turned negative, −1.915 pp … against the registered +0.312 pp” and that “the contrast moved by −2.226 pp at row 3” . The author properly states, “The registered primary and secondary hold for the recipe they were run under” and that this is “a dependence on the recipe and not a correction” . Still, this is not a minor sensitivity; it is a substantive reversal in direction. I recommend the paper foreground this dependence earlier and more prominently in the abstract and main conclusion. The current framing could mislead a reader into thinking the main result is recipe-robust when the manuscript’s own evidence shows it is not.
- Revisit the adequacy and interpretation of statistical power for the primary question.
The methods define the primary test as a bounded exclusion of the +1.6 pp planning value rather than a test of zero difference , and the registered result does exclude +1.6 pp with CI −0.849 to +1.472 for the main row-3 INT8 contrast . However, the exploratory assessment notes that the minimum detectable effect for the primary was 1.62 pp, “so the primary was built to detect about the planning value and nothing smaller” . That is a crucial caveat. The paper should explain more directly that the design is informative about ruling out a source-study-sized benefit under this specific setup, but not about smaller practically relevant gains. As written, some readers may conflate “no detectable gain” with evidence of near-zero effect, which the paper does not actually establish.
- Tighten the hardware-measurement claims given acknowledged incompleteness and unresolved path dependence.
A notable strength is the insistence on reference-kernel verification, including the strong warning that “a quantised model’s accuracy on a board is something to check against the reference, not read off the host” . But hardware inference claims remain limited by acknowledged gaps. The manuscript states: “No capture measured the supply voltage,” the RP2040 meter check “was not done,” and “what inside its global average pool causes the departure was not tested” . It also notes that some repeated capture conditions were missed . Because the hardware contribution is important and somewhat distinctive, I think the paper should temper language around energy and cross-board comparisons, explicitly labeling some values as useful operational measurements rather than validated device-independent estimates.
- Expand the discussion of data leakage / overlap risk from ImageNet-related pretraining sources and dataset provenance.
The limitations state: “Some of the web photographs may also be in ImageNet, which I had no practical way to check. Imagenette is drawn from ImageNet, so at every row any overlap reaches arms B and D alike, though I cannot show that it cancels” . This is important because the entire comparison relies on transfer sources. Even if overlap affects both B and D similarly, the manuscript cannot fully establish that assumption. I recommend a more explicit discussion of how potential overlap could affect interpretation of both absolute performance and the marginal effect of the synthetic stage, especially at row 4 where Imagenette initialization is central.
- The paper should sharpen what exactly has been “reproduced” and what has not.
The manuscript repeatedly emphasizes exact matching to TensorFlow Lite reference kernels for the ESP32-S3 after patching, which is excellent. But it also states that the host interpreter used for scoring “is not the reference arithmetic either” and that the RP2040 path departs from the reference at global average pooling . Later it notes that rescoring all registered models on both paths moved major estimates by less than 0.16 pp and did not change interpretation . This should be synthesized more clearly. Right now, the reproduction story is technically rich but somewhat fragmented. I recommend one concise subsection or figure summarizing: host path agreement, device path agreement, where mismatches occur, and what inferential claims remain unaffected.
Potential Minor Revisions
- Tighten title and abstract wording for precision.
The title’s broad formulation could be narrowed or qualified. The abstract does eventually narrow the result, but “A synthetic stage is worth adding only when real data are scarce” may read stronger than the evidence supports, since the paper itself calls the cause open and the evidence comes from one small CNN under one low-data setting . Consider revising toward “may be worth adding” or explicitly restricting the claim to the observed row-4 setting.
- Improve consistency in how exploratory status is signposted.
The manuscript is usually excellent on this, but some passages in Discussion and Conclusion compress exploratory and confirmatory outcomes into flowing prose. For example, the row-4 low-data gain and the hardware anomalies are discussed near confirmatory findings; those sentences would benefit from repeated “exploratory” labels where relevant .
- Clarify “planning value” early and repeatedly.
The manuscript uses “+1.6 pp planning value” in a technically precise way, derived from recomputation of Bird et al.’s working range . However, some readers may not immediately understand whether this is a target effect size, equivalence margin, or replication benchmark. A brief reminder in Results and Conclusion would help.
- Improve readability of arm notation.
The A/B/C/D and B′/D′ notation is rigorous but cognitively heavy. Readers outside the immediate methodology niche may struggle to track that “D−B” means synthetic-stage benefit over the same pretraining baseline, while row 4 changes initialization to Imagenette . A compact recurring notation box or schematic would help.
- Data-availability wording should be made slightly stronger if journal policy permits.
The paper says “Data will be made available on request” , while later specifying that code, results, checkpoints, firmware, and energy captures are available and that the addendum code is attached to OSF . Because the manuscript places high value on reproducibility, consider reconciling these statements and, if possible, making non-image artifacts openly accessible at publication rather than on-request.
- Minor stylistic issue: some sentences are very dense.
For instance, long interpretive sentences in Methods and Discussion pack several caveats into one structure, which may reduce readability for non-specialists. The prose is careful but occasionally compressed. Shortening some of these sentences would improve accessibility without sacrificing rigor.
- Typographic / formatting observations.
I did not identify many obvious typographical errors from the searchable excerpts alone. The manuscript appears carefully copyedited. One formatting issue to watch is the hyphenation / line-break rendering in extracted text, such as “manuscript prepar-ation process” and “hy-perparameters,” which may be PDF extraction artifacts rather than real typesetting errors . Please verify final production copy for line-break hyphenation consistency.
- AI content analysis.
The manuscript explicitly discloses: “During the preparation of this work the author used Anthropic’s Claude Sonnet 4.8 and 5 to correct language and grammar, improve readability and check formatting” . Based on the prose alone, I would estimate a low-to-moderate probability of AI-assisted language polishing but not AI-generated scientific content in the substantive sense. Estimated AI-assisted prose proportion: roughly 15–30%, concentrated in stylistic smoothing rather than core methodology. The disclosure appears appropriate and specific. I do not see obvious signs that the empirical argument, statistical structure, or technical observations were irresponsibly delegated to AI; if anything, the manuscript’s methodological specificity argues against generic machine-generated content. The epistemic impact appears limited because the author also states that the content was reviewed and the author takes responsibility .
Recommendations
-
Reframe the central claim as a bounded, design-specific result: synthetic pre-training did not yield a detectable full-data benefit under this recipe, dataset, quantization pipeline, and model ladder; a replicated low-data gain appeared only for the smallest CNN and remains mechanistically unresolved.
-
Move a compact “claims the evidence supports / does not support” paragraph into the Conclusion. This will prevent overreading and emphasize the distinction between: confirmatory null for full data, exploratory replicated gain for row 4 at 10%, recipe sensitivity, and hardware-verification findings.
-
Add a dedicated paragraph on inferential sensitivity: state explicitly that the row-3 exclusion of the +1.6 pp planning value is narrow, depends on the ten-fold design, and is not evidence against smaller benefits .
-
Make the low-data finding more actionable experimentally. The paper already proposes the right next tests: “running it at 96 px, or a MobileNetV2 at 48 px, would separate input size from model size” . I recommend elevating this from future work to a central interpretive caution in both abstract and conclusion.
-
Consolidate the device-verification contribution. This is one of the manuscript’s most original aspects. A summarizing diagram or table showing host path, ESP32-S3 path, RP2040 path, reference-kernel agreement, and consequence for interpretation would significantly improve clarity.
-
Strengthen transparency around unrun or incomplete checks. The paper already lists that “Two items were not run” and that one meter check “was not done” . I recommend a brief summary table in the main text with status: planned, completed, not run, and effect on conclusions.
-
Anti-fraud / coherence assessment. I do not find substantive indicators of fabrication or paper-mill behavior. The manuscript fits a coherent TinyML / transfer-learning / deployment-validation paradigm, uses preregistration, reports null and adverse findings, documents departures, and names unresolved anomalies rather than hiding them . The main risk is not fraud but overinterpretation of narrow and partly exploratory results. Tightening claim boundaries would materially improve trustworthiness.
Overall, I judge this to be a serious, methodologically conscientious manuscript with a genuinely useful systems-and-methods contribution, especially regarding reference-kernel verification on embedded devices. Its main conclusions are plausible and mostly well-supported, but they should be narrowed and clarified wherever recipe sensitivity, fold dependence, and unresolved mechanism limit generalization.
Reviewer3 (AI review)
AI Review (reviewer3.com)
Overview
Synthetic Pre-training Evaluation on Microcontrollers
Contribution
This study demonstrates that synthetic pre-training yields no detectable accuracy gain for INT8-quantized microcontroller models when full real training data is available, providing benefits only under extreme data scarcity.
Hardware Cost Characterization and Kernel Verification
Contribution
The author conducts detailed on-device measurements of latency, memory, and energy across two microcontroller architectures while establishing that host-evaluated INT8 accuracy must be verified against embedded reference kernels to detect vendor runtime defects.
Comments (2)
1. Extra training epochs not isolated from rendered images — Major
nothing from the synthetic stage, so the weak start does not explain the gain on its own. The gain also survives shuffled labels, which rules out label content. That leaves something in the rendered images themselves or in the extra training, acting on row 4’s model. The bridge sees
At the 10% real-data fraction, the 2-epoch synthetic stage (372 batches) adds roughly 100% more optimizer updates on top of the 42-epoch real-data stage (~378 batches), and even with shuffled labels the stage still provides +3.104 pp. Without an arm B' control trained for an equal number of total gradient steps on real data alone (or with noise/real images during the extra 372 steps), the design leaves open whether the +2.715 pp gain at Row 4 comes from rendered image statistics or simply from doubling the optimization steps under a constant learning rate.
2. Arm-invariance energy statistic omits Arm B variance — Major
. The statistic of Equation (3) was R = +0 . 100% (95% interval +0 . 019% to +0 . 181% ), and the two arm-A captures differed by 0.58% of arm B’s figure. Both guards held and | R | was far under 1%. The test passed. The interval reflects only the spread between windows, though, and two captures of the same arm differed by about six times R ; the second arm-A capture would give R = +0 . 68% , still inside the bound. Idle-subtracted, the first arm-A
The energy invariance test statistic R = (E_A1 - E_D) / E_B treats the denominator E_B as a fixed constant and computes the 95% confidence interval half-width solely from the window-to-window variance of A1 and D, rather than testing the B-versus-D contrast directly or propagating session-level capture drift (which was 0.58% between A1 and A2, more than three times the reported 95% CI upper bound of +0.181%). Because between-capture drift dominates within-capture window noise, the reported window-based confidence interval (+0.019% to +0.181%) overstates the precision of the energy invariance estimate.
XSci (AI review)
Summary
The authors ask whether a synthetic pre-training stage inserted between ImageNet weights and real photographs changes the accuracy of a classifier small enough for a microcontroller, what it costs to deploy, and whether each board computes what the host scored. Five models, from MobileNetV2 at 224 px to a 61K-parameter CNN, are trained in four initialisation arms on a six-class scene dataset, scored in float32 and INT8 over ten folds, and deployed on two boards. At full real data the stage gives no detectable gain (row 3, INT8: +0.312 pp, 95% CI -0.849 to +1.472); with 10% of the photographs the smallest CNN gains +3.438 pp on seed 1 and +2.715 pp over three replicates. The ESP32-S3 matched TensorFlow Lite's reference kernels exactly once an out-of-bounds write in ESP-NN 1.3.1's depthwise kernel was patched; the RP2040 and the host interpreter did not.
Contribution and Significance
The core contribution is a pre-registered, paired evaluation of synthetic pre-training carried through post-training quantisation and onto two microcontrollers, joined to an element-exact check of what each board actually computes. The verification leg matters most to this journal's readers: Section 3.5 shows that TensorFlow Lite's default interpreter path, on which INT8 accuracies in this area are routinely reported, is not the reference path (265 against 262 correct of 300 on one deployed file; 65 of 300 output vectors more than one quantisation step apart), and Section 7.2 that a vendor kernel can be silently wrong.
Bird et al. [4] supply the dataset and the comparison, but at desktop scale, unquantised, with arms fine-tuned for 50 and 10 epochs — a confound the authors themselves expose; Muralidharan et al. [6] test a different contrast; Schlögl et al. [27] is nearest to the verification leg but does not reach microcontrollers. Genuinely new are the quantised, on-device leg of the question and the per-model, per-board reference check. Two things matter less than the framing suggests: the latency and energy ladder is routine characterisation, as Section 2.3 concedes, and the accuracy finding belongs to one recipe, since the authors' own second recipe turns the row-3 null into a deficit.
Strengths
- Thresholds and decision rules were fixed before any data were collected (Section 3.1, Table 2), confirmatory and exploratory results kept apart, departures logged; a null reported this way carries weight a post hoc one would not.
- The verification is done properly rather than declared: noticing that 111 of the 120 row-3 output values sit on an INT8 limit, the authors add a pre-softmax probe (120/120 unsaturated values matched) and 52 per-layer checksums, which the built-in kernels without the default delegate match only 2 of (Section 7.1).
- The ESP-NN 1.3.1 depthwise overrun that crashed row 3 was patched, confirmed with a hardware watchpoint and bisected to upstream commit 85961e4; three further defects were found, one fixed in a merged pull request [57].
- Paired and run-to-run spread are distinguished rather than conflated (Section 4.3, Table 5), which this literature routinely does.
- The architecture measurements are controlled: the 290,019 B static-array link limit decides arena placement between rows 2 and 3, and the row-4 PSRAM control isolates placement from size (1.671 times latency, 1.359 times energy above idle).
Recommendation
Major Revision for JOURNAL OF SYSTEMS ARCHITECTURE. The paper offers what this journal's readers can use — a verified deployment pipeline, a cost ladder on two common parts, and toolchain defects reported upstream — with methodological discipline well above the norm for the area. What holds it back is that the one positive accuracy result, which the abstract turns into practical advice, is confounded with the number of optimiser steps, and that the headline bound is stated more strongly than the authors' own sensitivity analyses support. Both are addressable within the present approach, the first with a few cheap runs.
Major Concerns
-
The only positive result is confounded with the number of optimiser steps (Sections 3.3, 3.4; Table 6). Section 3.3 accepts the synthetic stage's 372 optimiser steps, "about 10% more" at full data, as part of the treatment; at reduced fractions that ratio is not 10%. From the manuscript's own figures — 2,695 training photographs per fold, batch size 32, 3,570 real-data steps at full data and "22 batches against 85" at 25% (Section 6) — the real-data stage runs 378 steps at 10%, 924 at 25% and 1,806 at 50%, so the synthetic stage adds 98%, 40% and 21% of them, against 10% at full data. The row-4 difference, +3.438, +0.868, +0.134 and -0.067 pp at those four fractions, is monotone in that quantity: at 10% real data arm D' receives roughly twice the gradient updates arm B' does.
Two of the authors' own results fit this reading. Shuffling the rendered labels leaves the gain essentially intact (+3.104 pp; true minus shuffled -0.389 pp), which is what an optimisation effect would do and not what transferred scene content would do. The calibration bridge does not exclude it either: it shows a 17.3-point train-validation gap against the CNN's 1.5 (Section 3.3), overfitting where the CNN underfits, so extra steps should help only the latter. The abstract's prescription, "A synthetic stage is worth adding only when real data are scarce", therefore rests on an attribution the design has not made. The control is cheap: arm B' at 10% with the same total optimiser steps.
-
The headline bound clears its target by less than half an image per fold (Sections 4.1, 6, 10; Table 4). The primary's registered reading is that the INT8 interval at row 3 excludes the +1.6 pp planning value; its upper bound is +1.472, a margin of 0.128 pp, in a test whose stated minimum detectable effect at 80% power is 1.62 pp (Section 6) — the planning value itself. Three of the authors' own checks erode it: excluding folds 0 to 2, used to fix the synthetic-stage settings, gives an upper bound of +1.710 that no longer excludes the value (item C14); the second augmentation recipe moves the contrast by -2.226 pp (item C12); and the +1.6 pp value is itself recomputed from a source comparison whose arms were fine-tuned for 50 and 10 epochs (Section 2.1), so it is not a quantity this design can meaningfully exclude. Section 10 concedes that "the registered bound is a narrow one", yet the abstract, Section 9 and the Conclusion all lead with the exclusion. State instead that no effect was detected at row 3 and that gains above roughly 1.5 pp are ruled out under this recipe, with Bird et al.'s value as context rather than the finding.
-
The contribution most relevant to this journal is subordinated and under-developed (Sections 1, 2.3, 7.1, 7.2). Section 2.3 concedes that the hardware figures claim no priority, and the accuracy study is a single-dataset question whose answer the authors show is recipe-dependent. What remains as a durable systems contribution is the verification work: that an exact output match means little when the outputs saturate; that a pre-softmax probe and per-layer checksums are needed to test the arithmetic underneath; that the host path scoring INT8 accuracy is not the reference path; and that vendor kernels carry defects. Restructure around it, stating the procedure as a reusable protocol — what to compare, and how to choose inputs so saturation does not dominate — and move the item-by-item registration bookkeeping to the supplementary tables that already hold it.
Minor Comments
- Table 1's "Start for arms B and D" cell reads "ImageNet" at row 4, contradicting the table's own note and Section 3.3, which say Imagenette.
- CMSIS-NN carries every RP2040 figure (Section 3.7, Table 12) but is never cited: L. Lai, N. Suda, V. Chandra, "CMSIS-NN: Efficient neural network kernels for Arm Cortex-M CPUs", arXiv:1801.06601, 2018.
- Section 2.2 should cite J. Lin, L. Zhu, W.-M. Chen, W.-C. Wang, C. Gan, S. Han, "On-device training under 256KB memory", NeurIPS 2022, which fine-tunes a width-0.35 MobileNetV2 under INT8 on a microcontroller.
- Section 5 reports that at full data the shuffled-label arm D' finished +0.756 pp above B' while the registered true-label secondary is -0.445 pp, without reading that reversal.
- "Data will be made available on request" sits badly with a paper whose thesis is that deployed numbers must be verifiable; deposit the results files, folds.csv, firmware and captures.
- A host-side reimplementation of MEAN's plausible accumulator orders and rounding would probably settle the RP2040 departure the paper leaves open (Section 7.1).
- No capture measured the supply voltage (Section 3.7, Table 3), so every absolute figure in Table 13 should carry the resulting uncertainty band.
- The ESP32-S3 captures of record had no warm-up while the RP2040's followed 1,800 s (Table 3); promote a warmed-up ESP32-S3 row before quoting board-to-board ratios.
- A Cortex-M33 part with DSP or vector extensions — the RP2350 named in Section 11 — would say whether the RP2040's tenfold penalty is the instruction set or the kernel library.
- The Discussion stops short of a decision rule: at what real-data budget and model size is a rendered corpus worth building, given the 911.8 s of GPU time the stage costs (Section 6)?