XSci

Offline-First Quantized AI: Design and Evaluation of an On-Device Maternal-Health Assistant for Low-Connectivity Environments

Okenla Qahhar

Submitted by Okenla Qahhar

Published November 19, 2026 · Version v1, November 19, 2026 · DOI 10.66977/xsci.2611.003d

Artificial Intelligence, Software Engineering

Abstract

Offline language models could extend access to maternal-
health information where internet connectivity is unreliable, but
deployment requires both clinically appropriate responses and
feasible local execution. This study presents an offline Android
assistant integrating ready-made quantized language models,
a clinically validated local knowledge package, conversational
context, and model-initiated local retrieval. Gemma 4 E2B and
Qwen3.5 0.8B were compared across 45 conversations and 312
turns per configuration, including a dedicated offline agentic
benchmark. Gemma achieved higher response-quality, context-
retention, and safety scores; Qwen offered faster generation and
lower memory use. Clinician assessment of 150 Gemma responses
across nine clinical domains yielded a mean quality score of
3.80/4, with 147 responses meeting the acceptance criterion. A
paired RAG-on/off experiment evaluated 74 questions: retrieval
improved 11 responses, left 56 neutral, produced four adverse
response changes, and left three pairs unassessable. RAG-on
had descriptively higher mean usefulness, safety, correctness,
and actionability in the conditional 70-pair subset excluding the
four flagged pairs from both conditions. Physical-device testing
demonstrated offline responses on a 6 GB Android handset, with
a 14.3 s median latency, alongside successful reconnection syn-
chronization in the integrated application. Together, the findings
document clinical response quality and practical performance of
the integrated offline maternal-health assistant.

View PDF

Review reports

xPeerd (AI review)

November 19, 2026

AI Review (xPeerd — DAReview)

Summary

This manuscript presents an evaluation of an offline Android maternal-health assistant that combines locally deployed compact language models, a clinician-validated knowledge package, bounded local retrieval, and conversational context. The stated research problem is whether a compact, locally deployed model can provide “clinically appropriate maternal health information and reliable conversational assistance without continuous internet access” in constrained settings . The work’s main claimed contributions are threefold: integrating a clinician-developed question bank and conversation workload into one system evaluation; comparing two unchanged compact model configurations under common conditions; and testing the selected model with retrieval enabled versus disabled while retaining unfavorable cases to understand failure modes . The manuscript is strongest when it separates response quality, retrieval behavior, action capability, and device constraints rather than collapsing them into a single score. The design also shows awareness that “a passage about the same broad topic cannot downgrade an emergency” and that safety boundaries must precede optional explanation .

Its main weaknesses are methodological and reporting-related. The evidence base is largely descriptive, several analyses rely on internal benchmarks and LLM judging, the conditional 70-pair retrieval display is explicitly post-outcome-selected, and reproducibility is limited because “implementation code… [is] currently maintained in a private project repository and [is] available only upon request” . The manuscript is thoughtful and technically coherent, but its deployment-oriented conclusions should be narrowed to the tested benchmark, hardware envelope, and evaluation pipeline.

Potential Major Revisions

  1. Reproducibility remains materially limited despite the manuscript’s emphasis on evaluation rigor.

The manuscript states that “Implementation code, benchmark documentation, scoring workbooks, evaluation scripts, clinical keys, model-artifact manifests, and figure sources are currently maintained in a private project repository and are available only upon request” . Although the authors add that “benchmark questions, expected facts and actions, evaluation instructions, response-level scores, and reproducible non-sensitive metadata can be shared,” the present wording still leaves the core implementation and verification chain inaccessible by default . For a study that repeatedly frames its contribution as “the evaluated system and the evidence it supplies for deployment decisions,” this is a serious constraint on independent verification .

The paper should specify exactly what is shareable immediately, what requires request-based access, and what cannot be released at all. It should also provide a minimal reproducibility package sufficient to regenerate all tables and figures from frozen outputs, even if sensitive source materials remain restricted. As written, the manuscript asks readers to trust an internal pipeline more than is ideal for a systems-and-evaluation paper.

  1. The retrieval-effect analysis contains a post hoc subset that risks overstating stable benefits.

The manuscript is commendably transparent that “A conditional 70-pair comparison excludes the same four audit-flagged identifiers from both conditions” and that “This exclusion was defined after the outcomes were examined” . It further notes that the display “describes performance outside the flagged cases rather than an overall effect across the complete cohort” . This is an important admission, but the resulting presentation still risks interpretive inflation, especially because Figure 8 visually foregrounds the improved subset while the harmful failures are moved into the audit table .

This issue is especially important because the flagged failures are clinically meaningful, not random noise. Table VI includes cases where “Referral became conditional. Preserve urgency before adding education,” “Advice shifted toward a routine visit,” and “Blood hCG was overemphasized as a usual dating method” . These are not marginal degradations; they are exactly the kinds of failures that matter most in safety-facing health systems. The paper should therefore center the full 74-pair analysis in the main narrative and treat the 70-pair view as sensitivity analysis only. It should also justify why post-outcome exclusion is informative rather than selectively reassuring.

  1. Claims about offline utility and accessibility should be narrowed more aggressively to the tested hardware and operating envelope.

The discussion itself acknowledges a critical boundary: “Claims of accessibility should therefore specify the installed artifact size, usable memory, workload, and device conditions, rather than assuming that offline availability implies suitability for every entry-level handset” . It also concedes that “These observations describe the tested application conditions rather than performance across Android devices” and that the local timing evidence came from only “nine local requests” that “did not invoke retrieval” .

These limitations are substantial. The paper’s framing repeatedly invokes constrained-access African settings and offline practicality fileciteturn0file10turn0file14, but the actual device evidence appears narrow. The manuscript should more explicitly state that the handset findings establish feasibility on one tested configuration, not broad deployability. If the title, abstract, or conclusion implies wider readiness, that implication should be moderated. A deployment-oriented manuscript needs a sharper distinction between proof of concept, benchmark success, and field readiness.

  1. The evaluation depends heavily on LLM judging, but uncertainty from that choice is not fully integrated into the conclusions.

The manuscript notes that “LLM-based judgments can vary across repeated assessments and with changes in evaluation wording, even when explicit rubrics guide scoring” . It also relies on rubric-based judging for multiple dimensions and uses preserved rationales and score anchors to structure assessment fileciteturn0file4turn0file7. This is methodologically defensible, but the inferential consequences are not fully absorbed into the paper’s rhetoric.

For example, the manuscript reports descriptive differences and bootstrap uncertainty for paired comparisons, yet also says that analyses “do not include variation from judge choice, model repeats, or future users” and that “No superiority test is inferred from the descriptive means” . That caveat is appropriate, but then much of the discussion still reads as if the selected configuration has been robustly established as preferable. The authors should better align their interpretive claims with the actual uncertainty budget. In particular, they should report whether a second judge configuration, repeated judging pass, or inter-rater triangulation produced materially similar rankings.

  1. The manuscript is internally coherent, but the novelty claim should be scoped carefully to evaluation integration rather than architectural innovation.

The authors explicitly state, “No new quantization algorithm, calibration of weights, conversion, or within-model precision experiment is claimed” . They also describe the architecture as “a narrower form of agentic control” where the existing model requests one allowlisted retrieval tool and “weights, vocabulary, and training remain unchanged” fileciteturn0file3turn0file12. This candor is welcome. However, because the contribution is primarily evaluative and integrative, the paper should avoid any wording that could be read as an advance in agentic-RAG method design per se.

The strongest defensible novelty lies in the combined benchmark design: clinically reviewed question bank, sustained conversation workload, benchmark-only action tasks, retrieval-on/off comparison, and physical-device feasibility evidence fileciteturn0file9turn0file17. The paper should foreground that integrative evaluation contribution and resist broader claims about advancing offline agentic retrieval architecture. That would improve epistemic precision and reduce the risk of overstating originality.

  1. Safety claims should be expressed more conservatively because the paper itself documents consequential adverse retrieval effects.

The manuscript’s strongest virtue is that it does not hide failures. It states that “four adverse changes demonstrated failures of clinical fit, urgency or evidence application” . It also defines critical errors to include “missed emergency, unsafe treatment, fabricated result, materially false tool-success claim, invalid consequential action, or a safety-critical request left unanswered” . This framework is appropriate, but it means that any concluding language suggesting stable safety or reliable assistance must be tightly constrained.

The paper should explicitly summarize the incidence and nature of critical or near-critical degradations in the abstract and conclusion, not only in the discussion and audit table. A maternal-health assistant that can worsen urgency framing in warning-sign scenarios must present that risk centrally. Otherwise, the narrative balance may unintentionally privilege average gains over clinically salient tail failures.

  1. The dataset and evaluation provenance raise possible optimism bias that should be addressed directly.

The manuscript reports that “In-house medical doctors developed and reviewed the 150-question bank” and that “Forty items marked for revision in the consensus review record were revised” to make decision-relevant context explicit fileciteturn0file10turn0file17. This careful curation is valuable, but because the same project appears to have defined the benchmark, keys, review conditions, and analysis framework, the paper should discuss the risk of benchmark alignment to the designed system more directly.

Even if no improper tuning occurred, readers need to know how separation was maintained between system design, benchmark authoring, and evaluation. The manuscript says that reviewer paths and key development were separated in the single-response process , but it should provide a clearer procedural account of independence across development and evaluation roles. This is especially important for health-adjacent benchmark studies with internally developed question banks.

Potential Minor Revisions

  1. Writing and phrasing issues
  • The phrase “The study addresses the following question: under what conditions can a compact, locally deployed language model provide clinically appropriate maternal health information and reliable conversational assistance without continuous internet access?” is clear but long and rhetorically heavy for a central research question; it could be shortened for readability .
  • The sentence “The contribution is the evaluated system and the evidence it supplies for deployment decisions” is concise, but slightly vague because “evidence” here spans benchmark scores, retrieval audits, and device observations; define that more concretely at first use .
  • The sentence “The operational problem has both a language-response and an evidence-acquisition component” would read more smoothly as “response-generation” rather than “language-response” .
  • The phrase “structured action capability must be evaluated independently of response fluency” is strong but could be introduced earlier in the Methods so readers understand sooner why a separate action benchmark is needed .
  1. Formatting and presentation issues
  • Several sections are information-dense and would benefit from more explicit signposting. For example, the Methods sequence spanning study design, agentic retrieval, context handling, rationale for retrieval, benchmark action assessment, and rubric judging is coherent but visually compressed in the current layout fileciteturn0file1turn0file2turn0file7.
  • Tables I–IV are central to the paper’s logic, but some table captions and labels appear typographically compressed in the extracted text, which may hinder readability in final typesetting fileciteturn0file3turn0file12turn0file17.
  • Figure 8 is potentially misleading in emphasis because the outcome-selected 70-pair display visually highlights improvement counts while the excluded failures require cross-reference to Table VI; the figure title/caption should foreground that this is a conditional display selected after reviewing outcomes fileciteturn0file8turn0file11.
  • The manuscript should ensure consistent terminology among “response quality,” “usability,” “usefulness,” and “clinical result,” since these terms refer to related but distinct constructs in different parts of the evaluation framework fileciteturn0file4turn0file7turn0file18.
  1. Reporting completeness issues
  • The paper should state more explicitly in one consolidated place which analyses are descriptive only. At present, this caveat appears in the statistical-analysis section but is easy to miss .
  • The authors should include a concise flow summary of the benchmark cohorts: 150 questions in the clinical bank, 45 conversations / 312 turns per model, 74 matched retrieval questions, and the derived 70-pair conditional subset fileciteturn0file17turn0file18turn0file11.
  • The data/code availability section should specify what “available only upon request” operationally means, such as whether review-time anonymous access is possible and whether there are legal or ethical restrictions .
  1. AI content analysis

Estimated percentage of AI-generated content: low-to-moderate likelihood, approximately 20–35%.

Rationale: The prose is highly uniform, technically polished, citation-dense, and structurally regular, with recurring sentence templates such as “The present…,” “The study…,” and “The evaluation…” across sections fileciteturn0file1turn0file14turn0file15. This style can arise from careful human drafting, AI-assisted polishing, or both. I do not see obvious incoherence, fabricated internal cross-references, or generic filler severe enough to suggest paper-mill generation. On the contrary, the manuscript maintains a stable argument and a consistent evaluation logic across sections. The strongest AI-assistance signals are stylistic regularity and repetitive rhetorical scaffolding rather than factual breakdown.

Highlighted likely AI-assisted sections:

  • Literature-synthesis passages that compact many citations into smooth but formulaic transitions, especially in Related Work fileciteturn0file9turn0file14turn0file16.
  • Some abstract-like discussion sentences in the conclusion and principal-findings sections, where the prose is highly optimized and repetitive in structure fileciteturn0file8turn0file15.
  • Methodological justification paragraphs that repeatedly explain distinctions among response quality, retrieval, and action capability in very polished language fileciteturn0file3turn0file4turn0file18.

Assessed epistemic impact: limited if AI use was editorial/polishing only; more serious if AI contributed to technical claims, synthesis of prior work, or statistical interpretation without careful human verification. In this manuscript, the main risk is not obvious fabrication of content but potential over-smoothing of uncertainty and over-optimization of presentation. Because the manuscript’s scientific value depends on evaluation design and interpretation rather than prose style alone, the authors should verify all claims, counts, and cross-references manually.

Anti-fraud / paper-mill assessment

I do not find strong paper-mill traits. The manuscript fits a coherent scientific paradigm: offline clinical conversational AI evaluation with bounded retrieval, benchmarked action capability, and device-level deployment constraints fileciteturn0file1turn0file3turn0file14. The contribution is incremental but intelligible. It does not exhibit the typical fraud markers of concept soup, impossible methods, meaningless equations, or irrelevant citation padding. That said, there are two integrity-sensitive areas requiring attention: first, the privately held code and benchmark infrastructure ; second, the internally developed and revised benchmark keys that may introduce alignment bias if procedural independence is not documented clearly fileciteturn0file10turn0file17.

Recommendations

  1. Reframe the paper explicitly as an evaluation-and-deployment study rather than an architectural advance. This would better match statements such as “No new quantization algorithm… is claimed” and avoid overselling methodological novelty .

  2. Promote the complete 74-pair retrieval analysis to the primary result, and move the 70-pair conditional display to a sensitivity-analysis role with stronger caveats about post-outcome exclusion fileciteturn0file11turn0file8.

  3. Strengthen reproducibility by releasing, at minimum, frozen outputs, scoring sheets, analysis scripts, and a figure-regeneration package without request-gating wherever possible .

  4. Add a concise benchmark-governance subsection explaining separation of roles among benchmark creation, system development, clinical review, and final scoring. This would improve trust in the internal validity of the evaluation fileciteturn0file7turn0file10turn0file17.

  5. Make safety limitations more prominent in the abstract, conclusion, and any claims of readiness. The documented harmful retrieval cases should not be narratively secondary fileciteturn0file6turn0file8.

  6. Narrow device-level conclusions to the tested configuration and clearly distinguish feasibility from general deployability, especially for low-resource Android environments fileciteturn0file6turn0file15.

  7. If possible, add an additional robustness check for the judging pipeline, such as repeat LLM judging, alternate judge prompts, or partial blinded human adjudication, so that evaluation uncertainty is not limited to bootstrap resampling over fixed judged outputs fileciteturn0file4turn0file6turn0file19.

Overall, this is a serious and mostly coherent manuscript with a useful benchmark-oriented contribution. Its main barrier to stronger acceptance is not obvious conceptual incoherence, but rather overextension from a controlled internal evaluation to broader claims about reliable offline maternal-health assistance.

Reviewer3 (AI review)

November 19, 2026

AI Review (reviewer3.com)

Overview

On-Device Perinatal AI Assistant

Contribution

The authors design and implement an offline Android maternal-health assistant that integrates compact quantized language models with a clinically validated local knowledge package and model-initiated local retrieval.

Retrieval Impact and Failure Analysis

Contribution

The study conducts a paired evaluation showing that while local evidence retrieval improves specific responses, it can also degrade performance by altering clinical urgency or substituting relevant guidance.

Comments (13)

1. Incomplete mathematical definition of the resource index — Major

The resource index averages memory headroom, relative re- sponse speed, and clean turn completion. The individual components remain necessary for interpretation: identical com-

Appendix B defines the resource index as the average of a reference memory-headroom score, response speed relative to the faster configuration, and the fraction of clean turns, and reports exact values of 63.3 for Gemma and 93.1 for Qwen. However, the formula for the reference memory-headroom score and which latency or throughput metric is used for relative response speed are omitted, preventing verification or reproduction of the resource index.

2. Unspecified LLM judge identity and configuration — Critical

ory. Retrieval and generation are timed separately. Reporting of model configurations, task definitions, oversight, and in- tervention components follows applicable TRIPOD-LLM and mERA principles [36], [17].

Although the manuscript states that an LLM judge scored both the 45 multi-turn conversations and the 74 paired retrieval responses (and explicitly claims to follow TRIPOD-LLM reporting principles), the identity, version, temperature, and prompting setup of the judge model are never disclosed. Without knowing which model acted as the evaluator—or whether it belongs to the same model family as either candidate—it is impossible to assess self-preference bias or reproduce the primary evaluation metrics.

3. Unreported confidence intervals and diagnosis for near-zero ICC — Major

The clinician-rating agreement analysis yielded ICC(A,1) values of 0.25 for correctness, 0.07 for completeness, 0.04 for safety, 0.22 for relevance, and 0.31 for clarity across the 147 complete rows. High mean ratings therefore coexisted with

The inter-rater reliability values for safety (0.04) and completeness (0.07) are close to zero, and although the text notes that 2,000 bootstrap replicates supplied uncertainty intervals for ICC(A,1), those intervals are withheld ('retained with the analysis files'). Moreover, the authors suggest restricted score variation or grading stringency might explain the low ICCs, but completeness had the lowest mean (3.51) and widest domain spread (down to 3.14), so ceiling effects cannot account for the lack of agreement on completeness without reporting the actual rater means and variance components (MS_Q, MS_R, MS_E).

4. High conversational critical-error rate contradicts clinical appropriateness claim — Critical

Critical errors occurred in 11/45 Gemma conversations and 24/45 Qwen conversations. The project rubric classified 25 Gemma and 8 Qwen conversations as provisional accepts.

While the single-turn clinician evaluation yielded a 98.0% acceptance rate for Gemma 4 E2B, the multi-turn conversational evaluation recorded critical errors—defined as missed emergencies, unsafe treatments, fabricated results, or unanswered safety-critical requests—in 11 of 45 conversations (24.4%), with only 25 of 45 conversations (55.6%) achieving provisional acceptance. Claiming that the integrated conversational system provides clinically appropriate responses is not well supported when nearly a quarter of simulated multi-turn conversations result in a critical safety failure.

5. Lack of blinding in paired RAG-on versus RAG-off judging — Major

Paired judging uses the same question, clinical key, and expected action for both conditions. Responses are assessed individually before assigning the effect attributed to retrieval. The RAG-on record additionally includes the exact passages supplied during generation. Context relevance asks whether

In the paired retrieval evaluation, the RAG-on record explicitly included the exact retrieved passages supplied during generation alongside the response, whereas the RAG-off record did not, and groundedness was scored only for RAG-on. Because the judge could trivially identify which response was RAG-on and which was RAG-off when scoring usefulness, safety, correctness, and actionability, the comparison is not blinded to treatment condition.

6. Unexplained memory discrepancy between emulator and physical handset — Major

tokens. Available memory at inference start was 1.4–2.1 GB and peak process PSS was 1.5–1.8 GB. These measurements

On the 8 GB emulator, Gemma 4 E2B required 4.23 GB peak RSS (Table V), and on the 4 GB emulator guest it was killed during initialization at ~2.68 GB peak RSS. Yet on the 6 GB physical Samsung handset, with only 1.4–2.1 GB of available memory at inference start, Gemma completed all 9 requests with a peak process PSS of just 1.5–1.8 GB and a throughput (22.2 tok/s) nearly triple that of the emulator (7.86 tok/s). Even accounting for differences between RSS and PSS, running a model that crashed at 2.68 GB during initialization within 1.4–1.8 GB of available memory without explanation leaves it unclear whether memory-mapped weights were excluded from PSS or if a different runtime configuration was used.

7. Missing quality results for Qwen on the 150-question benchmark — Major

The candidate comparison evaluates two complementary workloads: 150 isolated questions per configuration and 45 conversations containing 312 ordered turns per configuration.

Sections IV and IV-A state that the candidate comparison generated and completed a separate 150-question isolated response collection for each model alongside the 45 conversations. However, only latency, throughput, and peak RSS are ever reported for Qwen on the 150 isolated questions (Table V); no response quality or safety scores are provided for Qwen on that workload.

8. Model-initiated local retrieval is never actually evaluated — Critical

Thisstudy presents an offline Android assistant integrating ready-made quantized language models, a clinically validated local knowledge package, conversational context, and model-initiated local retrieval.

The system is presented as an offline assistant integrating model-initiated local retrieval as its only deployed tool, and the authors emphasize that invocation, admission, and application must be evaluated separately. However, in the 150-question collection and the 74-pair RAG experiment, retrieval was pre-supplied with tools disabled, and the 45-conversation benchmark tested a separate benchmark-only action interface (immunization lookup, date calculation, conversion, saving, reminders) rather than the retrieval tool. Consequently, the model's autonomous decision of when to invoke the local retrieval tool is never empirically evaluated in any experiment.

9. Fixed deterministic Run A/B labels fail to prevent judge bias — Major

dling, and runtime status. The multi-turn scoring preparation replaced model names with Run A and Run B labels as- signed consistently across conversations. Prior score fields

In the multi-turn evaluation, model identities were replaced with 'Run A' and 'Run B' deterministically across all 45 conversations rather than randomized per conversation, and the paper acknowledges documented position biases in LLM judging in Section II-B. If the judge evaluates Run A and Run B together or carries a label/order preference, consistent deterministic labeling totally confounds model identity with label and position bias.

10. Ambiguous critical-error aggregation rule across multiple clinician raters — Major

each question. Acceptance requires a mean quality score of at least 3.2 and no critical-error flag. Any recorded critical error overrides the numeric mean. These clinician ratings constitute

The acceptance rule specifies that any recorded critical-error flag overrides the numeric mean across the three reviewer columns, yet the inter-rater reliability for safety was near zero (ICC(A,1) = 0.04). It is not explained whether the critical-error flag is recorded by individual reviewers or reconciled by consensus, nor is inter-rater agreement on critical-error flags reported.

11. In-house clinician raters were not independent or blinded — Major

lated conversations. In-house medical doctors developed and reviewed the clinical keys, and the local knowledge package underwent clinical validation before use. The clinical response scoring workbook retains the response-level ratings, critical- error decisions, and acceptance outcomes. The experiments

The 150 Gemma responses were evaluated by an internal clinician panel from the same organization that developed the system, without any comparison or blinding against alternative models or baseline responses. Knowing that all 150 responses came from the team's selected candidate model introduces substantial expectation and confirmation bias.

12. Unexplained discrepancy in retrieval acceptance counts across identical runs — Major

The single-question collection enabled retrieval and disabled tools; accepted evidence was supplied for 26/150 questions.

Both the 150-question candidate collection (used for clinician rating) and the 150-question paired RAG experiment probed the exact same 150-question bank using deterministic SQLite FTS5/BM25 retrieval with tools disabled. However, accepted evidence was supplied for only 26/150 questions (61 passages) in the candidate collection versus 74/150 questions (81 passages) in the paired retrieval study. While Table III alludes to an 'original normalized package' versus a 'revised active package,' the text never explains why the clinician-evaluated responses used a different retrieval package or how that difference affects the validity of the 150-question evaluation.

13. Post-hoc exclusion of only harmed cases biases conditional subset — Major

response changes, and left three pairs unassessable. RAG-on had descriptively higher mean usefulness, safety, correctness, and actionability in the conditional 70-pair subset excluding the four flagged pairs from both conditions. Physical-device testing

The conditional 70-pair comparison selectively excludes the 4 pairs where RAG-on caused an adverse response change while retaining the 11 pairs where RAG-on improved the response, and highlights descriptively higher means for RAG-on in the Abstract, Figure 8, and Discussion. Conditioning on the absence of treatment-induced failures without symmetrically conditioning on treatment-induced successes creates a one-sided selection bias that artificially inflates the apparent mean performance of RAG-on.

XSci (AI review)

November 19, 2026

Summary

The manuscript describes an offline-first Android maternal-health assistant for low-connectivity settings in Nigeria, built from unmodified quantized artifacts (Gemma 4 E2B, Qwen3.5 0.8B), a clinically curated 21,673-passage on-device question–answer package, bounded conversation history, and one model-invoked local retrieval tool over SQLite FTS5/BM25. The two configurations are compared on 150 isolated questions and 45 conversations (312 turns) each, plus a benchmark-only action interface; Gemma is selected on quality (3.16/3.89/3.40 for response, context and safety versus 1.29/2.18/1.96) despite higher cost (16.75 s per turn, 4.23 GB peak RSS). Clinician review of the 150 Gemma answers gives a mean of 3.80/4 and 147/150 acceptances, and a paired RAG-on/off experiment on 74 matched questions reports 11 improved, 56 neutral, 3 unassessable and 4 adverse pairs. Device testing records nine offline generations on a 6 GB handset (median latency 14.3 s) and 46/47 reconnection synchronizations.

Contribution and Significance

The contribution is an integrated system evaluated jointly along four axes usually reported separately — clinical response quality against prospectively written keys, multi-turn context and safety, the effect of local evidence, and on-device execution cost — under one fixed mobile configuration. The most valuable result is the paired retrieval experiment (Section IV-E, Table VI): clinically validated local evidence is neutral in 56 of 74 pairs and in four cases degrades the response, including a referral that "became conditional" for headache with visual changes and advice that "shifted toward a routine visit" for severe nocturnal palm and sole itching. For deployment-oriented readers this is more useful than another demonstration that a small model runs on a phone, and the full 74-pair differences (−0.041, +0.014, +0.014, 0.000) are reported with intervals spanning zero.

What is new is the assembly and the failure audit, not the components: the models, quantization schemes, BM25 retriever and agentic-RAG pattern are adopted unchanged, as the authors state. Relative to MAM-AI [28] and Pocket RAG [26], the closest on-device medical retrieval systems, the differentiator is the clinical key set and the retrieval-harm analysis, but neither is compared empirically, so the advance rests on evaluation design rather than measured superiority. The study is also single-site throughout: corpus, questions, keys, raters and judge.

Strengths

  • Table VI names four separable retrieval failure mechanisms (weakened urgency, wrong symptom pattern, substituted topic, unsupported precision) and turns each into a concrete regression test, rather than reporting one aggregate RAG benefit.
  • The statistical reporting is candid: 20,000-replicate paired bootstrap intervals for both cohorts, Wilson intervals on effect proportions, an explicit refusal to infer superiority from descriptive means (Section III-M), and the excluded identifiers listed in Appendix A.
  • The knowledge package is auditable (Table IV: 36,636 source records → 36,548 normalized → 23,977 reviewed → 21,673 admitted), and Section III-D states curation had no access to the benchmark questions or model outputs.
  • Negative execution results are reported: on the 4 GB guest Gemma was killed at approximately 2,675,796 kB peak RSS and Qwen's first full RAG answer did not complete within ten minutes (Section IV-B).
  • The E0–E3 safety boundary (Table I) is assigned during clinical design rather than inferred from outputs, and the critical-error rule stays decoupled from the numeric threshold (Section III-J).

Recommendation

Major Revision for the leading journals of the field. The system and the retrieval-harm analysis are a contribution such journals would publish, and the manuscript is unusually transparent about its limits. What stands in the way is that the two headline positive claims — 147/150 clinician acceptance, and higher RAG-on scores on all four dimensions — rest on measurements the manuscript's own numbers show to be unreliable or circular. Both can be repaired with data already collected, which is why this is a revision rather than a rejection.

Major Concerns

  1. The clinician instrument behind the headline quality claim shows near-zero agreement, and conflicts with the conversation results for the same model (Sections III-J, IV-A, IV-D). The reported ICC(A,1) values are 0.25 for correctness, 0.07 for completeness, 0.04 for safety, 0.22 for relevance and 0.31 for clarity. Safety agreement of 0.04 means the three rating columns are close to independent for the dimension that matters most here, yet the acceptance rule (item mean ≥ 3.2, no critical error) is applied directly to those means and 147/150 reaches the abstract as evidence of clinical quality. The manuscript attributes this to restricted variance and stops. The conversation workload sharpens the problem: the same configuration recorded critical errors in 11 of 45 conversations and only 25 of 45 provisional accepts. A 98% single-response acceptance rate and a 24% conversation-level critical-error rate cannot both characterise the assistant, and the two are never reconciled.

    Report how many clinicians rated, their qualifications, and whether each column is one rater throughout; recompute acceptance under each rater separately and under a stricter rule (for example, all raters ≥ 3); and adjudicate the divergent items. If the ratings are at ceiling across all nine domains (no cell in Figure 7 falls below 3.14), say so, and present 3.80/4 as a screening result rather than an estimate of clinical adequacy.

  2. The displayed retrieval comparison is conditioned on the outcome it reports (Sections III-M, IV-E, V-C, Figure 8, abstract). Section III-M states that "This exclusion was defined after the outcomes were examined," and the four excluded pairs are exactly the four adverse ones. A comparison that removes only the adverse cases from both arms cannot do otherwise than favour RAG-on, so the 70-pair means (3.41/3.76/3.76/3.47 versus 3.39/3.67/3.70/3.41) carry no evidential weight about the effect of retrieval, and their intervals include zero in any case. Yet this subset is the abstract sentence, the plotted figure and the opening of Discussion V-C, while the pre-specified 74-pair result, which shows no benefit, appears later. The fix costs no new experiments: make the 74-pair cohort the headline in the abstract, Figure 8 and Section V-C — accepted local evidence produced no detectable average change, improved 14.9% of responses and degraded 5.4% — and keep the 70-pair display as a secondary sensitivity panel. Note there too that 30 of 74 accepted contexts were judged partially relevant or irrelevant, which bears on whether the Appendix B admission thresholds are set correctly.

  3. The judge supplying the model-selection evidence is unvalidated and only nominally blinded (Sections III-K, IV-A, Figure 5). All of RQ1's quality, context and safety numbers, the critical-error flags and the paired RAG dimension scores come from one LLM judge in a single scoring pass. Section III-K states that "Model labels were assigned deterministically, rather than randomized independently for each conversation," so any systematic preference for the Run A or Run B position applies identically to every conversation instead of averaging out; the position and verbosity biases of [23] are cited but not controlled. No repeated or order-reversed judgments are reported, and no judge–clinician agreement (Section V-H defers it to future work). Figure 5's quality index (82.7 versus 52.4), which drives the deployment recommendation, inherits all of this. The 3.16-versus-1.29 gap is wide enough that the ordering of the two models will probably survive, and the paper should say so; what does not survive is the magnitude, and with it the selection illustration. Rescore a sample of conversations with the clinicians who wrote the keys, report agreement on the ordinal dimensions and on critical errors, and rerun the judge with labels reversed and repeated.

Minor Comments

  1. Figure 4 shows Qwen's normalized tool score as 78.8% while the text reports a mean of 2.98, which normalizes to 79.8%; the quality index of 52.4 reproduces only with 0.798, so the figure appears to be the error.
  2. The five dimension means in Figure 6 (3.86, 3.51, 3.82, 3.66, 3.98) average to 3.77, not the 3.80 reported in Section IV-D.
  3. Peak memory appears as 4.23 GB RSS on the emulator, 1.5–1.8 GB PSS on the handset and a kill at roughly 2,675,796 kB on the 4 GB guest; these cannot be read together, and the paper's deployment question needs a stated minimum supported RAM class.
  4. The nine handset requests are one device and did not invoke retrieval, so the 14.3 s median does not describe the deployed path; temper the abstract and conclusion, and measure retrieved-answer latency on hardware.
  5. Evidence was accepted for 26/150 questions in the candidate collection but 74/150 in the paired study on the same bank, which is unexplained and matters because the selection answers were partly evidence-supported.
  6. The artifact URLs are mis-typed ("edge/liter-lm" in [39]; "litter-community/gemma-4-E2B-it-litter-lm" in [41], which should be litert-community/gemma-4-E2B-it-litert-lm), undermining the reproducibility the paper emphasises.
  7. Given the stated contribution, releasing the question bank, expected facts and actions, and response-level scores would add more than a private repository available on request.
  8. Section III-C refers to "Table I-I" where Table II is meant.
  9. Dispersion is missing where it matters: the conversation-level means carry none, and the ICC bootstrap intervals are said to be retained with the analysis files but never shown.
  10. Results are bound to one quantization scheme per family and to CPU inference; a third configuration, and a GPU or NPU delegate comparison, would show how far the quality–resource conclusion generalises.