Nepali-English Code-Mixed Speech Recognition: Data Preparation and Adaptation of a Speech Foundation Model
Jeevan Neupane, Janardan Bhetwal, Kshitiz Paudel, Anup Aryal, Bipin Bashyal
Abstract
Nepali speakers regularly mix English words into conversation, but multilingual speech recognisers often struggle with language switches and the writing script used for English words. We study how to adapt SraVaani, a 449M-parameter hybrid TDT-CTC model, using a 172.8-hour corpus covering six Nepali speech domains. Its pretrained tokenizer does not support digits and several common symbols. Rather than replace it or discard these recordings, we generate Nepali and English readings for numbers and use zero-shot ASR output to select a plausible form. Full fine-tuning reduces test word error rate (WER) from 48.6% to 17.1%; a 12-epoch run reaches 16.3%, compared with 16.4% for fine-tuned Whisper large-v3-turbo. Freezing the encoder increases WER to 25.5%, while excluding numeric clips raises WER on numeric test speech from 14.3% to 21.3%. We also evaluate on FLEURS using both its original references and mixed-script alternatives. The comparison shows that transcription conventions explain part, but not all, of the out-of-domain performance loss.
Review reports
xPeerd (AI review)
AI Review (xPeerd — DAReview)
Summary
This manuscript presents an adaptation study of SraVaani for Nepali–English code-mixed automatic speech recognition, using a 172.8-hour six-domain corpus and a fixed tokenizer that cannot represent digits and several symbols. The paper’s central contribution is a practical data-preparation pipeline that verbalizes numbers in Nepali or English and selects candidate labels using zero-shot ASR evidence, followed by fine-tuning and ablation analysis. Reported gains are large in-domain: SraVaani improves from 48.6% WER zero-shot to 17.1% after six epochs and 16.3% after twelve epochs, roughly matching fine-tuned Whisper large-v3-turbo at 16.4% while using fewer parameters. The manuscript is strongest when isolating the effects of encoder adaptation, numeric-clip retention, and scoring conventions for mixed-script references. It is also commendably explicit that the study is intended to identify “which changes matter, not to establish a new general-purpose state of the art.”
The main weaknesses concern dataset and evaluation validity rather than the overall idea. The paper itself acknowledges that splits are not guaranteed speaker-disjoint, that mixed-script FLEURS references were generated with LLM assistance but not fully human-verified, and that numeric label selection may inherit base-model bias. These are substantial limitations because they affect the reliability, reproducibility, and generalizability of the strongest claims.
Potential Major Revisions
-
The manuscript should more clearly delimit its main claim because the in-domain gains may be optimistic under the current split design. The paper states in “Limitations and Ethical Considerations” that “The inherited train, validation and test splits are not guaranteed to be speaker-disjoint. As a result, in-domain WER may be optimistic.” This is not a minor caveat. For ASR, speaker overlap can materially inflate apparent generalization, especially when the paper reports large improvements such as “test WER fell from 48.6% to 17.1% after six epochs and to 16.3% after 12 epochs.” The authors should either provide a speaker-disjoint re-evaluation or sharply narrow the interpretation of their in-domain claims throughout the abstract, results, and conclusion.
-
The numeric normalization pipeline is interesting, but its correctness remains insufficiently validated for such a central methodological component. The manuscript states, “We therefore construct alternative readings and select among them using the base model’s output as evidence, followed by quality checks.” Later it concedes: “Selecting number readings from the base model’s predictions may favour its existing errors” and “the final versions still require independent review.” The reported manual audit is limited: “We manually audited 150 numeric clips… Five contained errors (3.3%...)” and the “corrected numeric labels still need independent listening-based verification.” Because numeric retention is one of the manuscript’s headline findings, the paper needs a stronger validation protocol, ideally with human listening-based adjudication on a statistically meaningful sample and inter-annotator agreement.
-
The out-of-domain analysis is informative, but the use of LLM-assisted rewritten FLEURS references introduces a second evaluation layer whose validity is not yet established. The paper says: “An LLM made the initial text conversion from the original FLEURS references without seeing model predictions. We checked phonetic consistency and conversion rules automatically, but did not complete human verification.” It also notes in the limitations that these references were “generated with LLM assistance and automatically checked, but not independently validated by listening to each recording.” Since a major interpretive claim hinges on these rewritten references—namely that “changing the reference writing style reverses the apparent direction of the result” —the paper should either complete human validation or downgrade the strength of that conclusion.
-
The comparison with baselines is potentially under-controlled. The manuscript reports that “Whisper’s validation WER was still improving after its shorter training schedule.” Yet comparative performance claims are drawn from that baseline, including “Whisper large-v3-turbo reaches 16.4% WER, and 12-epoch SraVaani reaches 16.3%.” If one baseline had not converged, the comparison may not reflect each model’s best achievable performance under matched optimization effort. The paper should either retrain baselines to convergence or explicitly weaken any near-parity implication.
-
The manuscript sometimes moves too quickly from empirical improvement to causal interpretation. For example, in Section 6.3 it states, “Updating the encoder therefore appears to help recognise the speech itself, not just change its written form.” This is plausible, but still inferential. Romanised WER reduction alone does not fully isolate acoustic recognition improvements from decoder-side sequence modeling changes after adaptation. A more cautious phrasing or additional probing analyses would improve methodological rigor.
-
The novelty claim is practical rather than conceptual, and the manuscript should state that more explicitly. The paper itself says the contribution is not a new state of the art and positions itself around “fixed-tokenizer preparation, bilingual numbers and encoder adaptation.” That is acceptable, but the manuscript should sharpen exactly what is new relative to Devkota and Paudel, especially because it admits these are “directly relevant precedents for mixed-script Nepali recognition.” Right now, the novelty reads as incremental engineering plus careful ablation; that can still be publishable, but it should be framed with precision.
-
Reproducibility is good in parts but still incomplete for independent verification of the full findings. The paper helpfully releases code and model checkpoints, yet it also states that “These releases do not include the collected training audio, which remains subject to the original owners’ rights and permissions.” This means the principal adaptation results cannot be fully reproduced by third parties. The manuscript should discuss the practical reproducibility implications more explicitly and, if possible, provide a fully reproducible public subset experiment.
-
There is a mild paper-mill/fraud screening concern not because the manuscript appears fabricated overall, but because some parts rely on self-generated or model-assisted artifacts that are not yet independently verified. Specifically, the LLM-assisted FLEURS rewriting and the base-model-guided numeric relabeling create circularity risks if presented too strongly. The manuscript partly recognizes this, which is reassuring. I do not see strong signs of incoherence or fabrication; the paradigm, methods, and ablations are scientifically coherent. However, stronger human validation is needed before the most consequential evaluation claims can be considered fully reliable.
Potential Minor Revisions
-
Terminology consistency could be improved around evaluation claims. In the introduction the manuscript says, “A model can recognise the spoken word yet still produce the wrong script. Under standard word error rate (WER), that difference is treated as a recognition error.” This is conceptually fair, but “recognition error” conflates acoustic recognition with orthographic convention. Since later sections carefully separate these with romanised WER and English exact match, the introduction should use more precise wording.
-
The phrase “wrong script counts recognisable English words written in Devanagari” in Section 5.4 is useful but would benefit from a brief operational example in the main text, not only in tables.
-
There appears to be a page-flow or formatting artifact around page numbering and line joins, for example “We use corpus-level error totals, not a mean of clip-level scores. We 5estimate 95% confidence intervals...” where the page number interrupts the sentence. Similar joining artifacts appear elsewhere and should be cleaned in the camera-ready version.
-
Some hyphenation and OCR-like rendering artifacts reduce readability, such as “codemixing,” “finetuning,” and “outof-domain.” These should be normalized consistently throughout.
-
The sentence “These comparisons are intended to identify which changes matter, not to establish a new general-purpose state of the art” is appropriately modest, but it would read better in the abstract or conclusion if paired with one sentence stating the intended deployment or research use case.
-
The manuscript says “All scoring uses lowercased text with punctuation removed.” Given that script conventions are central to the paper, a short justification for lowercasing English and removing punctuation would help readers understand what distinctions are intentionally discarded.
-
The implementation details are brief: “one NVIDIA RTX 5090 with 32 GB of memory, NeMo 2.7.3 and the recorded PyTorch/CUDA environment.” It would help to specify optimizer settings, batch size, effective batch accumulation, decoding hyperparameters, and seed handling in one compact reproducibility subsection if these are not already fully listed in tables not surfaced here.
-
The statement “Only the main configuration was repeated with a second training seed” should be moved from limitations into a clearer reproducibility note near the experiments section, so readers immediately understand the extent of run-to-run validation.
-
Grammar/style edits:
- “The main preparation challenge was resolving digits and symbols into labels that fit the vocabulary and the spoken language.” This is clear, but “spoken language” could be misread; “spoken form” may be more precise.
- “The next steps are to verify numeric labels independently...” would be stronger as “independently by human listening-based review.”
- “This change does not mean the out-of-domain problem has disappeared.” is effective rhetorically, but a more formal phrasing would fit the paper’s otherwise technical tone.
- AI-content analysis for this 2026 manuscript: I estimate a low-to-moderate probability of partial AI-assisted drafting, roughly 20–35%, but not wholesale AI generation. This estimate is based on several signals: unusually even stylistic consistency, highly compressed declarative prose, recurring formulaic contrast structures, and the explicit use of LLM assistance in data preparation rather than in authorship. At the same time, the paper contains domain-specific methodological detail, non-generic ablation logic, and appropriately self-critical limitations that are less typical of fully generated manuscripts. Sections most suggestive of AI-assisted polishing are the introduction and conclusion, where the prose is especially smooth and generalized, as in “These comparisons are intended to identify which changes matter, not to establish a new general-purpose state of the art” and “The FLEURS experiment shows a limitation of reporting a single WER for code-mixed systems...” Epistemically, even if AI assistance was used for drafting, the impact on the scientific contribution seems limited compared with the more substantive risks arising from LLM-assisted reference rewriting and partially validated label generation. Those latter uses matter more than stylistic authorship assistance.
Recommendations
-
Re-run at least one primary evaluation under a verified speaker-disjoint split. If that is impossible for all data, provide a smaller but clean speaker-disjoint benchmark and treat current in-domain results as provisional.
-
Strengthen validation of the numeric relabeling pipeline with independent human listening checks, inter-annotator agreement, and error typology. This is essential because the manuscript’s central practical contribution depends on label correctness.
-
Fully validate, or substantially de-emphasize, the LLM-rewritten FLEURS reference set. If retained, present it as exploratory unless human verification is completed.
-
Bring all baselines to comparable convergence criteria before making close-performance comparisons, especially against Whisper.
-
Reframe causal interpretations conservatively. Replace strong mechanistic wording like “therefore appears to help recognise the speech itself” with claims more tightly bounded by the evidence unless additional analyses are added.
-
Clarify novelty as a practical systems contribution: fixed-tokenizer adaptation, bilingual number verbalization, and targeted ablation evidence for Nepali–English mixed-script ASR.
-
Improve reproducibility by releasing a fully public mini-benchmark or subset pipeline that reproduces at least the methodological trends, even if the full training corpus cannot be redistributed.
-
Clean the manuscript’s formatting artifacts and sharpen metric definitions early, especially where script mismatch and recognition quality are intentionally disentangled.
Overall, I judge the manuscript to be coherent, technically relevant, and potentially publishable after substantial revision. Its practical contribution is credible and useful, but several headline findings currently rest on evaluation components that the paper itself admits are not fully validated.
Reviewer3 (AI review)
AI Review (reviewer3.com)
Overview
Audio-guided bilingual number verbalization
Contribution
The authors introduce a method to resolve ambiguous numeric scripts in code-mixed speech data by leveraging base ASR predictions and phonetic distance to select between generated English and Nepali spoken candidates without changing fixed tokenizer vocabularies.
Encoder adaptation analysis for code-mixing
Contribution
This study demonstrates that updating the full encoder during fine-tuning substantially outperforms freezing it, reducing word error rates by 8.4 percentage points and showing that acoustic representation adaptation is essential beyond learning output formatting.
Transcription convention impact on evaluation
Contribution
The paper demonstrates that reference script conventions significantly distort out-of-domain evaluation metrics, reversing apparent performance trends when standard single-script references are adjusted to reflect code-mixed script usage.
Comments (1)
1. Table 12 caption cross-references wrong section — Editorial
Table 12: Results on FLEURS Nepali (%). WER CM : WER against references rewritten in our code-mixed convention (Sec. 6.5).
The caption for Table 12 states that the code-mixed FLEURS references are described in Section 6.5 (WER_CM: WER against references rewritten in our code-mixed convention (Sec. 6.5)). However, the rewriting procedure and rules for the mixed-script FLEURS references are defined in Section 5.3 ('Mixed-script FLEURS references'), whereas Section 6.5 only presents the evaluation results.
XSci (AI review)
Summary
The authors adapt SraVaani, a 449M-parameter FastConformer model with joint TDT and CTC heads, to Nepali–English code-mixed speech, using a 172.8-hour corpus from OpenSLR read speech and five YouTube-sourced domains. The obstacle is that the released 5,000-piece SentencePiece vocabulary cannot encode digits and several punctuation marks, affecting 29.7% of source clips (Table 1). Rather than rebuild the tokenizer, the authors rewrite transcripts into encodable strings and resolve ambiguous numbers by generating Nepali and English candidate readings, romanising both the candidates and the base model's zero-shot hypothesis, and keeping the closest match. Full fine-tuning reduces test WER from 48.6% to 17.1% at six epochs and 16.3% at twelve, against 16.4% for a fine-tuned Whisper large-v3-turbo and 19.9% for an adapted IndicConformer; freezing the encoder gives 25.5%, and dropping numeric clips raises WER on numeric test speech from 14.3% to 21.3%. FLEURS is then evaluated under two reference conventions to separate orthographic from acoustic degradation.
Contribution and Significance
The core contribution is a reproducible recipe for adapting a released Indic speech checkpoint to code-mixed speech without touching its vocabulary, together with measurements of which parts matter. Two generalise beyond this language pair. First, the official fine-tuning script freezes the encoder, and the authors show this costs 8.4 WER points (95% CI +8.0 to +8.8), with romanised WER at 23.5% versus 15.5% confirming the gain is acoustic rather than orthographic. Second, the dual-reference FLEURS experiment shows the same systems on the same audio moving from a degradation (22.6% to 31.1%) to an improvement (30.0% to 23.4%) depending only on how English loans are written — a concrete demonstration of a point the code-switching metrics literature has argued in the abstract.
The number-verbalisation procedure is the most original component, but it extends an existing idea rather than introducing one: selecting among candidate verbalisations by matching them against an ASR decode of the same audio is the method of Geneva and Shopov, "Towards Accurate Text Verbalization for ASR Based on Audio Alignment" (RANLP 2019 Student Research Workshop), which should be cited. New here are the two-language candidate set and romanised scoring, which the code-mixed setting requires; the by-source reading statistics (99.0% Nepali in OpenSLR, 75.6% English in lectures) are an informative finding in their own right. Since the corpus cannot be released (Section 8), the lasting contribution is the recipe and the checkpoints rather than the data.
Strengths
- The encoder-adaptation result (Section 6.3, Table 10) is large, tightly bounded and controlled against the obvious confound, and it contradicts the vendor's own recipe.
- The numeric-data ablation is well targeted: removing numeric training clips costs 7.0 points on the 1,125 numeric test clips but only +0.26 points elsewhere (95% CI −0.04 to +0.56), and Table 11 localises the effect to number words (4.9%/6.4% rising to 16.6%/29.8%).
- The diagnostic suite (Section 5.4, Table 9) is well chosen: switch-point error falls from 65.9% to 14.3% against 35.5% to 15.2% elsewhere, showing the switch-point penalty essentially disappears rather than merely tracking the overall rate.
- The data accounting is complete and checkable: Table 4 reconciles 116,175 source clips to 115,107 final clips step by step, and the clip and duration figures in Sections 3.3–3.8 sum correctly.
- The Limitations section names the real weaknesses rather than only the convenient ones, and checkpoints, test outputs and the new FLEURS references are released.
Recommendation
Major Revision for the leading journals of the field. The contribution is real and the encoder result is convincing, but two issues leave the headline numbers in doubt: the test references for numeric clips were chosen using the base model that is itself under evaluation, and the splits are not speaker-disjoint. Both can be resolved inside the authors' own approach, by hand-verifying the numeric test references and re-splitting by source recording.
Major Concerns
-
The numeric test references are produced by the system under test, and the final labels were never audited (Sections 3.6 and 5.2; Tables 4, 10, 11). The label pipeline runs the base SraVaani model on the audio and keeps the candidate reading closest to its romanised output. This is applied to all splits: Table 4 shows 59 tie clips and 13 far clips removed from the test set. On the 1,125-clip numeric subset the fine-tuned system is therefore scored against references its own initialisation helped write, and the cases where the base model was most uncertain have been deleted from the evaluation. Any residual bias runs in SraVaani's favour and against Whisper and IndicConformer, which are scored on the same references. The audit does not close this: 150 numeric clips gave 5 errors (3.3%, 95% CI 1.4–7.6%), but Section 3.6 then reports that revising the candidate rules changed the chosen reading in 539 of 634 affected clips, so the audited labels are not the labels used. Verifying the numeric test references by listening (1,125 clips is feasible) and re-reporting Table 10's Num. column and Table 11 against them would settle it, and would let the tie clips be restored to the test set.
-
The splits are not speaker- or session-disjoint, which conditions every reported number (Section 3.3; Limitations). Deduplication was by audio hash and by duplicated sentence strings, neither of which prevents clips from the same YouTube lecture, podcast episode or news programme appearing in both training and test. Since single recordings were segmented into many 3–20 second clips, overlap is close to certain for the lecture, podcast and news domains, which supply 2,662 of the 4,610 test clips. The effect is not confined to absolute WER: the no-OpenSLR ablation, the numeric ablation and the Whisper comparison are all measured on this split, and session memorisation would inflate the in-domain side of each. The gap between in-domain (17.1%) and FLEURS (31.1%) performance is consistent with such an effect, though not proof of it. The authors list speaker-disjoint splits as future work; for the central claims it is not future work. Re-splitting by source video or speaker and re-running at least the main, frozen-encoder and no-numeric configurations would show how much of the gain survives.
-
The out-of-domain evidence rests on unverified references, and no external code-mixed set is used (Sections 5.2, 5.3, 6.5). The mixed-script FLEURS references were produced by an LLM, changed 390 of 586 sentences, and Section 5.3 states that human verification was not completed; the conclusion that changing the reference writing style "reverses the apparent direction of the result" therefore depends on a reference set whose correctness is unestablished. The complementary half of the claim is better supported, since romanised CER ignoring script and spaces still rises from 5.1% to 7.6%. Separately, FLEURS is read speech with limited English content, so the code-mixing claims are untested on any external data, while Paudel [17], cited in Section 2 as a spontaneous Nepali–English podcast benchmark, is directly suitable and unused. Human verification of the rewritten references, with inter-annotator agreement, plus results on one external code-mixed set would put Section 6.5 and the conclusion on firm ground.
Minor Comments
- Whisper was trained for three epochs with validation still improving while SraVaani received six and twelve (Sections 5.1, 6.7), and the 12-epoch run had also not converged (Section 6.3), so the abstract's "16.3% ... compared with 16.4%" should carry the caveat Section 6.7 states.
- Section 6.1 asserts the SraVaani–Whisper difference "is not statistically significant under our paired bootstrap" without reporting the interval; please give it, as Table 10 does for the ablations.
- In Section 3.6, 45.1% Nepali and 53.1% English readings account for only 98.2% of the 35,157 resolved numbers; state what the remaining 1.8% are.
- The no-numeric ablation also removes 32.8 of 152.9 training hours, so a size-matched random-removal control would strengthen the attribution beyond the already helpful non-numeric contrast.
- Only the main configuration was repeated with a second seed (Table 10); a second seed for the frozen-encoder and no-numeric runs would confirm those larger deltas are stable.
- No inference cost is reported, which weakens the 449M-versus-809M efficiency argument; please add real-time factors alongside the 19.1 GiB peak memory and the roughly three-hour training time.
- IndicConformer was adapted with a newly trained tokenizer and reinitialised decoders while SraVaani kept both (Section 5.1), so the 19.9% versus 17.1% gap conflates model with adaptation protocol and should be described as such.
- The qualitative evidence cannot be checked as typeset: Table 13 contains a single garbled row, Tables 2 and 7 have misaligned columns, and the Nepali particles quoted in Section 6.8 appear in Tamil rather than Devanagari script.
- The CM% and EN% columns of Table 3 sit under the Test column group, but Section 3.8 reports the same values (43.8%, 24.2%) for the whole corpus; please clarify which they describe.
- Since the transcription convention removes punctuation and spells numbers out (Section 3.5), state whether inverse normalisation is provided for the released checkpoints.
- Reference [20] is a technical blog post; please cite the peer-reviewed version, Y. Zhang, E. Bakhturina, K. Gorman and B. Ginsburg, "NeMo Inverse Text Normalization: From Development to Production," Proc. Interspeech 2021, pp. 4468–4472.