Why Yoruba NLP Is Harder Than You Think

When engineers first encounter Yoruba as an NLP target, they often assume the primary challenge is data scarcity — and they are not wrong. But data scarcity is a problem you can brute-force with annotation budgets and synthetic data pipelines. The deeper challenges are structural, and they require rethinking assumptions baked into every standard NLP toolkit.

Tone as Meaning

Yoruba is a tonal language. The word "owo" means hand, money, or broom depending on the tones applied to its vowels. In speech, tones are carried by pitch. In text, they are marked by diacritics — acute, grave, and mid-tone markers over vowels. The critical problem: most Yoruba text on the internet does not include diacritics. Digital communication, especially on mobile keyboards, drops tonal marks almost universally.

This means a corpus scraped from Yoruba social media or news sites is not just low-resource — it is ambiguous at a fundamental level. A sentence without diacritics can have dozens of valid interpretations. A tokeniser trained on such text learns a degraded representation of the language.

Our solution: we built a tone restoration model as a preprocessing step. Given undiacritised Yoruba text, it predicts the most probable diacritics using context. We trained it on a parallel corpus of diacritised academic and literary texts paired with stripped versions, achieving 94.3% character-level tone accuracy on held-out test data. This became the foundation layer for everything downstream.

Morphological Complexity

Yoruba is an isolating language with relatively low morphological complexity compared to Bantu languages — but it compensates with complex nominal and verbal serialisation. Serial verb constructions chain multiple verbs into a single predicative unit sharing arguments. Standard dependency parsers trained on Indo-European languages consistently misparse these constructions.

We found that token-level representations from mBERT were significantly better than fastText or word2vec for Yoruba, because the contextual embeddings partially capture serialisation patterns. However, we needed to fine-tune on Yoruba-specific dependency trees — of which approximately 1,200 exist in the Universal Dependencies treebank for Yoruba, compared to over 50,000 for English.

The Orthographic Standardisation Problem

Unlike French or German, Yoruba has no single dominant orthographic standard. Academic publications, government documents, and popular media each use slightly different conventions for representing certain phonemes. When building training corpora by aggregating across sources, these inconsistencies create noise that degrades model performance.

We addressed this with a normalisation layer applied before tokenisation, mapping variant spellings to a canonical form based on the International Yoruba Alphabet standard. This reduced out-of-vocabulary rates by 31% on our legal domain test set.

What the Field Needs

The Yoruba NLP ecosystem needs three things above all else: a large-scale diacritised corpus (we are working on open-sourcing ours), standardised evaluation benchmarks for legal and government domains, and consistent funding for the annotation work that no foundation model can replace. The computational challenges are tractable. The data infrastructure challenges require community coordination and sustained investment.

TNG is committed to publishing our Yoruba preprocessing toolkit under an open-source licence. Watch this space.