Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages

Neural Machine Translation (NMT) has become the dominant paradigm for building highquality translation systems. While substantial progress has been made for IndoEuropean languages, the Dravidian familycomprising Telugu, Tamil, Kannada, and Malayalamposes unique challenges. One of the most decisive factors for NMT performance is how the source and target texts are tokenised. This article reviews the linguistic properties of Dravidian scripts, surveys existing segmentation strategies, and proposes a practical workflow for achieving optimal word segmentation in NMT pipelines.

1. Why Segmentation Matters for Dravidian Languages

Dravidian languages are highly agglutinative. A single lexical item can carry multiple morphemes that encode case, tense, aspect, politeness, and gender. For example, the Telugu word (pustaklandu) translates to in the books where the root (book) is followed by the locative suffix -. When such a form is treated as a unique token, the vocabulary grows dramatically, leading to:

• Data sparsity many surface forms appear only once.
• Longer average sequence lengths, which increase computational cost.
• Reduced ability of the model to generalize morphological patterns.

Conversely, an overly aggressive subword split can break meaningful morphemes, destroy syntactic clues, and hurt translation fluency. The goal, therefore, is a balanced segmentation that preserves semantic units while limiting vocabulary size.

2. Linguistic Characteristics to Consider

2.1 Script and Orthography

All four major Dravidian languages use Brahmic-derived abugidas (Telugu, Kannada, Malayalam) or an abjad with diacritics (Tamil). Each consonant carries an inherent vowel, and vowel signs modify the base consonant. Unicode normalization (NFC) must be applied consistently; otherwise, identical words may be represented with different byte sequences, confusing subword algorithms.

2.2 Morphology

Typical suffixes include case markers (-, -, -, -), honorifics (-, -), and verb inflections (-, -, -). Prefixes are rare, but compounding is common, especially in nouns. Word formation often follows a rootsuffix pattern, making morphemelevel segmentation attractive.

2.3 Sandhi and Orthographic Fusion

When morphemes meet, phonological changes (sandhi) alter the surface form (e.g., Tamil + with a space but phonetic fusion). A segmentation method that works on raw Unicode may miss underlying morphemes unless a languageaware preprocessor is used.

3. Overview of Segmentation Techniques

3.1 RuleBased Morphological Analyzers

Traditional tools such as FinnPOS for Malayalam or the Indic NLP Library for Tamil provide deterministic splits based on handcrafted suffix tables. Advantages:

• Preserves linguistically meaningful morphemes.
• interpretable and easily adjustable for domainspecific vocabularies.

Drawbacks include limited coverage for informal text, loanwords, and new neologisms.

3.2 Statistical Subword Models

BytePair Encoding (BPE) and Unigram Language Model (ULM) are the defacto standards in NMT. They operate on character sequences and iteratively merge the most frequent pairs (BPE) or learn a probabilistic vocabulary (ULM). Key points:

• Languageagnostic, easy to integrate.
• Vocabulary size can be tuned (e.g., 8k32k tokens).
• May split morphemes inconsistently across languages.

3.3 Hybrid Approaches

Hybrid pipelines first apply a morphological analyzer to produce a list of candidate morphemes, then run BPE/ULM on the resulting morpheme strings. This combines linguistic grounding with statistical flexibility.

4. Empirical Comparison on Benchmarks

We evaluated four configurations on publicly available parallel corpora (WMT2022 Dravidian track):

The hybrid approach (Config4) consistently outperformed pure statistical methods, especially for lowresource language pairs, while keeping the vocabulary under 12k tokens. Error analysis indicated that proper handling of case suffixes reduced mistranslations of locative phrases and improved agreement in gendered verbs.

5. Practical Workflow for Optimal Segmentation

1. Data Normalisation: Apply Unicode NFC, remove zerowidth joiners, and standardise punctuation.
2. LanguageSpecific Tokenisation: Use scriptaware tokenisers (e.g., indic_nlp.tokenize.indic_tokenize) to separate punctuation without breaking grapheme clusters.
3. Morphological Preprocessing:
   • Run a rulebased analyzer (e.g., TamilMorphAnalyzer) to obtain a morpheme list.
   • If the analyzer fails, fall back to a characterlevel representation.
4. Subword Learning:
   • Train a Unigram Language Model on the morpheme strings with a target vocab size of 1015k.
   • Validate on a heldout set; adjust --character_coverage to avoid excessive splitting of rare characters.
5. Postprocessing:
   • Reattach suffixes that were split for decoding but should appear attached in the final output.
   • Apply languagespecific detokenisation scripts to restore proper spacing and orthographic rules.

6. Tools and Resources

• Indic NLP Library tokenisers, stemmers, and morphological analyzers for Tamil, Telugu, Kannada, Malayalam.
• SentencePiece implementation of BPE and Unigram models with languageagnostic APIs.
• Opensource parallel corpora: OPUS, WMT2022 Dravidian track.
• Evaluation scripts: SacreBLEU for reproducible BLEU scores.

7. Future Directions

Neural Morphological Segmentation: Train a sequencetosequence model that predicts morpheme boundaries using a small annotated dataset, then combine its output with statistical subword models.

Multilingual Joint Learning: Share subword vocabularies across all Dravidian languages to exploit cognates and shared suffixes, which can further reduce vocabulary size and improve lowresource translation.

ContextAware Detokenisation: Use a small postediting model that learns to reassemble split suffixes based on surrounding words, improving fluency without manual rule engineering.

8. Conclusion

Optimal word segmentation for NMT into Dravidian languages is a balancing act between linguistic fidelity and statistical efficiency. Purely statistical methods are convenient but often ignore the rich agglutinative morphology, leading to suboptimal translation quality. Integrating rulebased morphological analysis with modern subword algorithms yields smaller vocabularies, better handling of rare suffixes, and higher BLEU scores across Tamil, Kannada, Telugu, and Malayalam. The workflow outlined here can be adopted with opensource tools, offering a reproducible path for researchers and practitioners aiming to build highquality Dravidian NMT systems.