A Hybrid Approach to Design Automatic Spelling Corrector and Converter for Transliterated Bangla Words
Debnath, T., Sajnin, S., & Hamid, M. M.
23rd International Conference on Computer and Information Technology (ICCIT), IEEE · 2020
Transliterated Bangla ("Benglish") — Bangla written with the Latin alphabet — has no spelling rules, so social media text is full of deviated, ambiguous words. We built a hybrid spelling corrector and converter that combines dictionary lookup, the Damerau–Levenshtein edit distance, and linear search; the best single algorithm corrected 87 of 100 deviated words, and the hybrid handled words that every individual algorithm failed on.
The problem
Transliterated Bangla users are increasing rapidly because the style is convenient on Internet platforms, but the writing style has no grammatical rules or obligations. Typographical errors, intentional vowel omission to shorten words (e.g., "kmn" for "kemon"), short forms, and extra letters ("Naaaa" for "Na") make transliterated Bangla hard to read — and there is no official dictionary of correct spellings.
What we did
We propose a hybrid model with three parts: error detection using a dictionary-lookup approach over a "spelling book" corpus mapping core words to their deviated forms; error correction combining the Damerau–Levenshtein minimum edit distance algorithm with linear search; and conversion of corrected transliterated words to pure Bangla script using linear search.
To choose the correction algorithm, we benchmarked 14 algorithms from the TextDistance library — edit-based, token-based, and sequence-based — on a set of 100 deviated words collected from various Internet platforms, then re-tested the four best on 100 words with complicated variations. The full pipeline handles words, sentences, and documents end to end: "Amii Vaat Khaboo" is corrected to "ami bhat khabo" and converted to Bangla script.
Key results
- Damerau–Levenshtein was the best-performing algorithm, correcting 87/100 deviated words — and 85/100 words with complicated variations, ahead of LCSSI (78), Ratcliff–Obershelp (77), and LCS (75).
- Combining Damerau–Levenshtein with linear search over the deviated-word corpus handled the maximum number of deviated words, including words every individual algorithm had failed on (e.g., "hatobhagga" → "hotobhaga").
- The linear-search conversion model worked correctly for the core words of the corpus, converting corrected transliterated words to pure Bangla script.
- The main limitation is the dataset: transliterated Bangla words can have more than one acceptable spelling, and a richer deviated-word collection would further improve detection and correction.
From the paper
