Diving into the World of Spell Checks - Niharika - CCDays
0 likes3,046 views
This document discusses spell checking techniques. It begins by introducing the speaker and their background in machine learning and natural language processing. It then discusses how spell checkers work by pointing out errors and suggesting alternatives using techniques like phonetics, edit distance, and symmetric delete spelling correction. It covers specific algorithms like Soundex, Levenshtein distance, Jaro-Winkler distance and how they calculate string similarity. Finally, it discusses the symmetric delete spelling correction algorithm used in the Symspellpy package and how it works.
![>>>from nltk.metrics import edit_distance
>>>edit_distance(“rain”,“shine”)
3
>>>b = TextBlob(“I havv goood speling”)
>>>print(b.correct())
I have good spelling!
>>>from spellchecker import SpellChecker
>>>spell = SpellChecker()
>>>misspelled = spell.unknown([“cmputr”, “study”,“watr”])
>>>for word in misspelled:
>>> print(spell.correction(word))
>>> print(spell.candidates(word))
computer
{'caput', 'caputs', 'compute', 'computor', 'impute', 'computer'}
water
{'water', 'watt', 'warr', 'wart', 'war', 'wath', 'wat'}](https://image.slidesharecdn.com/ccdays-divingintotheworldofspellchecks-200619044417/85/Diving-into-the-World-of-Spell-Checks-Niharika-CCDays-6-320.jpg)
![Algorithms
➢ HAMMING DISTANCE
○ SUBSTITUTION
○ Only applies to strings of the same length
➢ JARO
○ TRANSPOSITION + Matching Characters
○ Range [0,1] : 0 - Least Similar, 1 - Most Similar
➢ JARO-WINKLER
○ TRANSPOSITION + Matching Characters + Prefix
○ Uses a prefix scale of ‘p’ which gives more favourable ratings to strings that match from the
beginning for a set prefix length](https://image.slidesharecdn.com/ccdays-divingintotheworldofspellchecks-200619044417/85/Diving-into-the-World-of-Spell-Checks-Niharika-CCDays-14-320.jpg)
Diving into the World of Spell Checks - Niharika - CCDays
- 1. DIVING INTO THE WORLD OF SPELL CHECKS! Niharika Krishnan Cloud Community Days Conference - 19 June 2020
- 2. Before we get started…. NIHARIKA KRISHNAN ➢ Machine Learning Engineer, TCS ○ Build Chatbots for a living! ➢ Founding Member of PyLadies Chennai ○ Community of 100+ women tech enthusiasts ➢ Speaker ○ PyCon Canada’19, India’19 ○ Google Women Techmakers, Global Diversity CFP ➢ AI and NLP enthusiast
- 3. ~ 20 mails in a day 269 Billion Emails in 2019
- 6. >>>from nltk.metrics import edit_distance >>>edit_distance(“rain”,“shine”) 3 >>>b = TextBlob(“I havv goood speling”) >>>print(b.correct()) I have good spelling! >>>from spellchecker import SpellChecker >>>spell = SpellChecker() >>>misspelled = spell.unknown([“cmputr”, “study”,“watr”]) >>>for word in misspelled: >>> print(spell.correction(word)) >>> print(spell.candidates(word)) computer {'caput', 'caputs', 'compute', 'computor', 'impute', 'computer'} water {'water', 'watt', 'warr', 'wart', 'war', 'wath', 'wat'}
- 7. What happens under the hood ?
- 8. Spell-checks ➢ Spell Checker points to spelling errors and possibly suggests alternatives ➢ Autocorrector automatically picks the most likely word ➢ Types: ○ PHONETICS ○ EDIT DISTANCE (Peter Norvig) ○ SYMMETRIC DELETE SPELLING CORRECTION (SymSpell) ➢ Real word Errors vs Non-Word Errors
- 9. Phonetics ➢ Detect similar-sounding words even if they are spelt differently like Smith & Schmidt ➢ Creates a specific phonetic representation of a single word ➢ Algorithms: ○ SOUNDEX ○ METAPHONE
- 10. Soundex
- 11. Edit Distance ➢ Quantifying how dissimilar two strings are to one another ➢ Minimum number of edit operations required to transform s1 into s2 ○ Insertion, Deletion ○ Substitution, Transposition
- 12. Algorithms ➢ LEVENSHTEIN ○ Insertion + Deletion + Substitution ○ RECIEVE → RECEIVE → Edit Distance = 2 ○ RECEIVE → RECEIPT → Edit Distance = 2 ○ Very different semantically and context ➢ DAMERAU - LEVENSHTEIN ○ Insertion + Deletion + Substitution + Transposition ○ Character swapping ➢ LEAST COMMON SUBSEQUENCE ○ Insertion + Deletion
- 13. Kitten → Sitten (substitute "s" for "k") Kitten → itten (delete "k" at 0) sittEn → sittIn (substitute "i" for "e") itten → Sitten (insert "s" at 0) sittin → sittinG (insert "g" at the end) sittEn → sittn (delete "e" at 4) sittn → sittIn (insert "i" at 4) sittin → sittinG (insert "g" at 6) Levenshtein vs Longest Common Sequence
- 14. Algorithms ➢ HAMMING DISTANCE ○ SUBSTITUTION ○ Only applies to strings of the same length ➢ JARO ○ TRANSPOSITION + Matching Characters ○ Range [0,1] : 0 - Least Similar, 1 - Most Similar ➢ JARO-WINKLER ○ TRANSPOSITION + Matching Characters + Prefix ○ Uses a prefix scale of ‘p’ which gives more favourable ratings to strings that match from the beginning for a set prefix length
- 15. Symmetric Delete Spelling Correction ➢ Delete-only edit candidate generation ➢ 5 letter word → 3 Million Possibilities vs 25 Possibilities (Edit Distance: 3) INSERTION delete (dictionary entry,edit_distance) input entry goa delete(goal,1), delete(goat,1) goa DELETION dictionary entry delete(input entry,edit_distance) goall goal delete(goall,1) SUBSTITUTION & TRANSPOSITION delete(dictionary entry,edit_distance) delete(input entry,edit_distance) goal delete(goal,1), delete(goat,1) delete(goak,1) 1 Million times faster
- 16. ➢ Verbosity parameter: ○ Top: highest term frequency + smallest edit distance ○ Closest: smallest edit distance found, ordered by term frequency ○ All: All suggestions within maxEditDistance, ordered by edit distance, term frequency ➢ maxEditDistance ➢ Word frequency dictionary: ○ LoadDictionary ○ CreateDictionary (Customize it for your use-case!) Symspellpy Let’s see how symspell works!
- 17. QUESTIONS niharikakrishnan linkedin.com/in/niharikakrishnan @Nihaaarika Slide Deck: https://github.com/niharikakrishnan/Talks Want to explore further? Let’s connect!