Fuzzy Hash Map
Download as PPTX, PDF5 likes4,847 views
The document discusses the implementation of an efficient fuzzy search enabled hash map developed by Vasile Topac, a PhD student. It highlights the limitations of conventional hash maps that only retrieve exact matches, presenting a fuzzy hash map that uses methodologies from soft computing to find approximate matches. The effectiveness of the fuzzy hash map is validated through accuracy and speed tests demonstrating its capability to handle uncertain data, along with suggestions for future enhancements.
Fuzzy Hash Map
- 1. Efficient Fuzzy Search Enabled Hash MapVasile TopacPhD StudentDepartment of Information Technology and Computer Science“Politehnica” University Of TimisoaraEmail: vasile.topac@aut.upt.ro4th International Workshop On Soft Computing Applications SOFA2010 – Arad, ROMANIA
- 2. How it all started&SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 3. Java HashMap widely used Java data structure
- 4. stores (key, value) pairs
- 6. very fast
- 7. a hash function generates a hash code for indexation
- 8. Uses equals method to compare trough the keys
- 9. only values for existing keyscan be retrieved SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 10. Java HashMapphone book exampleSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 11. Java HashMapCollisionSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 12. Java HashMapSearch for “Lisa Smith”hashMap.get(“Lisa Smith”);Result: “521-8976”SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 13. Problemonly values for existing keyscan be retrieved Search for “Lissa Smith”hashMap.get(“Lissa Smith”);Result: nullSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 14. Problem search for “Lissa Smith”Brute force solution: - iterate trough the set of entries and search approximate matches Works, but is time expensiveFuzzy data structures – currently available for database SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 15. Fuzzy Hash Map“ Soft computing (SC) is a collection of methodologies that are trying to cope with the main disadvantage of the conventional (hard) computing: the poor performances when working in uncertain conditions. ”SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 16. Fuzzy Hash MapUML Class DiagramSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 17. Fuzzy Hash MapHow it worksFuzzyKey overridden methods hashCode()
- 18. prehashing- create collisions to cluster data
- 19. substring substring(“Fuzzy Search”, 0, 4) = “Fuzz”
- 20. soundexsoundex(“Fuzzy Search”) = F226
- 21. equals(Object o)
- 22. string metrics
- 23. Levenshtain DistanceLD(computing, computation)=4
- 24. Hamming DistanceHD(computing, computers)=3SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 25. Fuzzy Hash MapExample(law terminology dictionary) hashCode()
- 26. prehashing
- 27. substring 4
- 28. equals(Object o)
- 29. Levenshtain DistanceSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 30. Fuzzy Hash Map“the judge has the option of either adjudicating you as guilty or..”fuzzyHashMap.get(“adjudicating”) = nullfuzzyHashMap.getFuzzy(“adjudicating”, 2) = “a decision or sentence imposed by a judge…” hashCode()substring 4 = “adju” equals(Object o)LD(adjudicating, adjudication) = 2SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 31. Fuzzy Hash MapfuzzyHashMap.getFuzzy(“violent”)= “violence”LD(violent, violence) = 2LD(violent, violation) = 5“violence” is returnedSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 32. Fuzzy Hash MapExample(phone book) hashCode()
- 33. prehashing
- 34. soundex
- 35. equals(Object o)
- 36. Levenshtain DistanceSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 37. ResultsAccuracy TestTest conditions- Substring(0,4) hashing function- Levenshtein Distance fuzzy matching algorithm Distance threshold value 2
- 38. medical terminology dictionary populated with 1030 English medical termsTest resultsParse text from American Family Physicians Journal
- 39. text of 568 words
- 40. 43 words identified as medical terms
- 41. 9 were incorrect matches
- 42. 80% accuracy
- 43. Parse text from eMedicine web site
- 44. text of 2730 words
- 45. 260 were recognized
- 46. 7 were incorrect matches
- 47. 97% accuracySOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 48. ResultsSpeed TestExact matches onlySOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 49. ResultsSpeed TestFuzzy matches onlySOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 50. ResultsSpeed TestExact & fuzzy matchesSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 51. ConclusionFuzzyHashMap data structures proved to have very good performance on working with uncertain data
- 52. Flexible (can choose different pre-hashing functions and string metrics)
- 53. available as open source http://fuzzyhashmap.sourceforge.net/
- 54. community can extend the functionality
- 55. Future work:
- 56. adding more string metrics
- 58. implement Fuzzy TreeMapSOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac
- 59. Thank you!sources at:http://fuzzyhashmap.sourceforge.netcontactvasile.topac@aut.upt.ro SOFA2010 – Arad, ROMANIA - Efficient Fuzzy Search Enabled Hash Map - VasileTopac