![Bayesian Classifier e-mail message as a vector of N features X = X 1 , X 2 , X 3 , ..., X N - For example, X 42 might be ‘the e-mail contains “ money ”’ - x 42 =0 means “the message described by x does not contain the “ money ”’. classify messages in K classes C = {c 1 , c 2 } = {junk, legit} (K=2) Now suppose we see a new e-mail message, with encoding x . We seek the probability that the class C is junk, Pr[ C =junk | X = x ] shorthand for Pr[ C =junk | X 1 =x 1 & X 2 =x 2 & … & X N =x N ]](https://image.slidesharecdn.com/mailfilterppt1404/85/mailfilter-ppt-32-320.jpg)
mailfilter.ppt
- 1. E-Mail Filtering Soonyeon Kim
- 2. Good Site for Data Mining http://liinwww.ira.uka.de/bibliography/ - The Collection of Computer Science Bibliographies Major Conferences in Data Mining - KDD 2000 of ACM SIGKDD http://www.acm.org/sigs/sigkdd/kdd2000/ - SIGMOD 2000 of ACM SIGMOD Other Conferences - VLDB, IEEE ICDE, PAKDD conference
- 3. Text Mining: Finding Nuggets in Mountains of Textual Data Author - Jochen Dorre, Peter Gerstl, Roland Seiffert - {doerre,gerstl, seiffert }@de.ibm.com Method to find this paper - Searching from The Collection of Computer Science Bibliographies - key word used : Data mining & Text classification
- 4. Brief Description What is Text Mining? - same analytical functions of data mining to the domain of textual information. How Text mining differs from Data mining? - Data mining : addresses a very limited part of data (structured information available in database) - Text mining : helps to dig out the hidden gold from textual information & requires the very complex feature extraction function Describe in more detail the unique technologies that are key to successful text mining
- 5. Ifile: An Application of Machine Learning to E-mail Filtering Author - Jason D. M. Rennie Artificial Intelligence Lab, MIT - jrennie@ai.mit.edu Method to find this paper - KDD 2000 of ACM SIGKDD
- 6. Outline of Paper Introduction - need for automated e-mail filtering - Ishmail - important issues regarding mail filtering Mail Filtering - Classification Efficiency - Features - Naïve Bayes algorithm IFILE Experiment Conclusion
- 7. Introduction Popular E-mail clients allow users to manage their mail into folders by meaningful topic - popular e-mail client : Netscape Messenger, Pine, Microsoft Outlook, Eudora and EXMH Ishmail - purpose of a prioritization system - alert the user when high-priority mail is arrived or a large number of messages have accumulated in a lower-priority folder Barrier - implementation for mail filters (speed efficiency, database size, collection of supervised training data) - integration into e-mail clients
- 8. Classification Efficiency Traditional classification method - kNN, C4.5, Naïve Bayes Recent development - SVM (Support Vector Machine), Maximum Entropy discrimination) Efficiency Problems - SVM and MEM provide significant improvement in accuracy, but at the cost of simplicity and time efficiency - kNN : time to classify
- 9. Classification Efficiency(2) Naïve Bayes - efficient training, quick classification and extensibility to iterative learning - training : updating word counts - classification : normalized sum of the counts corresponding to the words in the document in question
- 10. Personal E-mail Filtering Every user has a unique collection of e-mail User organizes their e-mail in unique way It pertains directly to his preference Key fact for effective personal e-mail filtering - using the information made through the user interface of the mail client
- 11. Learning Architecture Label is assigned to newly filtered e-mail message Added to the classification model Update the classification model : every filtered e-mail is a training example - assumed to be correct if user does not move the message to another folder - update the model if user moves misclassified mail into the appropriate folder Update for Naïve Bayes - shift word counts from one folder to another
- 12. Features Classification model act as a function f F C - F : Features C : class Mail filter is a special classifier F C - F : characteristics of e-mail message C : mail folder - by considering each e-mail message as a bag of words function f maps an unordered set of words to a folder name f
- 13. Features(2) Naïve Bayes keeps the track of word frequency statistics Reduce the number of features for classification to make filtering more efficient Feature selection cutoff - old, infrequent words are dropped - word that occur fewer than log(age)-1 times should be discarded from the model - age : number of e-mail messages added to the model since statistic has been kept for that word e.q. if “baseball” occurred in the 1 st document and occurred 5 or fewer times in the next 63 document, the word and statistics would be eliminated from database.
- 14. Maintaining Dictionary Cutoff Algorithm - word that occur fewer than log(age)-1 times should be discarded from the model e.q. “datamining” occurred in the 1 st document 63 documents are coming after the document age = 1 + 63 = 64 log(age) – 1 = 5 if “datamining” appears 5 or fewer times, the word and statistics would be eliminated from database.
- 15. Maintaining Dictionary(2) -----------.idata------------- A B C list of folders(A:0 B:1 C:2) 5 2 6 total word instances 2 1 1 # of message party 4 0:2 1:1 belch 3 0:1 yellow 4 0:2 2:3 word age folder:frequency kick 2 1:1 peep 1 2:2 two msg in A - "party party belch yellow yellow" one msg in B - "party kick" one msg in C - "peep peep yellow yellow yellow"
- 16. Word Selection Header Trimming E-mail Body: content Header : list of fields pertaining to the message From: To: Subject: - keep this part Received: Date: Message-id - remove
- 17. Tokenizing text Two techniques Using stop list - decrease the amount of noise in the data by eliminating uninformative words e.g) pronoun, modifier, adverb Stemming - link together words which have the same root e.g) serve, service, serves, served => same root serv
- 18. Naïve Bayes What is Naïve Bayes? - Simple, yet effective classifier of text documents - Statistical Machine learning algorithm Assumption each document is considered as a set of words Each word is independent
- 19. Naïve Bayes-1 st step Probability of d having been generated by c i - With the assumption that attribute values are independent,
- 20. Naïve Bayes-second step Computing P(c i |d) for all classes Find the class to be classified Maximum likelyhood - Probability values are only used for comparison Purpose, P(d) can be dropped
- 21. Naïve Bayes M-estimate purpose : to give a reasonable probability in the case of sparse data -n j : number of instances of w j in the documents of class c i -n : total number of words in documents of class c i -|Vocabulary| : number of distinct words
- 22. Experimental Result Information about the e-mail corpora on which classification experiments were conducted. Four volunteers including author
- 24. Experimental Result Individual Experiments with different setting Alpha lexer, stoplist used, header trimming, feature selection, no stemming Alpha only lexer replaces alpha lexer White lexer replaces alpha lexer No stoplist is used Stemming is used No feature selection is used All headers are used for classification purposes
- 25. Experimental Result Three Lexers Alpha lexer - default lexer - tokenizes strings of alphabetic characters Alpha only lexer - tokenizes only strings of alphabetic characters - does not lex e-mail addresses into tokens White lexer - tokenizes strings separated by whitespace
- 26. Experimental Result Result - No experimental environment setting provide the best results across all users Experiment with highest average accuracy - experiment #1 shows the best average result (89% accuracy) - ranging from 86% to 91%
- 27. Experimental Result Time Efficiency - Naïve Bayes : “fast enough” - 27 seconds to build a model of 7000+ e-mail messages (average 259 msg/second) (tar-gzip of same msg requires 17 seconds) Space Efficiency - classification model built on 7000+ messages across 49 folders requires only 447,090 bytes
- 28. Filtering Junk E-Mail Soonyeon Kim
- 29. A Bayesian Approach to Filtering Junk E-Mail Authors - Mehran Sahami, Susan Dumais, David Heckerman, Eric Horvitz - Stanford University & Microsoft Researh From - AAAI 98 (American Association for Artificial Intelligence
- 30. Problems of Junk-mail Wasting time - Many users must now spend a non-trivial portion of their time because of unwanted messages Content of Material - Some of these messages can contain offensive material such as graphic pornography Space problem -Junk-mails also quickly fill up file server storage space
- 31. Machine Learning Approach Learning - system S learns from experience E with respect to a class of tasks T and performance P Learning in junk-mail S : E-mail classifier T : classify an e-mail message as junk/legitimate P : fraction of correct prediction E : a set of pre-classified e-mail messages Vector Space Model - to represent mail messages as feature vectors - e-mail message has single fixed-length feature vector - individual message can be represented as a binary vector denoting which word are present or absent. (1 for present 0 for absent)
- 32. Bayesian Classifier e-mail message as a vector of N features X = X 1 , X 2 , X 3 , ..., X N - For example, X 42 might be ‘the e-mail contains “ money ”’ - x 42 =0 means “the message described by x does not contain the “ money ”’. classify messages in K classes C = {c 1 , c 2 } = {junk, legit} (K=2) Now suppose we see a new e-mail message, with encoding x . We seek the probability that the class C is junk, Pr[ C =junk | X = x ] shorthand for Pr[ C =junk | X 1 =x 1 & X 2 =x 2 & … & X N =x N ]
- 33. Bayesian networks (a) a Naïve Bayesian classifier (b) a more complex Bayesian classifier with limited dependencies between the features
- 34. Bayesian Rule Bayes theorem ssume that each Xi is independent
- 35. Features Words - fixed width vector <X = X 1 , X 2 ,…, X n > Hand-crafted Phrasal Features - “FREE!”, “only $” ( as in “only $4.95”) and “be over 21” - 35 such hand-crafted phrases are included Domain-specific features - domain type of sender (.edu or .com) - junk mail is usually not from .edu domain Resolving familiar E-mail address - i.e. replace sdumais @ microsoft .com with Susan Dumais Time - most junk E-mail is sent at night
- 36. Features(2) Peculiar punctuation - percentage of non-alphanumeric characters in the subject of a mail - “$$$$$ MONEY $$$$$” X : subject has peculiar punctuation Y : pct of total messages
- 37. Feature Selection Mutual Information - Mutual information MI(A,B) is a numeric measure of what we can conclude about A if we know B, and vice - versa. - Example: If A and B are independent, then we can’t conclude anything: MI(A, B) = 0 - Select 500 features with greatest value
- 38. Evaluation Three ways Using Domain-specific Features - Words only - Words + Phrases - Words + Phrases + Extra Features Three way Categorization - 3 categories {porn-junk, other-junk, legit} instead of 2 categories {junk, legit}. “ Real” scenario
- 39. Using different features The cost of missing legitimate email is much higher than the costing of inadvertently reading junk. The authors wanted to make their system very “ optimistic ” so that it only predicts “junk” if it is very certain -- uses threshold 99.9%. 1789 hand-tagged e-mail messages 1578 junk 211 legit Split into… 1538 training messages (86%) 251 testing messages (14%)
- 40. Using different features Result of experiment - words only - words + 35 phrasal features - words + phrasal features + 20 non-textual domain-specific features
- 41. Using different features Junk Precision = A / (A + C) Junk Recall = A / (A + B) Legit Precision = D / (D + B) Legit Recall = D / (D + C) Junk precision is of greatest concern to most users, because they would not want their legitimate mail discarded as junk D (true negatives) C (false positives) actually Legit B (false negatives) A (true positives) actually Junk predict Legit predict Junk
- 42. Using different features Precision/ Recall curves for junk mail
- 43. Sub-classes of junk E-mail Three way Categorization - 3 categories {porn-junk, other-junk, legit} instead of 2 categories {junk, legit} Consider that classifying is correct if any “junk” messages is classifed either “porn-junk” or “other-junk” Unfortunately, it didn’t work! - Probably because more parameters means need (exponentially!) more data to estimate them accuractely - some feature may be very clearly indicative of junk versus legitimate, but may not be powerful in three categories (they do not distinguish well between the sub-classes of junk
- 44. Sub-classes of junk E-mail Precision/recall curves considering sub-groups of junk mail
- 45. Real Usage Scenario Three kinds of messages 1. Read and keep 2. Read and discard (ex. Joke from a friend) 3. Junk Result Misclassified mails – news stories from a e-mail news service that the user subscribes to. (No loss of significant information)
- 46. Real Usage Scenario Precision/recall curves in real usage scenario