11/04 Regular Meeting: Monority Report in Fraud Detection Classification of Skewed Data

Minority Report in Fraud Detection: Classification of Skewed Data Clifton Phua, Damminda Alahakoon, and Vincent Lee SIGKDD 2004 Reporter: Ping-Hua Yang

Abstract This paper propose an innovative fraud detection method to deal with the data mining problem of skewed data distributions. This method uses Back-propagation, together with Naïve Bayesian and C4.5 algorithms on data partitions derived from minority over-sampling with replacement. This paper compares the new fraud detection method against C4.5 trained using under-sampling, over-sampling, and SMOTE without partitioning. The most interesting find is confirming that the combination of classifiers to produce the best cost savings has its contributions from all three algorithms.

Outline Introduction Fraud detection Experiments Results Discussion Conclusion

Introduction Fraud, or criminal deception, will be a costly problem for many profit organizations. Data mining can minimize some of these losses by making use of the massive collections of customer data. However fraud detection data being highly skewed or imbalanced in the norm. There are two typical way to proceed when faced with this problem. The first approach is to apply different algorithms. The second approach is to manipulate the class distribution.

Introduction This paper introduces the new fraud detection method for skewed data. The innovative use of NB, C4.5, and BP classifier to process the same partitioned numerical data has the potential of getting better cost saving. The selection of the best classifier of different algorithms using stacking and the merger of their predictions. One related problem caused by skewed data includes measuring the performance of the classifiers. Success can’t be defined in terms of predictive accuracy because the minority class in the skewed data usually has a significantly higher cost.

Fraud detection Existing fraud detection methods The new fraud detection method Fraud detection algorithms

Existing Fraud detection methods Insurance fraud The hot spot methodology applies a three step process: the k-means for cluster detection, the C4.5 for decision tree rule induction, and domain knowledge , statistical summaries and visualization tools for rule evaluation. [Williams G, Hung Z,1997] Expanded the hot spot architecture to use genetic algorithm to generate rules and to allow the domain user. [Williams G, 1999] Credit Card Fraud The Bayesian Belief Network (BBN) and Artificial Neural Network (ANN) comparison the STAGE algorithm for BBN and BP algorithm for ANN in fraud detection. [Maes S, Tuyls K, Vanschoenwinkel B, Manderick B, 2002]

Existing Fraud detection methods Telecommunications Fraud The advanced security for personal communications technologies (ASPECT) research group focuses on neural networks to train legal current user profiles that store recent user information and user profile histories that store long term information to define normal patterns of use. [Weatherford M, 2002]

The New Fraud detection method The idea is to simulate the book’s Precrime method of precogs and integration mechanism with existing data mining methods and techniques.

Fraud detection algorithms This study provides a slight variation of cross validation. Instead of using ten data partition, an odd-numbered eleven data partitions. Bagging combines the classifiers trained by the same algorithm using unweighted majority voting on each example or instance. Stacking combines multiple classifiers generated by different algorithms with a meta-classifier. To classify an instance the base classifiers from the tree algorithms present their predictions to the meta-classifier which then makes the final prediction. This paper propose Stacking-bagging which is a hybrid technique. To train the simplest learning algorithm first, followed by the complex ones.

Experiments Data Understanding Cost Model Data Preparation Modeling

Data Understanding The available fraud detection data set in automobile insurance is provided by Angoss KnowledgeSeeker software. This paper split the main data set into a training data set and a scoring data set. The Class labels of the training data are known, and the training data is historical compared to the scoring data. This data set contains 11338 examples from January 1994 to December 1995 (training data), and 4083 instances from January 1996 to December 1996 (scoring data). It has a 6% fraudulent and 94% legitimate distribution The original data set has 6 numerical attributes and 25 categorical attributes

Cost Model This cost model has two assumptions All alters must investigated. The average cost per claim must be higher than the average cost per investigation. In 1996, the average cost per claim for the score data set is approximated at USD$2,640.

Cost Model The evaluation metrics for the predictive models on the score data set to find the optimum cost savings are:

Data preparation In a related study, it is recommended that data partitions should neither be too large for the time complexity of the learning algorithms nor too small to produce poor classifiers. Randomly select different legal examples from the years 1994 and 1995 (10840 legal examples) into eleven sets of y legal examples (923). x fraud examples (615) with a different set of y to form eleven x:y partitions (615:923) with a fraud:legal distribution of 40:60. Other possible distributions are 50:50 (923:923), and 30:70 (396:923). Minority over-sampling with replacement/replication. In rotation, each data partition of a certain distribution is used for training, testing and evaluation. A training data partition is used to come up with a classifier, a test data partition to optimize the classifier’s parameters and an evaluation data to compare the classifier with others.

Data preparation Test The algorithm trained on partition 1 to generate classifier 1. The algorithm tested on partition 2 to refine the classifier. The algorithm evaluation on partition 3 to assess the expected accuracy of classifier.

Modeling In figure 3. Each rectangle represents an experiment. Each circle depicts a comparison of cost savings between experiments. Each bold arrow indicates the best experiment from the comparisons. Decision threshold (except for experiments V and IX) and cost model for these experiments will remain unchanged. Experiment V and IX will produce BP predictions need to be converted into categorical ones using the decision threshold value.

Modeling Table 4. Lists the eleven tests, labeled A to K, which were repeated for each of experiments I to V In other words, there are 55 test in total for experiments I to V. Each test consisted of training, testing, evaluation, and scoring. The score set was the same for all classifiers but the data partitions labeled 1 to 11 were rotated. The overall success rate denotes the ability of an ensemble of classifiers to provide correct predictions. The bagged overall success rates X and Z were compared to averaged overall success rates W and Y.

Modeling Experiments I, II, and III were designed to determine the best training distribution under the cost model. Which one of the above three training distributions is the best for the data partitions under the cost model? Experiment IV and V used the best training distribution determined from comparison 1. Experiment IV and V produce a bagged Z. Experiment VI, VII, and VIII determine which ensemble mechanism produces the best cost savings. Experiment VI used bagging to combine three sets of perditions from each algorithm. Experiment VII used stacking to combine all predictions. Experiment VIII proposed to bag the best classifiers determined by stacking.

Modeling Experiment IX implemented the BP algorithm on unsampled and unpartitioned data. This experiment was then compared with the other six before it. Which one of the above seven different classifier systems will attain the highest cost savings? Experiment X, XI, and XII were constructed to find out how each sampling method performs on unpartitioned data and if they could yield better results than the multiple classifier approach. Experiment XII’s data consists of the same number of examples as XI. But for XII, the minority class used SMOTE. Can the best classifier system perform better than the sampling approaches in the following results section?

Results Table 5 show in experiments I, II, and III, the bagged success rates X outperformed all the averaged success rates W. When applied on the score set, bagged success rate Z performed marginally better than the averaged success rates Y.

Results In figure 4., experiment IV highlights C4.5 as the best learning algorithm for this particular automobile insurance data set. The resultant predictions of experiment VIII (stacking-bagging) were better than those of C4.5 algorithm.

Results In figure 5, these three experiments performed comparably well at 40:60 and 50:50. Experiment XI and XII substantiate the claims that SMOTE is superior to minority oversampling with replacement. The undersampled data provides the highest cost saving of $165,242 at 60:40, it also incurs the highest expenditure (-$266,529). This is most likely due to the number of legal examples getting very small.

Discussion Table 6 ranks all the experiments using cost savings. Stacking-bagging achieves the highest cost savings which is almost twice the of the conventional BP procedure used by many fraud detection. The optimum success rate is 60% for highest cost savings in this slewed data set and, as the success rate increases, cost savings decrease.

Discussion Table 7 illustrates the top fifteen, out of 33 classifiers, produced from stacking.

Conclusion In this paper, existing fraud detection methods are explored and a new fraud detection method is recommended. The choice of the three classification algorithm and one hybrid meta-learning technique is justified for the new method. To extend the fraud detection method based on Minority Report to find out the properties of a data set, data partition, or data cluster which will make on classifier more appropriate.

11/04 Regular Meeting: Monority Report in Fraud Detection Classification of Skewed Data

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to 11/04 Regular Meeting: Monority Report in Fraud Detection Classification of Skewed Data (20)

11/04 Regular Meeting: Monority Report in Fraud Detection Classification of Skewed Data