Fraud Detection using Data Mining Project
- 1. DATA MINING PROJECT Fraud Detention using Data Mining JUNE 7, 2015 NORTHWESTERN UNIVERSITY, MSIS 435 ALBERT KENNEDY
- 2. 1 TABLE OF CONTENTS Abstract............................................................................................................................................... 2 Introduction........................................................................................................................................ 3 Data Mining Applications.................................................................................................................... 4 Data Mining Themes........................................................................................................................... 4 CRISP-DM Methodology...................................................................................................................... 5 Data Understanding............................................................................................................................ 7 Data Preparation................................................................................................................................. 9 Data Mining Algorithm...................................................................................................................... 10 Experimental Results and Analysis................................................................................................ 11 Conclusion..................................................................................................................................... 14 Future Work.................................................................................................................................. 15 References .................................................................................................................................... 16
- 3. 2 Fraud Detention 1 ABSTRACT The Adoption of data mining can be great for many use cases and organization that have a special need and understanding of what can be done with existing data. Many organizations don’t understand the power and value they have in what you control. With this, the benefits of using a process for making smarter decision will be discussed in this paper. For the purpose of explaining not yet a data mining topic and its benefits but also to address common problems in the Fraud and identity sector where many businesses and individuals can take advantage of. Fraud detention should be more important today than ever before. With the growing e-commence business that moving rapidly and people having more access to important, financial institutions need to be more aware of ways to detect possible fraudulent acts. We can achieve this goal with the use of data mining. This document will show case a typical problem with sample data taken from borrowers and their information that related to credit approval. Another data set is similar but present typical account holders and a common profile of related attributes that make up a “type” of customer. We will go into detail and the proper process of how to solve this problem using the outline: Defining the business problem Collection the data and enhancing that data Choosing a model strategy and algorithm that fixes the business need Executing the model through a training set then test that model Evaluating the results of the model Decide for the model or make any changes Deploy the model into an actionable project The above outline is based on a framework that many data scientist use today called “The Cross Industry Standard Process for Data Mining” (CRISP-DM)1 . This foundation is what will be use to analyze the fraud detention use case of both the German Credit fraud data and the Give Me Some Credit data set to compare against using two separate techniques. This two datasets have been thoroughly cleanse and checked for correctly and free of any bias input to shew any result toward any direction. In order to become successful in building a fraud detention system, it is important to understand the data mining tools applications used in the industry. Second, it’s important to know the themes of data mining and most importantly the CRISP-DM methodology that will be used to support our business problem and data mining design for a reliable fraud detention system. 1 Information related to CRISP-DM, see CRISP-DM 1.0, Step-by-Step data mining guide, SPSS,
- 4. 3 2 INTRODUCTION In America and many other others of the world, crime appears to be relevant in today’s society. As the government and we as people continue to find ways to prevent and coach individuals to shrey away from unlawful event, people find alternative and intelligent ways to be corrupted. Using our government law enforcement has only become as good as to catch perpetrators after a particular wrong doing event has been either reported or caught. Not all actions we can’t control and stop before happening that’s within any person’s private setting. How about we can help prevent those crimes that can be warned if not stop prior to happing? This type of crime that we people such as businesses and individual victims can have control over with the used of data analysis. The crimes of identity thief and Credit approval can be defended. The purpose of this study is to empower financial businesses and individuals with the ability to combat potential thief of personal and company information to avoid misuse for another person’s gain. Description of Problem: In today’s fast growing technology sociality, individuals are completing more and more online transactions and sending data to multiple sources of businesses using the same vital information to identify a person. Examples of this could be applying for credit via an online store. So what information is needed for this? In many cases Full name Telephone number or street address Social security number (SSN) The above information is all that is needed for a creditors to approve someone for a line of credit or account under a person’s name. There’s an issue here; anyone that doesn’t know you can obtain this information easily. The only harder bit of information to obtain is an SSN. The best way for someone to get a person’s SSN, and through personal work records; a person that has access to this who handle’s administrative tasks for employees. This is a problem, because a telephone number and street address can be any number or address that the creditor does not care for other than a place to mail bills to. Typically a creditor will validate this through mailing information. Say a different phone number was given and validated with a phone call that could also be possible. We can use smarter ways to combat this, which some companies do with multiple levels on validating (through phone call, matching current address and identification). Our Objective: We can help fix these problems through the use of data mining and making sound business decisions in order to complete a credit transition. First we need we can identify the types of data needed to solve an identity or fraud similar action. We will use pervious data from users of different credit accounts to do the analysis work on. Then, we need to choose one or many different data mining algorithms to test this data for an outcome of what we are analyzing to make a recommendation, prediction, classifications, and/or description of for better business decision.
- 5. 4 So, let’s explore the many different data mining applications that can be used that’s related to this problem 3 DATA MINING APPLICATIONS There are many related data mining applications that can be used for the purpose of detecting fraudulent activities. This is a growing study that many established organizations have seek out and completed in depth research that more so expose the issues and weakness more so than real world working applications that actual identity the issue and combat the problem in a defense matter. A company called Morpho, has a mission where they are the market leader in security solutions who are the pioneer in identification and detection systems. They deliver many products that many target government and national agencies with dedication tools and systems to safeguard sensitive information. They completed a study using data mining and the relation to identity fraud as an application to prevent and or warn businesses, government organizations and individuals of a possible fraudulent act. From there “Fighting Identity Fraud with Data Mining,” paper on Safran product2 , they speak about a comprehensive fraud-proof process. A second company that should be worth mention who use data mining methods to conducted a similar study was Federal Data Corporation and the SASA institute Inc. These two completed a thorough study on “Using Data Mining Techniques for Fraud Detection.” This was solved in conjunction with using the SAS Enterprise Miner software. The two use cases presented where 1) Health Care Fraud Detection and 2) Purchase Card Fraud detection. Both have similar if not the same business problems and ending goals. The first case, the FDC and SAS used Decision Trees to group all the nominal values of input into smaller group that will in turn give a predictive target outcome. The second case study, they used a modeling strategy that was clustering. Their analysis included three clusters that help explain the cluster analysis efficiently segments data into groups of similar cases. The overall conclusions for both unveiled unknown patterns and regularities in their data. 4 DATA MINING THEMES The study of data mining to best explained and organized into different themes. These different area are better described by the four core data mining tasks. According to “Introduction to Data Mining,” by Pang-Ning Tan, these themes are covered under the four core tasks; Predictive modeling, cluster analysis, Association Analysis Anomaly detection.3 To briefly describe each theme of data mining, it is best to show by examples. These themes in detailed are: Classification Clustering Anomaly 2 Product from the Morpho Inc., Safran, “Fighting Identity Fraud With Data Mining” 3 See more information on themes from Assignment 1: Data Science Applicaiton, Kennedy, Albert
- 6. 5 First off, predictive modeling is split into two types 1) Classification which is used for discrete target variables and 2) regression, which is used for continuous target variables” Tan. The Classification type is mostly common for making a prediction for an outcome which is the target variable of a single action. Whereas the regression, may analyze the cost that a consumer may spend monthly on an e-commerce website. These two types of variables (classification and regression) are what help define predictive model. The second them, Cluster analysis “seeks to find groups of closely related observations so that observations that belong to the same cluster are more similar to each other than observations that belong to other cluster”, Tan. Good examples of this would be grouping different customers’ purchasing behaviors. Doing this helps define the types of customers and their purchases a clothing retail store have for analysis. Then lastly, there is the Association Analysis them. This theme is used “to discover patterns that describe strongly associated features in the data” Tan. The association theme is commonly used in the retailer or grocery market businesses for analysis. We can group liked things together that have similarities based on related attributes and/or pair transitions from users. These three themes all have its purpose to help solve particular data science problems. With solving these problems, businesses can make decisions using these techniques. Understanding its definition is key to ensure the right solution/theme is being utilized for the correct problem. Once the understanding is complete, analyst can make use of the right tool for apply for the most appropriate case. In many uses, not just one theme may apply to a case, but multiple themes can be applied for better analysis and comparisons for the best results. 5 CRISP-DM METHODOLOGY For many of these data mining techniques, we don’t want to apply the wrong or less effective solution. Lucky, there is a well-organized methodology that gives businesses the steps and processes to handle out these type of data mining project. We use what’s called the “Cross Industry Standard Process for Data Mining” or CRIPS-DM for short. The CRIPS-DM is a structured framework with hierarchical steps to follow in order to help guide through a proper data mining problem and solution. The CRISP-DM include six phases:4 4 Information related to CRISP-DM, see CRISP-DM 1.0, Step-by-Step data mining guide, SPSS
- 7. 6 Figure 5-1: CRISP-DM diagram 1) Business Understanding – this makes sense as an initial step. Any data mining problem has an business need and problem that needs to be understood. This stage “represents a part of the craft where the analysts’ creativity plays a large role…the design team should think carefully about the use scenario” Provost. In this stage questions such as what needs to be done and how it needs to be done are asked. 2) Data Understanding – this phase is a self-explanatory step yet could take a lot of time. Making sure as an analyst you become knowledgeable of the data can make for an easier process. This phase enables you to become “familiar with the data, identify data quality problems, discover first insights into data, and/or detect interesting subsets to form hypotheses regarding hidden information”, SPSS. 3) Data Preparation – in order for us to make a good analysis, we need tools that will enable us to process the data in the best matter suitable for the necessary model. Examples of this phase may require converting data in a simple tabular format, removed pointless attributes not relevant to the data problem and/or converting a data file to a particular file format in order to operate in the chosen data mining tools.
- 8. 7 4) Modeling – “The modeling stage is the primary place where data mining techniques are applied to the data” Provost. Simply put, this is where the magic happens and the actual data mining craft and chosen algorithm(s) are put into work. 5) Evaluation – at the evaluation phase, we take time to access the results of the outcomes from the models that were built from our data. The most important aspect of this phase going through this evaluation is to gain confidence in the model’s outcome. We would like to analysis the results and understand its outcome to ensure it’s reliable for meet the original business problem’s needs. 6) Deployment – Now the model has been created and test, now we can make use of this reliable model into a real life production case. What can we do with it…“the knowledge gained will need to be organized and presented in a way that the customer can use it”, SPSS. Depending on the businesses need for the data mining model that was craft, the deployment can be simple or complex. Simple being as creating the results to report to managers or taking that model and actually implementing in to the business in need. Within each phase, there are a set of tasks that the business will help generate for both the data mining team and business to complete in order to complete through the CRISP-DM cycle successfully. 6 DATA UNDERSTANDING For the purpose of proving the data mining algorithms used for the case of fraud detention, we will examine two different data sets. The first is the German Credit fraud data from Dr. Hans Hofmann, the University of Hamburg in Germany. This particular data has 1000 instances of customer information for preparation of understanding approval of credit. There are 20 attributes used to help describe each instance and there uniqueness. German Creadit Data Definition: Variable Name Description Type over_draft Status of existing checking account qualitative credit_usage real Duration in month numerical credit_history A30 : no credits taken/ all credits paid back duly A31 : all credits at this bank paid back duly A32 : existing credits paid back duly till now A33 : delay in paying off in the past A34 : critical account/ other credits existing (not at this bank) qualitative purpose Type of credit/loan needed for (new car, used car, furniture/equipment, radio/tv, repairs, education, vacation, business,other qualitative current_balance real Credit amount numerical Average_Credit_Balance Savings account/bonds qualitative employment Present employement since a date qualitative
- 9. 8 Location real Installment rate in percentage of disposble income numerical Personal_status Personal status and sex qualitative other_parties Other debtors / guarantor qualitative residence_since real Present residence since a date qualitative property_magnitude Property qualitative cc_age real Age count in months numerical other_payment_plans Other installment plans qualitative housing Housing type - rent, own, for free qualitative existing_Credits real Number of existing credits at this bank numerical job Job and type Unemployed/unskilled, skilled, management/self-employed, highly qualified qualitative num_dependents real Number of people being liable to provide maintenance numerical own_telephone Telephone qualitative foreign_worker Indicate Yes or No if Foreign worker qualitative class the cost matrix used for indicating is customer is Good or Bad qualitative The above information is financial data taken from year 1994 of customer that submitted for credit. Details of the attributes used are of types in integers and categorical formats. This data set is best used as a Classification data mining task. The second data source comes from Kaggle, Give Me Some Credit competition. According to Kaggle the purpose of this second is to so “state of the art in credit scoring by predicting the probability that somebody will experience financial distress in the next two years.” This dataset uses less attributes and input for descripting the customers. However this dataset has 4000 instances Credit distress probability Data Definition: Variable Name Description Type SeriousDlqin2yrs Person experienced 90 days past due delinquency or worse Y/N RevolvingUtilizationOfUnsecuredLines Total balance on credit cards and personal lines of credit except real estate and no installment debt like car loans divided by the sum of credit limits percentage age Age of borrower in years integer NumberOfTime30- 59DaysPastDueNotWorse Number of times borrower has been 30-59 days past due but no worse in the last 2 years. integer DebtRatio Monthly debt payments, alimony,living costs divided by monthy gross income percentage MonthlyIncome Monthly income real NumberOfOpenCreditLinesAndLoans Number of Open loans (installment like car loan or mortgage) and Lines of credit (e.g. credit cards) integer NumberOfTimes90DaysLate Number of times borrower has been 90 days or more past due. integer NumberRealEstateLoansOrLines Number of mortgage and real estate loans including home equity lines of credit integer NumberOfTime60- 89DaysPastDueNotWorse Number of times borrower has been 60-89 days past due but no worse in the last 2 years. integer
- 10. 9 NumberOfDependents Number of dependents in family excluding themselves (spouse, children etc.) integer The goal of this particular data is to help build a model to help those that are borrowing make better financial decisions. This will be under a Classification task type as well. 7 DATA PREPARATION The process used to was very simple for the German Credit fraud data. Decided to use an ARFF file format due to the source of the data from UC Irvine, Machine Learning Repository to organize. Had collect the attributes needed for the analysis and paste into notepad application with the @ beginning symbols to denote that these variables are the attributes. Then taking the @data field and pasting below the raw data. As long as the raw data that’s divided by commas has the same number of values to match the number of attributes given, the file will be accurate. Here’s a snapshot of what the inside of an ARFF file will contains: @relation german_credit @attribute over_draft { '<0', '0<=X<200', '>=200', 'no checking'} @attribute credit_usage real @attribute credit_history { 'no credits/all paid', 'all paid', 'existing paid', 'delayed previously', 'critical/other existing credit'} @attribute purpose { 'new car', 'used car', furniture/equipment, radio/tv, 'domestic appliance', repairs, education, vacation, retraining, business, other} @attribute current_balance real @attribute Average_Credit_Balance { '<100', '100<=X<500', '500<=X<1000', '>=1000', 'no known savings'} @attribute employment { unemployed, '<1', '1<=X<4', '4<=X<7', '>=7'} @attribute location real @attribute personal_status { 'male div/sep', 'female div/dep/mar', 'male single', 'male mar/wid', 'female single'} @attribute other_parties { none, 'co applicant', guarantor} @attribute residence_since real @attribute property_magnitude { 'real estate', 'life insurance', car, 'no known property'} @attribute cc_age real @attribute other_payment_plans { bank, stores, none} @attribute housing { rent, own, 'for free'} @attribute existing_credits real @attribute job { 'unemp/unskilled non res', 'unskilled resident', skilled, 'high qualif/self emp/mgmt'} @attribute num_dependents real @attribute own_telephone { none, yes} @attribute foreign_worker { yes, no} @attribute class { good, bad} @data '<0',6,'critical/other existing credit',radio/tv,1169,'no known savings','>=7',4,'male single',none,4,'real estate',67,none,own,2,skilled,1,yes,yes,good '0<=X<200',48,'existing paid',radio/tv,5951,'<100','1<=X<4',2,'female div/dep/mar',none,2,'real estate',22,none,own,1,skilled,1,none,yes,bad A compiled version of this can be viewed here http://weka.8497.n7.nabble.com/file/n23121/credit_fruad.arff The second data source is a CSV file that required some modification. The original data set file had 150,000 instances which was too large for the WEKA data mining tool’s heap size to handle. I manually removed sections of the data set by dividing the total 150,000 into four sections and removing 3,750
- 11. 10 from bottom half of the sections. I did this so I have an even amount of distributed data instead of only taking the top 4000 instances. This file was then saved as cs-traning.csv file for WEKA input. For both files, I created Data definitions to elaborate on their chosen attributes. The purpose of this is to explain what inputs are used and its purpose for a clearer business understanding when we need to make a decision after reviewing the results. 8 DATA MINING ALGORITHM Select data mining algorithm for your project, elaborate chosen algorithm in detail with reason why algorithm was chosen over other algorithms. German Credit Fraud Use case: The first dataset using the German credit fraud data was ideal for the use of Decision tree algorithm. This type of classification algorithm, the decision tree is a very first good technique for this particular use case. Because this is a very common algorithm to use, we have many reason of a benefit to do an analysis with this method. First, it’s relatively simple approach for classification type of data. It gives the ability to take sample data with known attributes and place them into categories. Second, it help you visualize the workflow of how the data is been broken down into sections that make decisions. Last, you can determine a predictive outcome from the results. Using Decision Tree, we need to explain more in depth how this method works and its structure. When data is ran through this type of analysis, it structures the data into these three areas called nodes: The root node is an attribute used to question the initial question if or if not something is in a particular group or not to start off with. This can only be a single node item where the groups are then branched off from it much like a tree from the bottom. Then there grows the internal nodes which is a title of the proceeding branch off the root node. The purpose of the internal node is to give information only pertaining to that group. Lastly, there’s the leaf node(s). Think of these are individual leafs as answers to the internal nodes. There can be multiple leads branching off an internal node. These finalize the answer of the item in a particular internal node group.
- 12. 11 Figure 8-1 Decision tree The simple method works each time for data that has multiple attributes that has classification of types in them. For the purposes of our fraudulent credit problem, out root node answers the major questions of those that will over draft or not. In this cases, I wouldn’t care to use the nodes that are derived, but more so of the confusion matrix. The Confusion Matrix helps classifies the actual classes from those that are potentially negative classed numbers.5 We use the confusion matrix to help spate the decisions made by the classifier making explicit how one class is being confused for another. This way error can be handled separately. We do this be looking at the True class items and the predicted class items in a matrix box. TP = true positive FN = false negative PREDICTED CLASS ACTUAL CLASS Yes No Yes a=TP b=FN Yes c=FN d=TN Figure 8.2 – Confusion Matrix The goal here is to have the model obtain the highest possible accuracy rate or lowest error rate. **Confusion matrix 9 EXPERIMENTAL RESULTS AND ANALYSIS To test our fraudulent data, the test was executed with two different data mining techniques. Decision tree and the Simple K-means algorithms to generate the results. German Credit Data Use: Examining the first data set of the German credit fraud data, we analyze this data using the decision tree algorithm. The outcome below shows the summary of the results. === Stratified cross-validation === === Summary === Correctly Classified Instances 705 70.5 % Incorrectly Classified Instances 295 29.5 % Kappa statistic 0.2467 Mean absolute error 0.3467 Root mean squared error 0.4796 Relative absolute error 82.5233 % Root relative squared error 104.6565 % Total Number of Instances 1000 === Detailed Accuracy By Class === 5 See more information from Assignment 2: Marketing Campaign Effectiveness, Kennedy, Albert
- 13. 12 TP Rate FP Rate Precision Recall F-Measure ROC Area Class 0.84 0.61 0.763 0.84 0.799 0.639 good 0.39 0.16 0.511 0.39 0.442 0.639 bad Weighted Avg. 0.705 0.475 0.687 0.705 0.692 0.639 === Confusion Matrix === a b <-- classified as 588 112 | a = good 183 117 | b = bad What we are looking for here is a way to determine a good enough model for use. Based on the results, we have a 70% accuracy of correct classified instances in the data set. This is decent for justification. However, the ROC area curve is at 64%, which isn’t bad nor near perfect or ideal. How about we consider the confusion matrix for deeper analysis. Remember from below the confusion matrix is separated on four different section to help determine where our good and bad classifiers fall into best. A = True Positive B = False Negatives Our matrix has an overwhelming amount, 588 instances that fall under the True Positive section where it also equals true for the predicted and actual class. Say we take the percentage for the A class 588 + 183 = 771 (total) 588 (TP) / 771(total) = 76% accuracy In order to generate this results, the data mining tool used was Weka v3-6-12 Give Me Some Credit Use: Using the data set from the Kapple competition, we have a difference approach of how we want to view the results. Initially, trying the decision tree algorithm for this dataset did not yield any solid enough results for analysis purposes or to make any business sense. So, it was best to do a Cluster algorithm type and view the results. Choosing the Simple K Means was the algorithm used for this. kMeans ====== Number of iterations: 10 Within cluster sum of squared errors: 5429.211249046848 Missing values globally replaced with mean/mode Cluster centroids:
- 14. 13 Time taken to build model (full training data) : 0.05 seconds === Model and evaluation on training set === Clustered Instances 0 401 ( 40%) 1 266 ( 27%) 2 333 ( 33%) To help explain the results, it will help to define the K-means method and its use. The K-means is a commonly used cluster algorithm which is “simple, iterative way to approach and divide a data set into specific number of cluster”- Manning. This process run through any dataset and uses a “closeness” as a way to measure the distance of items in a dataset. This is called the Euclidean distance. To help explain how the Euclidean distance works there’s a center point that’s created which is the centroid as the cluster. Any number of items of K, that is within that distance of the centroid is defined as a cluster that it is assigned to. If one item is closer in distance to another centroid than another centroid point, that closer item is part of a different cluster. In the above analysis, we have three defined clusters separated into their groups based on like attributes their share. Cluster 0: This cluster has the strongest poll of instances. The results would suggest that in this cluster more individuals that has an existing paid credit history and happen to be in the younger age group (31) , females and are most likely to be requesting a NEW CAR for credit.
- 15. 14 Cluster 1: This cluster has the weakness amount of instances. The results would suggest that customers in this group are the oldest (age 40) are seeking for credit for USED CARS. Cluster 2: This cluster has a distribution of 33%. The results would suggest these are SINGLE MALE customers seeking credit for RADIO/TV. 10 CONCLUSION The completed analysis drawn from two different dataset would yield two different ending business decision results. If visually inspecting the first dataset using the decision tree algorithm, I would initial conclude this is not a valid enough test to make as a solid choice for fraud detention. We have to remember the goal is to define the problem where fraud used is being done where customer information is being misuse in place to benefit another person by creating credit accounts. However, the confusion matrix does present some promising results to consider. What a financial institution can take from that is a starter point of where to predict. However it gives about a 76% accuracy of the changes of this results be true for prediction. The second analysis is to be view from a different approach not so much as for predictive, but for a clear view of where customer land is their relative attributes that tie to them. Using the Simple K-Mean algorithm and picking 3 clusters, we can analysis a few important things: 1) where the most important group of customers are at 2) the related attributes in comparison to other groups 3) purpose of line of credit 4) Instances or distribution percentage of which group has the highest activity and means to apply for credit. With the clustering results, a business can answer the above questions ahead of time. We can pick and choose the determinant for our analysis. For our purpose we wish to detent the reason for a line of credit.
- 16. 15 11 FUTURE WORK Describe next steps to continue work on this project There are organizations that already taking the necessary actions to benefit from these findings. Companies like Morpho with their Safran product. Here’s a list of procedures and steps that should be consider in order for this study to become successful. 1) Ensure those data scientist and analyst placed with the responsibility to craft these results to use proper data mining practices and methodologies like CRISP-DM. 2) Take risk: I believe there are enough intelligent groups of people that understand the worth and capable of drafting similar fraud detention analysis. There needs to more action on the deployment phase. 3) Implement a production system where this process is automated. Based on the analysis and results we have a plausible solution to solve a business problem with fraud due to credit applications.
- 17. 16 12 REFERENCES Tan, Pang and Michel Steinbach. Introduction to Data Mining. Boston: Pearson Addison Wesley, 2005. Print (Chp 1, Chp 4). Provost, Foster, and Fawcett, Tom, Data Science for Business, Sebastopol, CA, 2014. Print (Chp 2, Chp 7) Morpho, Safran, Fighting Identity Fraud with Data Mining, Groundbreaking means to prevent fraud in identity management solutions, France, Print (page 4, and page 7) Federal Data Corporation and SAS, Using Data Mining Techniques for Fraud Detection, Solving Business Problems using SAS Enterprise Minder Software, Cary, NC, Print (page 1, page 15 and page 20) Dr. Hofmann, Hans, University at Hamburg, UCI Machine Learning Reposity, CA, 2000, http://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29 Kaggle, Give Me Some Credit, 201, https://www.kaggle.com/c/GiveMeSomeCredit