![CREDIT_EXTENDED). For these 1,000 accounts, we also have
information on how profitable each account turned out to be
(the variable NPV). A negative value indicates a net loss, and
this typically happens when the debtor defaults on his/her
payments.
1. Create a categorical variable that indicates whether or not a
new credit extension will result in a positive NPV.
2. Create dummy variables for all categorical variables with
more than two values (if appropriate).
3. Split the data into two parts using the splitting variable that
is a part of the data set[footnoteRef:1]. This is to ensure a more
balanced split between the validation and training samples.
After the data partition you should have 666 rows in your
training data and 334 in your validation data. [1: If you run
into issues that your # of columns exceeds 50, you may leave
out the employment variable.]
Please answer all questions. Supply supporting documentation
and show calculations as needed. Please submit a single well-
formatted Word file. In addition, please upload an Excel file
with your model outputs .
Classification trees
Classify customers as profitable/not profitable with a](https://image.slidesharecdn.com/week2individualassignment2quantitativeanalysisofcredit-221013143641-9a809d38/85/Week-2-Individual-Assignment-2-Quantitative-Analysis-of-Credit-docx-19-320.jpg)
Week 2 Individual Assignment 2 Quantitative Analysis of Credit - .docx
- 1. Week 2 Individual Assignment 2: Quantitative Analysis of Credit - Solution s This assignment is based on the data we used during our two live sessions, but it has been updated to include a splitting variable (credit2.xlsx). In the spreadsheet under the tab “Data," you will find data pertaining to 1,000 personal loan accounts. The tab “Data Dictionary” contains a description of what the various variables mean. As a part of a new credit application, the company collects information about the applicant. The company then decides an amount of the credit extended (the variable CREDIT_EXTENDED). For these 1,000 accounts, we also have information on how profitable each account turned out to be (the variable NPV). A negative value indicates a net loss, and this typically happens when the debtor defaults on his/her payments. The goal in this assignment is to investigate how one can use this data to better manage the bank's credit extension program. Specifically, our goal is to develop a classification model to
- 2. classify a new credit account as “profitable” or “not profitable." Secondly we want to compare its performance in the context of decision support to a linear regression model that predicts NPV directly. Please answer all the questions. Supply supporting documentation and show calculations as needed. Please submit a single, well-formatted PDF or Word file. The instructor should not need to go searching for your answers! In addition, please upload an Excel file with your model outputs – the file will not be graded, but will help the instructor give you feedback, if your model differs substantially from the solutions. For extra assistance, you may want to access the tutorials located on the course resource center page.Data Preparation The data preparation repeats the steps from the live session: a) The goal is to predict whether or not a new credit will result in a profitable account. Create a new variable to use as the dependent variable. b) Create dummy variables for all categorical variables with more than 2 values (or if you prefer, you can sort your variables into numerical and categorical when you run the model). c) Split the data into 2 parts using the splitting variable that has been added to the data set. This is to ensure a more balanced
- 3. split between the validation and training samples. Note that Analytic Solver Data Mining only allows 50 columns in the analysis, so leave out your base dummies (if you created them) when partitioning. After the data partition, you should have 666 rows in your training data and 334 in your validation data. The Assignment 1. Applying Logistic Regression If one fits a Logistic Regression Model using all the independent variables, one observes a) a gap in the classification performance between the training data and the validation data, and b) very high p-values for some of the variables. The performance gap between the training and validation may be a sign of overfitting, and the high p-values may be a sign of “useless” variables in the model, or of multicollinearity.
- 4. a) Our goal is to classify credit requests into “profitable” and “not profitable." To that end, select to run “forward selection," and set FIN down to 1.5 (this lowers the threshold for a variable to enter the model, resulting in more models to choose from). Select one of the forward selection models based on the principles discussed in the book and/or the tutorials on the course resource center and run it. Note: Exclude Credit Extended and any other variables not appropriate for the analysis. Include the model (the variables and the corresponding regression coefficients) as an Exhibit. Predictor Estimate Intercept -0.1409 AGE 0.0350 NUM_CREDITS -0.3472 DURATION -0.0208 INSTALL_RATE
- 6. EMPLOYMENT_2 0.3505 EMPLOYMENT_3 0.7936 TYPE_2 1.9168 TYPE_3 0.5290 TYPE_4 0.6752 Please refer to the Excel solutions for additional details. b) Why did you select this particular model? From the feature selection output we chose to run the model with 18 coefficients. This model was chosen because it has Cp close to the number of coefficients in the model (and not higher), it has probability above .05 and the improvement in RSS if we expand the model further is relatively small. c) Based on your model, and setting the cut-off value to 0.5, please provide the following information (based on the validation data): · The sensitivity of the model: 0.88
- 7. · The specificity of the model: 0.495 In other words, at the default cut-off we correctly identify 88% of the profitable customers, but include around 50% of the unprofitable customers. 2. ROC Curves a) We now want to compare the predictive performance of the model on the training sample and on the validation sample. Create a single figure that compares the ROC curves for both the training sample and the validation sample. Please refer to the ROC tutorials in the resource center as needed for a step-by- step guide for creating an ROC curve. Alternatively, you can combine the two curves that Analytical Solver Data Mining provides into a single plot. Include a clean figure as an Exhibit. 3. Finding the "best" cut-off a) Create a data-table to calculate the total NPV (assuming we extend credit to all classified as “profitable” as a function of the cut-off based on the training
- 8. data. Select the best cut- off. Include the table as an Exhibit. cut-off 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 NPV training -56740 -26705 -2644 18771 45734 68023 88312 90200 84550
- 9. 63883 0 NPV validation -39141 -15636 -6309 13945 31128 36599 51636 51623 43999 32005 0 Please refer to the solutions for a more detailed table. b) What is your selected cut-off? 0.725 (based on a more detailed table in the Excel file) c) Create the same table for the validation data. Include the table as an Exhibit. Refer to the table above d) Apply the cut-off you selected based on the training data to
- 10. the validation data. What is the total profit on the validation data? $51,330 e) Provide a figure that shows the cumulative NPV as a function of the cut-off for both the training data and the validation data. 4. Comparison with linear regression a) Repeat our model development from our first live session (note you need to repeat the steps as we now have a new data split). Rerun a variable selection model to find a "good model" using the updated data. Include the model (the variables and the corresponding regression coefficients) as an Exhibit. For my linear regression model I selected to run a stepwise selection with the default parameters. Note that this is not the only "correct" model, a careful analysis would have included both backwards, forwards and stepwise variable selection and the comparison of a couple of candidate models, before selecting one based on their performance (and you can define performance in multiple ways
- 11. as we have discussed). Hopefully your model’s performance exceeds the performance of the model discussed here! Predictor Estimate P-Value Intercept 594.6747 <0.001 RENT -288.596 0.010 INSTALL_RATE -152.409 <0.001 AMOUNT_REQUESTED -0.17395 <0.001 CHK_ACCT_1 327.4346 0.005 CHK_ACCT_2 396.0738 0.033 CHK_ACCT_3
- 12. 563.3212 <0.001 SAV_ACCT_4 378.5465 0.001 TYPE_2 525.5611 <0.001 TYPE_5 -489.718 0.011 TYPE_6 -500.304 0.001 b) Create a data table that summarizes the total profit as a function of the NPV cut-off for extending credit on the training data (note that now your cut-off is in $ you will need to investigate what is a good cut-off, for example -$50 or $50, or something else). Select the best cut-off. Include the table as an Exhibit. Please refer to the Excel file for detailed information, the table
- 13. below shows some highlights. total training NPV validation -750 74541 16890 -700 75131 19502 -650 73680 24582 -600 71729 27223 -550 79911 28620 -500 77482
- 17. 5329 c) What is your selected cut-off? -$400 d) Create the same table for the validation data and include it as an Exhibit. Please refer to the table above e) Apply the cut-off you found to the validation data. What is the total profit on the validation data? $33,657 f) Provide a figure that shows the cumulative NPV as a function of the cut-off for both the training data and the validation data. 5. Model comparison a) Compare the performance of the logistic regression model and the linear regression model. How does the total profit compare for the two models? Which model would you select as the foundation of a decision support system and why?
- 18. When we compare the performance as measured by the total NPV is higher for both the training and the validation sample, as a result I would select my Logistic Regression model over the Linear Regression model (although it may be worth the effort to try to improve on the linear regression model). Classification Trees and k-NN applied to Bank Credit This assignment concludes the analysis of the credit data, exploring whether we can improve on our earlier analysis that utilized linear and logistic regression. Please refer to the earlier assignments for the data description, and repeat if needed the data preparation steps, using the credit2.xlsx data: In the spreadsheet under the tab “Data," you will find data pertaining to 1,000 personal loan accounts. The tab “Data Dictionary” contains a description of what the various variables mean. As a part of a new credit application, the company collects information about the applicant. The company then decides an amount of the credit extended (the variable
- 19. CREDIT_EXTENDED). For these 1,000 accounts, we also have information on how profitable each account turned out to be (the variable NPV). A negative value indicates a net loss, and this typically happens when the debtor defaults on his/her payments. 1. Create a categorical variable that indicates whether or not a new credit extension will result in a positive NPV. 2. Create dummy variables for all categorical variables with more than two values (if appropriate). 3. Split the data into two parts using the splitting variable that is a part of the data set[footnoteRef:1]. This is to ensure a more balanced split between the validation and training samples. After the data partition you should have 666 rows in your training data and 334 in your validation data. [1: If you run into issues that your # of columns exceeds 50, you may leave out the employment variable.] Please answer all questions. Supply supporting documentation and show calculations as needed. Please submit a single well- formatted Word file. In addition, please upload an Excel file with your model outputs . Classification trees Classify customers as profitable/not profitable with a
- 20. classification tree 1. Run the Classification Tree algorithm using all the relevant independent variables (excluding as before Credit Extended, Obs# etc. ) including all the dummy variables (recall that one does not exclude base values when running classification trees), with the profitable/not profitable as the output variable. Use the validation data to prune back the tree, and select to use the best pruned tree for scoring. a. Include the classification confusion matrix for the validation sample and a figure of the best pruned tree as Exhibits. 2. Analyze the output. a. How many decision nodes are in the best pruned tree? b. What is the error rate for i) the training data and ii) the validation data in the best pruned tree? c. What explains the difference in the error rate? d. Which applicants for credit will get rejected by the model (using the best pruned tree)? (Describe the type of customers using the English language.) 3. Using the model for decision making. a. Consider a 27-year-old domestic student that has $100 in her checking account but no savings account. The student has one existing credits, which has so far been paid back duly. The
- 21. credit duration is 12 months. The applicant has been renting her current place for less than 12 months, does not own any real estate, just started graduate school (the present employment variable is set to 1 and nature of job to 2). The applicant has no dependents and no guarantor. The applicant wants to buy a used car and has requested $4,500 in credit, and therefore the installment rate is quite high, or 2.25%. However, the applicant does not have other installment plan credits. Finally, the applicant has a phone in her name. How would the best pruned tree classify the student? k-NN Classify customers as profitable/not profitable with k-NN 4. Run the k-NN algorithm for classification, testing all values of k from 1 to 10, selecting to score the data on the best k (remember to standardize/normalize the data). Request detailed output for both the training and validation data. a. Using the search log, plot the %Error of the validation sample. Include the plot in your assignment. b. What is the best value of k? c. Briefly explain why the % Error is zero for the training sample when k=1, but not for the validation sample. 5. Analyze the output. a. What some of the main differences are between the customers identified as most likely to be profitable and the customers that
- 22. are identified as least likely to be profitable? Briefly discuss. Method comparisons You have now run three different classification algorithms on this data; logistic regression, classification tree and k-NN. Compare their performance in two ways. First using statistical measures and second using their possible impact on the credit extension process. Feel free to take advantage of the solutions to Individual Assignment 2 as a starting point. Hint: Below is a potential set-up to measure the business impact. First, collect the predicted probability of being profitable for both the training and validation data as well as the true NPV into a single spreadsheet. Perhaps similar to this: Then select a cell for a cut-off (in my case I used E1). Then for each method and each sample we can calculate the cumulative profit, for a specific cut-off using the sumifs() function in Excel. Specifically, the following formula sums up the NPV of all credit extensions that are made using the training sample and logistic regression: You then need to extend this approach to both data samples and all three methods. Perhaps similarly to this: You can then create data tables to investigate the best cut-off
- 23. for each method and the corresponding NPV on the validation data. DataOBS#AGECHK_ACCTSAV_ACCTNUM_CREDITSDURA TIONHISTORYPRESENT_RESIDENTEMPLOYMENTJOBNU M_DEPENDENTSRENTINSTALL_RATEGUARANTOROTHE R_INSTALLOWN_RESTELEPHONEFOREIGNREAL_ESTATE TYPEAMOUNT_REQUESTEDCREDIT_EXTENDEDNPVSplitt ing Variable1671026444210400110141,1691,052243v22521112211 211300000011,2951,165- 473v3480139442210400110012,1341,920268t436241182312103 01110161,9131,721416t53003130224310401100042,3331,16623 5t62920124321110401100032,3331,866480v7272119212210201 110161,391695118t83001136222210101100028,1336,5061,407v 94400224211210100110045,9435,943- 1,826v10560111244421040010014618556163v11410212424411 1400000111,4691,322376t12312226324220101100061,4491,304 252t132511221222210300110141,835917- 508t142420148223210100100018,4875,092976t1527341622101 02001011170949693t16420216244220201010041,3461,076295t 173602124444220300110142,8722,584744t18331121503422140
- 57. _x000D__x000D_ Data DictionaryVar. #Variable NameDescriptionVariable TypeCode Description1OBS#Observation No.NumericalSequence number in dataset2AGEAge in yearsNumerical3CHK_ACCTChecking account statusCategorical0 : < 0 1: 0 < ...< 2002 : => 2003: no checking account4SAV_ACCTAverage balance in savings accountCategorical0 : < 1001 : 100<= ... < 5002 : 500<= ... < 10003 : =>10004 : unknown/ no savings account5NUM_CREDITSNumber of existing creditsNumerical6DURATIONDuration of credit in monthsNumerical7HISTORYCredit historyCategorical0: no credits taken 1: all credits at this bank paid back duly 2: existing credits paid back duly till now 3: delay in paying off in the past 4: critical account 8PRESENT_RESIDENTPresent resident since - yearsCategorical0: <= 1 year1<…<=2 years2<…<=3 years3:>4years9EMPLOYMENTPresent employment sinceCategorical0 : unemployed1: < 1 year2 : 1 <= ... < 4 years 3 : 4 <=... < 7 years4 : >= 7 years10JOBNature of jobCategorical0 : unemployed/ unskilled - non-resident1 : unskilled - resident2 : skilled employee / official3 : management/ self-employed/highly qualified employee/ officer11NUM_DEPENDENTSNumber of people for whom
- 58. liable to provide maintenanceNumerical12RENTApplicant rents Binary 0: No, 1: Yes13INSTALL_RATEInstallment rate as % of disposable incomeNumerical14GUARANTORApplicant has a guarantorBinary 0: No, 1: Yes15OTHER_INSTALLApplicant has other installment plan creditBinary 0: No, 1: Yes16OWN_RESApplicant owns residenceBinary 0: No, 1: Yes17TELEPHONEApplicant has phone in his or her nameBinary 0: No, 1: Yes18FOREIGNForeign workerBinary 0: No, 1: Yes19REAL_ESTATEApplicant owns real estateBinary 0: No, 1: Yes20TYPEPurpose of CreditCategorical0: Other1: New Car2: Used Car3: Furniture4: Durable5: Education6: Retraining21AMOUNT_REQUESTEDCredit Amount Applied forNumerical22CREDIT_EXTENDEDCredit MadeNumerical23NPVNet Profit from the Loan (Net Loss if Negative)Numerical24Splitting VariableA variable added to ensure a balanced partitionCategoricalt: training record, v: validation record _x000D__x000D_ _x000D__x000D_ _x000D__x000D_ _x000D__x000D_