ICPSR - Complex Systems Models in the Social Sciences - Lecture 7 - Professor Daniel Martin Katz
2 likes839 views
The document discusses various approaches to predicting decisions of the US Supreme Court: experts, crowds, and algorithms. It describes a classification and regression tree (CART) algorithm developed called "{Marshall}+" that uses random forests to predict individual justice votes (70.9% accuracy) and case outcomes (69.6% accuracy) based on past Supreme Court data from 1953 to 2014. The algorithm benchmarks predictions against what was known prior to the actual decisions.
![Benchmarking
since 1953
+
Using only data
available prior to
the decision
Mean Court Direction [FE]
Mean Court Direction 10 [FE]
Mean Court Direction Issue [FE]
Mean Court Direction Issue 10 [FE]
Mean Court Direction Petitioner [FE]
Mean Court Direction Petitioner 10 [FE]
Mean Court Direction Respondent [FE]
Mean Court Direction Respondent 10 [FE]
Mean Court Direction Circuit Origin [FE]
Mean Court Direction Circuit Origin 10 [FE]
Mean Court Direction Circuit Source [FE]
Mean Court Direction Circuit Source 10 [FE]
Difference Justice Court Direction [FE]
Abs. Difference Justice Court Direction [FE]
Difference Justice Court Direction Issue [FE]
Abs. Difference Justice Court Direction Issue [FE]
Z Score Difference Justice Court Direction Issue [FE]
Difference Justice Court Direction Petitioner [FE]
Abs. Difference Justice Court Direction Petitioner [FE]
Difference Justice Court Direction Respondent [FE]
Abs. Difference Justice Court Direction Respondent [FE]
Z Score Justice Court Direction Difference [FE]
Justice Lower Court Direction Difference [FE]
Justice Lower Court Direction Abs. Difference [FE]
Justice Lower Court Direction Z Score [FE]
Z Score Justice Lower Court Direction Difference [FE]
Agreement of Justice with Majority [FE]
Agreement of Justice with Majority 10 [FE]
Difference Court and Lower Ct Direction [FE]
Abs. Difference Court and Lower Ct Direction [FE]
Z-Score Difference Court and Lower Ct Direction [FE]
Z-Score Abs. Difference Court and Lower Ct Direction [FE]
Justice [S]
Justice Gender [FE]
Is Chief [FE]
Party President [FE]
Natural Court [S]
Segal Cover Score [SC]
Year of Birth [FE]
Mean Lower Court Direction Circuit Source [FE]
Mean Lower Court Direction Circuit Source 10 [FE]
Mean Lower Court Direction Issue [FE]
Mean Lower Court Direction Issue 10 [FE]
Mean Lower Court Direction Petitioner [FE]
Mean Lower Court Direction Petitioner 10 [FE]
Mean Lower Court Direction Respondent [FE]
Mean Lower Court Direction Respondent 10 [FE]
Mean Justice Direction [FE]
Mean Justice Direction 10 [FE]
Mean Justice Direction Z Score [FE]
Mean Justice Direction Petitioner [FE]
Mean Justice Direction Petitioner 10 [FE]
Mean Justice Direction Respondent [FE]
Mean Justice Direction Respondent 10 [FE]
Mean Justice Direction for Circuit Origin [FE]
Mean Justice Direction for Circuit Origin 10 [FE]
Mean Justice Direction for Circuit Source [FE]
Mean Justice Direction for Circuit Source 10 [FE]
Mean Justice Direction by Issue [FE]
Mean Justice Direction by Issue 10 [FE]
Mean Justice Direction by Issue Z Score [FE]
Admin Action [S]
Case Origin [S]
Case Origin Circuit [S]
Case Source [S]
Case Source Circuit [S]
Law Type [S]
Lower Court Disposition Direction [S]
Lower Court Disposition [S]
Lower Court Disagreement [S]
Issue [S]
Issue Area [S]
Jurisdiction Manner [S]
Month Argument [FE]
Month Decision [FE]
Petitioner [S]
Petitioner Binned [FE]
Respondent [S]
Respondent Binned [FE]
Cert Reason [S]
Mean Agreement Level of Current Court [FE]
Std. Dev. of Agreement Level of Current Court [FE]
Mean Current Court Direction Circuit Origin [FE]
Std. Dev. Current Court Direction Circuit Origin [FE]
Mean Current Court Direction Circuit Source [FE]
Std. Dev. Current Court Direction Circuit Source [FE]
Mean Current Court Direction Issue [FE]
Z-Score Current Court Direction Issue [FE]
Std. Dev. Current Court Direction Issue [FE]
Mean Current Court Direction [FE]
Std. Dev. Current Court Direction [FE]
Mean Current Court Direction Petitioner [FE]
Std. Dev. Current Court Direction Petitioner [FE]
Mean Current Court Direction Respondent [FE]
Std. Dev. Current Court Direction Respondent [FE]
0.00781
0.00205
0.00283
0.00604
0.00764
0.00971
0.00793
TOTAL 0.04403
Justice and Court Background Information
Case Information
0.00978
0.00971
0.00845
0.00953
0.01015
0.01370
0.01190
0.01125
0.00706
0.01541
0.01469
0.00595
0.02014
0.01349
0.01406
0.01199
0.01490
0.01179
0.01408
TOTAL 0.22814
Overall Historic Supreme Court Trends
0.00988
0.01997
0.01546
0.00938
0.00863
0.00904
0.00875
0.00925
0.00791
0.00864
0.00951
0.01017
TOTAL 0.12663
Lower Court Trends
0.00962
0.01017
0.01334
0.00933
0.00949
0.00874
0.00973
0.00900
TOTAL 0.07946
0.00955
0.00936
0.00789
0.00850
0.00945
0.01021
0.01469
0.00832
0.01266
0.00918
0.00942
0.00863
0.00894
0.00882
0.00888
Current Supreme Court Trends
TOTAL 0.14456
Individual Supreme Court Justice Trends
0.01248
0.01530
0.00826
0.00732
0.01027
0.00724
0.01030
0.00792
0.00945
0.00891
0.00970
0.01881
0.00950
0.00771
TOTAL 0.14323
0.01210
0.00929
0.01167
0.00968
0.01055
0.00705
0.00708
0.00690
0.00699
0.01280
0.01922
0.02494
0.01126
0.00992
0.00866
0.01483
0.01522
0.01199
0.01217
0.01150
TOTAL 0.23391
Differences in Trends](https://image.slidesharecdn.com/icpsrlecture82013-130731154428-phpapp01/85/ICPSR-Complex-Systems-Models-in-the-Social-Sciences-Lecture-7-Professor-Daniel-Martin-Katz-34-320.jpg)
ICPSR - Complex Systems Models in the Social Sciences - Lecture 7 - Professor Daniel Martin Katz
- 1. Complex Systems Models in the Social Sciences (Lecture 7) daniel martin katz illinois institute of technology chicago kent college of law @computationaldanielmartinkatz.com computationallegalstudies.com
- 2. consider the applied case of judicial prediction
- 3. Every year, law reviews, magazine and newspaper articles, television and radio time, conference panels, blog posts, and tweets are devoted to questions such as: How will the Court rule in particular cases?
- 6. There are 3 Known Ways to Predict Something
- 8. We could apply this to a wide range of problems
- 9. For today we will apply these approaches to the decisions of the Supreme Court of United States
- 10. this is an example of what is possible with other data
- 12. Experts
- 13. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
- 14. experts
- 15. Case Level Prediction Justice Level Prediction 67.4% experts 58% experts From the 68 Included Cases for the 2002-2003 Supreme Court Term
- 16. these experts probably performed badly because they overfit
- 17. they fit to the noise and not the signal
- 19. we need to evaluate experts and somehow benchmark their expertise
- 24. Crowds
- 25. crowds
- 31. Algorithms
- 32. Black Reed Frankfurter Douglas Jackson Burton Clark Minton Warren Harlan Brennan Whittaker Stewart White Goldberg Fortas Marshall Burger Blackmun Powell Rehnquist Stevens OConnor Scalia Kennedy Souter Thomas Ginsburg Breyer Roberts Alito Sotomayor Kagan 1953 1963 1973 1983 1993 2003 2013 9-0 Reverse 8-1, 7-2, 6-3 19 19 19 19 19 20 20 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - Reverse 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - 8-1, 7-2, 6-3 9-0 19 19 19 19 19 20 20 algorithms
- 33. we have developed an algorithm that we call {Marshall}+ extremely randomized trees (ERT)
- 34. Benchmarking since 1953 + Using only data available prior to the decision Mean Court Direction [FE] Mean Court Direction 10 [FE] Mean Court Direction Issue [FE] Mean Court Direction Issue 10 [FE] Mean Court Direction Petitioner [FE] Mean Court Direction Petitioner 10 [FE] Mean Court Direction Respondent [FE] Mean Court Direction Respondent 10 [FE] Mean Court Direction Circuit Origin [FE] Mean Court Direction Circuit Origin 10 [FE] Mean Court Direction Circuit Source [FE] Mean Court Direction Circuit Source 10 [FE] Difference Justice Court Direction [FE] Abs. Difference Justice Court Direction [FE] Difference Justice Court Direction Issue [FE] Abs. Difference Justice Court Direction Issue [FE] Z Score Difference Justice Court Direction Issue [FE] Difference Justice Court Direction Petitioner [FE] Abs. Difference Justice Court Direction Petitioner [FE] Difference Justice Court Direction Respondent [FE] Abs. Difference Justice Court Direction Respondent [FE] Z Score Justice Court Direction Difference [FE] Justice Lower Court Direction Difference [FE] Justice Lower Court Direction Abs. Difference [FE] Justice Lower Court Direction Z Score [FE] Z Score Justice Lower Court Direction Difference [FE] Agreement of Justice with Majority [FE] Agreement of Justice with Majority 10 [FE] Difference Court and Lower Ct Direction [FE] Abs. Difference Court and Lower Ct Direction [FE] Z-Score Difference Court and Lower Ct Direction [FE] Z-Score Abs. Difference Court and Lower Ct Direction [FE] Justice [S] Justice Gender [FE] Is Chief [FE] Party President [FE] Natural Court [S] Segal Cover Score [SC] Year of Birth [FE] Mean Lower Court Direction Circuit Source [FE] Mean Lower Court Direction Circuit Source 10 [FE] Mean Lower Court Direction Issue [FE] Mean Lower Court Direction Issue 10 [FE] Mean Lower Court Direction Petitioner [FE] Mean Lower Court Direction Petitioner 10 [FE] Mean Lower Court Direction Respondent [FE] Mean Lower Court Direction Respondent 10 [FE] Mean Justice Direction [FE] Mean Justice Direction 10 [FE] Mean Justice Direction Z Score [FE] Mean Justice Direction Petitioner [FE] Mean Justice Direction Petitioner 10 [FE] Mean Justice Direction Respondent [FE] Mean Justice Direction Respondent 10 [FE] Mean Justice Direction for Circuit Origin [FE] Mean Justice Direction for Circuit Origin 10 [FE] Mean Justice Direction for Circuit Source [FE] Mean Justice Direction for Circuit Source 10 [FE] Mean Justice Direction by Issue [FE] Mean Justice Direction by Issue 10 [FE] Mean Justice Direction by Issue Z Score [FE] Admin Action [S] Case Origin [S] Case Origin Circuit [S] Case Source [S] Case Source Circuit [S] Law Type [S] Lower Court Disposition Direction [S] Lower Court Disposition [S] Lower Court Disagreement [S] Issue [S] Issue Area [S] Jurisdiction Manner [S] Month Argument [FE] Month Decision [FE] Petitioner [S] Petitioner Binned [FE] Respondent [S] Respondent Binned [FE] Cert Reason [S] Mean Agreement Level of Current Court [FE] Std. Dev. of Agreement Level of Current Court [FE] Mean Current Court Direction Circuit Origin [FE] Std. Dev. Current Court Direction Circuit Origin [FE] Mean Current Court Direction Circuit Source [FE] Std. Dev. Current Court Direction Circuit Source [FE] Mean Current Court Direction Issue [FE] Z-Score Current Court Direction Issue [FE] Std. Dev. Current Court Direction Issue [FE] Mean Current Court Direction [FE] Std. Dev. Current Court Direction [FE] Mean Current Court Direction Petitioner [FE] Std. Dev. Current Court Direction Petitioner [FE] Mean Current Court Direction Respondent [FE] Std. Dev. Current Court Direction Respondent [FE] 0.00781 0.00205 0.00283 0.00604 0.00764 0.00971 0.00793 TOTAL 0.04403 Justice and Court Background Information Case Information 0.00978 0.00971 0.00845 0.00953 0.01015 0.01370 0.01190 0.01125 0.00706 0.01541 0.01469 0.00595 0.02014 0.01349 0.01406 0.01199 0.01490 0.01179 0.01408 TOTAL 0.22814 Overall Historic Supreme Court Trends 0.00988 0.01997 0.01546 0.00938 0.00863 0.00904 0.00875 0.00925 0.00791 0.00864 0.00951 0.01017 TOTAL 0.12663 Lower Court Trends 0.00962 0.01017 0.01334 0.00933 0.00949 0.00874 0.00973 0.00900 TOTAL 0.07946 0.00955 0.00936 0.00789 0.00850 0.00945 0.01021 0.01469 0.00832 0.01266 0.00918 0.00942 0.00863 0.00894 0.00882 0.00888 Current Supreme Court Trends TOTAL 0.14456 Individual Supreme Court Justice Trends 0.01248 0.01530 0.00826 0.00732 0.01027 0.00724 0.01030 0.00792 0.00945 0.00891 0.00970 0.01881 0.00950 0.00771 TOTAL 0.14323 0.01210 0.00929 0.01167 0.00968 0.01055 0.00705 0.00708 0.00690 0.00699 0.01280 0.01922 0.02494 0.01126 0.00992 0.00866 0.01483 0.01522 0.01199 0.01217 0.01150 TOTAL 0.23391 Differences in Trends
- 35. Total Cases Predicted Total Votes Predicted 7,700 68,964
- 36. Justice Prediction Case Prediction 70.9% accuracy 69.6% accuracy From 1953 - 2014
- 37. Relies upon Random Forest but first lets look at CART
- 40. Given Some Data: (X1, Y1), ... , (Xn, Yn) Now We Have a New Set of X’s We Want to Predict the Y
- 41. Form a BinaryTree that Minimizes the Error in each leaf of the tree CART (Classification & RegressionTrees)
- 42. Observe the Correspondence Between the Data andTrees
- 43. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 Adapted from Example By Mathematical Monk
- 44. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 Adapted from Example By Mathematical Monk We want to build an approach which can lead to the proper classification (labeling) of new data points ( ) that are dropped into this space
- 45. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 Adapted from Example By Mathematical Monk
- 46. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 Adapted from Example By Mathematical Monk L e t s B e g i n t o Partition the Space
- 47. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 1 2 1 2 Adapted from Example By Mathematical Monk L e t s B e g i n t o Partition the Space split 1 (a)
- 48. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 1 2 1 2 Adapted from Example By Mathematical Monk This Split Will Be Memorialized in theTree split 1 (a)
- 49. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 1 2 1 2 Adapted from Example By Mathematical Monk We Ask the Question is Xi1 > 1 ? - with a binary (yes or no) response split 1 (a) Xi1 > 1 ? YesNo
- 50. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 1 2 1 2 Adapted from Example By Mathematical Monk If No - then we are in zone (a) ... we tally the number of zeros and ones Using Majority Rule do we assign a classification to this rule this leaf split 1 (a) Xi1 > 1 ? YesNo (0,5) Classify as 1 zone (a)
- 51. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 1 2 1 2 Adapted from Example By Mathematical Monk Here we Classify as a 1 because (0,5) which is 0 zero’s and 5 one’s split 1 (a) Xi1 > 1 ? YesNo (0,5) Classify as 1 zone (a)
- 52. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 1 2 1 2 Adapted from Example By Mathematical Monk Using a Similar Approach Lets Begin to Fill in the Rest of theTree split 1 (a) Xi1 > 1 ? YesNo (0,5) Classify as 1 zone (a)
- 53. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0 1 2 1 2 Adapted from Example By Mathematical Monk split 1 (a) Xi1 > 1 ? YesNo (0,5) Classify as 1 zone (a) Xi2 > 1.45 ? No Yes split 2
- 54. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0split 1 split 2 split 3 1 2 2.2 1 2 Xi1 > 1 ? (0,5) Xi2 > 1.45 ? (4,1)(2,3) Xi1 < 2 ? Classify as 1 Classify as 1 Classify as 0 (a) zone (a) 1.45 YesNo Adapted from Example By Mathematical Monk No (b) (c) zone (b) zone (c) YesNo Yes
- 55. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0split 1 split 2 split 3 split 4 1 2 2.2 1 2 Xi1 > 1 ? (0,5) Xi2 > 1.45 ? Xi1 > 2.2 ? (1,4)(5,0)(4,1)(2,3) Xi1 < 2 ? Classify as 1 Classify as 1 Classify as 0 (a) zone (a) 1.45 YesNo Adapted from Example By Mathematical Monk No (b) (c) (d) (e) zone (b) zone (c) YesNo YesNo Yes zone (d) Classify as 0 Classify as 1 zone (e)
- 56. Okay Lets Add Back the ( ) which are new items to be classified
- 57. For simplicity sake there is one in each zone
- 58. We Will Use theTree Because theTree Is Our Prediction Machine
- 59. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0split 1 split 2 split 3 split 4 1 2 2.2 1 2 Xi1 > 1 ? (0,5) Xi2 > 1.45 ? Xi1 > 2.2 ? (1,4)(5,0)(4,1)(2,3) Xi1 < 2 ? Classify as 1 Classify as 1 Classify as 0 (a) zone (a) 1.45 YesNo Adapted from Example By Mathematical Monk No (b) (c) (d) (e) zone (b) zone (c) YesNo YesNo Yes zone (d) Classify as 0 Classify as 1 zone (e)
- 60. 1 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 01 0 Xi1 Xi2 0split 1 split 2 split 3 split 4 1 2 2.2 1 2 Xi1 > 1 ? (0,5) Xi2 > 1.45 ? Xi1 > 2.2 ? (1,4)(5,0)(4,1)(2,3) Xi1 < 2 ? Classify as 1 Classify as 1 Classify as 0 (a) zone (a) 1.45 YesNo Adapted from Example By Mathematical Monk No (b) (c) (d) (e) zone (b) zone (c) YesNo YesNo Yes zone (d) Classify as 0 Classify as 1 zone (e) 1 1 1 0 1 0
- 61. In this simple example, we eyeballed the 2D space, partitioned it and stopped after 4 Splits
- 62. Most Real Problems are Not So Simple ...
- 63. Real problems are n-dimensional (not 2D) (1)
- 64. For real problems, you need to select criteria (or a criterion) for deciding where to partition (split) the data (2)
- 65. For real problems you must develop a stopping condition or pursue recursive partitioning of the space (3)
- 66. Solutions to these 3 Problems are among the core questions in algorithm selection / development
- 67. From an Algorithmic Perspective - TheTask is to Develop a Method to Partition theTrees
- 68. Must Do So Without Knowing the Specific Contours of the Data / Problem in Question
- 70. So How Do We TraverseThrough The Data?
- 71. Optimal Partitioning of Trees is NP-Complete
- 72. “Although any given solution to an NP-complete problem can be verified quickly (in polynomial time), there is no known efficient way to locate a solution in the first place; indeed, the most notable characteristic of NP-complete problems is that no fast solution to them is known.That is, the time required to solve the problem using any currently known algorithm increases very quickly as the size of the problem grows”
- 73. key implication is that one cannot in advance determine the “optimal tree”
- 74. Breiman, et al (1984) uses a Greedy Optimization Method
- 75. Greedy Optimization Method is used to calculate the MLE (maximum-likelihood estimation)
- 76. Greedy is a Heuristic “makes the locally optimal choice at each stage with the hope of finding a global optimum. In many problems, a greedy strategy does not in general produce an optimal solution, but nonetheless a greedy heuristic may yield locally optimal solutions that approximate a global optimal solution in a reasonable time.”
- 79. Get the Data Here: http://www.stat.cmu.edu/~cshalizi/350/hw/06/cadata.dat
- 80. x <- read.table("http://www.stat.cmu.edu/~cshalizi/350/hw/06/cadata.dat") Get the Data Here: Load the DataSet: http://www.stat.cmu.edu/~cshalizi/350/hw/06/cadata.dat
- 81. http://www.stat.cmu.edu/~cshalizi/350/lectures/22/lecture-22.pdf x <- read.table("http://www.stat.cmu.edu/~cshalizi/350/hw/06/cadata.dat", header=TRUE) Get the Data Here: Load the DataSet: http://www.stat.cmu.edu/~cshalizi/350/hw/06/cadata.dat Follow Example on Page 4-7 (example 2.1)
- 90. Random Forest
- 91. One well-known problem with standard classification trees is their tendency toward overfitting
- 92. This is because standard decision trees are weak learners
- 93. Random forest is an approach to aggregate weak learners into collective strong learners (think of it as statistical crowd sourcing)
- 94. Random Forest: Group of DecisionTrees Outperforms and is more Robust (i.e. is less likely to overfit) than a Single DecisionTree
- 95. Ensemble method that leverages bagging (bootstrap aggregation) Brieman (1996) With Random Substrates Brieman (2001) Random Forest:
- 97. bootstrap aggregation is applied to the training data random substrates is applied to / about the variables Two Layers of Randomness
- 98. bootstrap aggregation (row) is applied to the training data random substrates (column) is applied to / about the variables Two Layers of Randomness
- 100. What is Bagging?
- 101. bagging = bootstrap aggregation
- 104. “if the outlook is sunny and the humidity is less than or equal to 70, then it’s probably OK to play.” http://bit.ly/1icRlmE Single Decision Tree
- 106. Sample N cases at random with replacement to create a subset of the data STEP 1: (Blackwell 2012)
- 107. M predictor variables are selected at random from all the predictor variables. The predictor variable that provides the best split, according to some objective function, is used to do a binary split on that node. At the next node, choose another m variables at random from all predictor variables and do the same.” STEP 2: “At each node:
- 113. Additional Notes For Random Forest Trees are not pruned As potentially overfit individual trees combine to yield well fit ensembles
- 114. http://machinelearning202.pbworks.com/w/file/fetch/37597425/ performanceCompSupervisedLearning-caruana.pdf Trees (particularly with optimization) have proven to be unreasonably effect
- 115. 10 Different Binary Classification Methods on 11 Different Datasets (w/ 5000 training cases each) Trees and Forest were surprisingly effective
- 121. For most problems ... ensembles of these streams outperform any single stream
- 122. Humans + Machines
- 123. Humans + Machines >
- 126. Ensembles come in various forms
- 127. Here is a well known example
- 128. Poll Aggregation is one form of ensemble where the learning question is to determine how much weight (if any) to assign to each individual poll
- 129. poll weighting
- 131. A Visual Depiction of How to build an ensemble method in our judicial prediction example
- 132. expert crowd algorithm ensemble method learning problem is to discover when to use a given stream of intelligence
- 133. expert crowd algorithm via back testing we can learn the weights to apply for particular problems ensemble method learning problem is to discover when to use a given stream of intelligence
- 137. {Marshall}+ improvement will likely come from determining the optimal weighting of experts, crowds and algorithms for various types of cases
- 138. ERISA cases thus might look like this
- 139. Patent cases Perhaps might look like this
- 140. Search/Seizure cases while could look like this
- 142. this is one slice our research effort ...
- 143. and we are working on a series of improvements to the model