![following is a description of the non-GA approaches we instance class is selected from the maximum weighted
used for the performance comparison studies. average of the predicted class over all decision trees.
1) Induction Decision Trees 3) Neural Networks
The construction of a decision tree is divided into two Inspired in part by biological learning systems, neural
stages. First, creating an initial, large decision tree using a networks approach is built from a densely interconnected set
set of training set. Second, pruning the initial decision tree, of simple units. Since this technique offers many design
if applicable, using a validation set. Given a noise-free selections, we fixed some of them to the ones that had been
training set, the first stage will generate a decision tree that well proven to be good or acceptable design choices. In
can classify correctly all examples in the set. Except that the particular, we use a network architecture with one hidden
training set covers all instances in the domain, the initial layer, the back-propagation learning algorithm (Rumelhart,
decision tree generated will over fit the training data, which Hinton & William, 1986), the delta-bar-delta adaptive
then reduce its performance in the test data. The second learning rates (Jacobs, 1988), and the Nguyen-Widrow
stage helps alleviate this problem by reducing the size of the weight initialization (Nguyen & Widrow, 1990). Discrete-
tree. This process has an effect in generalizing the decision valued attributes fed into the networks input layer is
tree that hopefully could improve its performance in the test represented as 1-of-N encoding using bipolar values to
data. denote the presence (e.g., value 1) and the absence (e.g.,
During the construction of an initial decision tree, the value –1) of an attribute value. Continuous-valued attribute
selection of the best attribute is based on either the is scaled into a real value in the range [–1, 1]. We varied the
information gain (IG) or gain ratio (GR). A binary split is design choices for batch versus incremental learning, and 1-
applied in nodes with continuous-valued attributes. The best of-N versus single network output encoding.
cut-off value of a continuous-valued attribute is locally 4) Naive Bayes
selected within each node in the tree based on the remaining Naive Bayes classifier is a variant of the Bayesian learning
training examples. The tree’s node expansion stops either that manipulates directly probabilities from observed data
when the remaining training set is homogeneous (e.g., all and uses these probabilities to make an optimal decision.
instances have the same target attribute values) or when no This approach assumes that attributes are conditionally
attribute remains for selection. The decision on a leaf independent given a class. Based on this simplifying
resulting from the latter case is determined by selecting the assumption, the probability of observing attributes’
majority of the target attribute value in the remaining conjunction is the product of the probabilities for the
training set. individual attributes. Given an instance with a set of
Decision tree pruning is a process of replacing sub-trees attribute-value pairs, the Naive Bayes approach will choose
with leaves to reduce the size of the decision tree while a class that maximizes the conditional probability of the
retaining and hopefully increasing the accuracy of tree's class given the conjunction of attributes values. Although in
classification. To obtain the best result from the induction practice the independence assumption is not entirely correct,
decision tree method, we varied the use of pruning it does not necessarily degrade the system performance
algorithm to the initial decision tree. We considered using (Domingos & Pazzani, 1997).
the following decision trees pruning algorithms: critical- We also assume that the values of continuous-valued
value pruning (Mingers, 1987), minimum-error pruning attributes follow Gaussian distribution. Hence, once the
(Niblett & Bratko, 1986), pessimistic pruning, cost- mean and the standard deviation of these attributes are
complexity pruning and reduced-error pruning (Quinlan, obtained from the training examples, the probability of the
1987),. corresponding attributes can be calculated from the given
2) Induction Decision Trees with Boosting attribute values. To avoid zero frequency count problem that
Decision Tree with Boosting is a method that generates a can dampen the entire probabilities calculation, we use an
sequence of decision trees from a single training set by re- m-estimate approach in calculating probabilities of discrete-
weighting and re-sampling the samples in the set (Quinlan, valued attributes (Mitchell, 1997).
1996; Freund & Schapire, 1996). Initially, all samples in the
4.3 Comparison results
training set are equally weighted so that their sum is one.
For each database, k-fold cross-validation method is used
Once a decision tree has been created, the samples in the
for evaluation. In this method, a data set is divided equally
training set are re-weighted in such a way that misclassified
into k disjoint subsets. k experiments are then performed
examples will get higher weights than the ones that are
using k different training-test set pairs. A training-test set
easier to classify. The new samples weights are then
pair used in each experiment is generated by using one of
renormalized, and next decision tree is created using higher-
the k subsets as the test set, and using the remaining subsets
weighted samples in the training set. In effect, this process
as the training set. Given k disjoint subsets, for example, the
enforces the more difficult samples to be learned more
first experiment takes the first subset for the test set, and
frequently by decision trees. The trees generated are then
uses the second subset through the kth subset for the training
given weights in accordance with their performance in the
set. The second experiment uses subset 1 and subset 3
training examples (e.g., their accuracy in correctly
through subset k for the training set; and takes subset 2 for
classifying the training data). Given a new instance, the new](https://image.slidesharecdn.com/cec01docdoc1079/85/cec01-doc-doc-5-320.jpg)
cec01.doc.doc
- 1. An Entropy-based Adaptive Genetic Algorithm for Learning Classification Rules Linyu Yang Dwi H. Widyantoro Thomas Ioerger John Yen Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science Texas A&M University Texas A&M University Texas A&M University Texas A&M University College Station, TX 77843 College Station, TX 77843 College Station, TX 77843 College Station, TX 77843 lyyang@cs.tamu.edu dhw7942@cs.tamu.edu ioerger@cs.tamu.edu yen@cs.tamu.edu Abstract- Genetic algorithm is one of the commonly used instead, the consequent of a rule is determined by the approaches on data mining. In this paper, we put majority of training examples it matches. forward a genetic algorithm approach for classification Four important factors are considered in our evaluation problems. Binary coding is adopted in which an functions. Error rate is calculated by the predicting results individual in a population consists of a fixed number of on the training examples. Entropy is used to measure the rules that stand for a solution candidate. The evaluation homogeneity of the examples that a certain rule matches. function considers four important factors which are Rule consistency is a measure on the consistency of error rate, entropy measure, rule consistency and hole classification conclusions of a certain training example ratio, respectively. Adaptive asymmetric mutation is given by a set of rules. Finally, hole ratio is used to evaluate applied by the self-adaptation of mutation inversion the percentage of training examples that a set of rules does probability from 1-0 (0-1). The generated rules are not not cover. We try to include related information as complete disjoint but can overlap. The final conclusion for as possible in the evaluation function so that the overall prediction is based on the voting of rules and the performance of a rule set can be better. classifier gives all rules equal weight for their votes. An adaptive asymmetric mutation operator is applied in Based on three databases, we compared our approach our reproduction step. When a bit is selected to mutate, the with several other traditional data mining techniques inversion probability from 1-0 (0-1) is not 50% as usual. including decision trees, neural networks and naive The value of this probability is asymmetric and self- bayes learning. The results show that our approach adaptive during the running of the program. This is made to outperformed others on both the prediction accuracy reach the best match of a certain rule to training examples. and the standard deviation. For crossover, two-point crossover is adopted in our approach. Keywords: genetic algorithm, adaptive asymmetric We used three real databases to test our approach: credit mutation, entropy, voting-based classifier database, voting database and heart database. We compared our performance with four well-known methods from data 1. Introduction mining, namely Induction Decision Trees (ID3) (Quinlan, 1986), ID3 with Boosting (Quinlan, 1996), Neural Genetic algorithms have been successfully applied to a wide Networks, and Naive Bayes (Mitchell, 1997). The range of optimization problems including design, appropriate state-of-the-art techniques are incorporated in scheduling, routing, and control, etc. Data mining is also these non-GA methods to improve their performances. The one of the important application fields of genetic algorithms. results show that our GA approach outperformed other In data mining, GA can be used to either optimize approaches on both the prediction accuracy and the standard parameters for other kinds of data mining algorithms or deviation. discover knowledge by itself. In this latter task the rules that In the rest of the paper, we will first give a brief overview GA found are usually more general because of its global of related work. Our GA approach is then discussed in search nature. In contrast, most other data mining methods detail. The results of our application on three real databases are based on the rule induction paradigm, where the and the comparison with other data mining methods are algorithm usually performs a kind of local search. The followed. Finally we make some concluding remarks. advantage of GA becomes more obvious when the search space of a task is large. In this paper we put forward a genetic algorithm approach 2. Related works for classification problems. First we use binary coding in Many researchers have contributed to the application of GA which an individual solution candidate consists of a fixed on data mining. In this section, we will give a brief number of rules. In each rule, k bits are used for the possible overview on a few representative works. k values of a certain attribute. Continuous attributes are In early 90’s, De Jong et al. implemented a GA-based modified to threshold-based boolean attributes before system called GABIL that continually learns and refines coding. Rule consequent is not explicitly coded in the string, concept classification rules (De Jong, 1993). An individual
- 2. is a variable-length string representing a set of fixed-length classification problems. Some parameter values used in the rules. Traditional bit inversion is used on mutation. In algorithm might be task dependent. crossover, corresponding crossover points in the two parents 3.1 Individual’s encoding must semantically match. The fitness function contains only Each individual in the population consists of a fixed number the percent of correct examples classified by an individual of rules. In other words, the individual itself is a complete rule set. They compared the performance of GABIL with solution candidate. In our current implementation, we set that of four other traditional concept learners on a variety of this fixed number as 10 which well satisfies the requirement target concepts. of our testing databases. The antecedent of a certain rule in In GIL (Janikow, 1993), an individual is also a set of the individual is formed by a conjunction of n attributes, rules, but attributes values are encoded directly, rather than where n is number of attributes being mined. K bits will be bits. GIL has special genetic operators for handling rule used to stand for an attribute if this attribute has k possible sets, rules, and rule conditions. The operators can perform values. Continuous attributes will be partitioned to generalization, specialization or other operations. Besides threshold-based boolean attribute in which the threshold is a the correctness, the evaluation function of GIL also includes boundary (adjacent examples across this boundary differ in the complexity of a rule set. Therefore it favors correct, their target classification) that maximizes the information simple (short) rules. gain. Therefore two bits will be used for a continuous Greene and Smith put forward a GA-based inductive attribute. The consequent of a rule is not explicitly encoded system called COGIN (Greene, 1993). In contrast to the in the string. In contrast, it is automatically given based on above two approaches, the system’s current model at any the proportion of positive/negative examples it matches in point during the search is represented as a population of the training set. We will illustrate the encoding method by fixed length rules. The population size (i.e., the number of the following example. rules in the model) will vary from cycle to cycle as a Suppose our task has three attributes, and they have 4, 2, function of the coverage constraint is applied. The fitness 5 possible values respectively. Then an individual in the function contains the information gain of a rule R and a population can be represented as following: penalty of the number of misclassifications made by R. A1 A2 A3 A1 A2 A3 A1 A2 A3 Entropy measure is used to calculate the information gain of 0110 11 10110 1110 01 10011 …… 1100 11 01110 rule R based on the number of examples rule R Matched Rule 1 Rule 2 …… Rule 10 and Unmatched. However, it couldn’t evaluate the entropy Ten rules are included in this individual. The architecture measure of the entire partition formed by the classification of each rule is same. We will use rule 1 to explain the due to its encoding method. meaning of encoding. In this example, the meaning of the In view of the situation that most of the data mining work antecedent of rule 1 is: emphasizes only on the predictive accuracy and If (A1=value 2 OR value 3) AND (A2=value 1 OR value 2) comprehensibility, Noda et al. (Noda, 1999) put forward a AND (A3=value 1 OR value 3 OR value 4) GA approach designed to discover the interesting rules. The If all the bits belong to one attribute are 0s, it means that fitness function consists of two parts. The first one measures attribute can not equal to any possible value therefore this is the degree of interestingness of the rule, while the second meaningless. To avoid this, we add one step before the measures its predictive accuracy. The computation of the evaluation of the population. We will check each rule in consequent’s degree of interestingness is based on the each individual one by one, if the above case happens, we following idea: the larger the relative frequency (in the will randomly select one bit of that attribute and change it to training set) of the value being predicted by the consequent, one. the less interesting it is. In other words, the rarer a value of a The consequent of a rule is not encoded in the string. It goal attribute, the more interesting a rule predicting it is. will be determined by the proportion situation of the training Since the values of the goal attribute in the databases we examples that rule matches. Suppose i is one of the tested are not unevenly distributed seriously, we didn’t classifications, the consequent of a rule will be i if specially consider interestingness of a rule in our current N matched _ i N training _ i implementation but mainly focus on including related > , (1) factors as complete as possible to improve the predicting N matched N training accuracy. However, we did consider how to treat uneven N matched _ i where is the number of examples whose distribution of goal attribute values somehow. We will discuss this in details in the next section. classification is i and matched by that rule, N matched is the total number of examples that the rule matches; N training _ i is 3. Our GA approach the number of training examples whose classification is i, In this section we present our GA approach for N training is the total number of training examples. classification problem. The key idea of the algorithm is For example, if the distribution of the positive examples general and should be applicable for various kinds of and negative examples in the training set is 42% and 58%, and among the examples of rule 1 matches positive /
- 3. negative examples are half to half, then the consequent of n rule 1 should be positive because 0.5 > 0.42. Since the Entropy ( R ) = − ∑(p i log 2 ( p i )) (4) testing databases we use at this time don’t have a very i =1 uneven distribution on the classification of training where n is the number of target classifications. examples, in our current implementation we didn’t specially While an individual consists of a number of rules, the consider the interestingness of rules but use this strategy to entropy measure of an individual is calculated by averaging keep enough rules to match examples with minor the entropy of each rule: NR classification. Our encoding method is not limited to two- category classification but applicable to multi target value ∑ Entropy( R ) i =1 i (5) problems. Entropy (individual ) = NR 3.2 Fitness function where NR is number of rules in the individual (in our current It is very important to define a good fitness function that implementation it is 10). rewards the right kinds of individuals. We try to consider The rationale of using entropy measure in fitness function affecting factors as complete as possible to improve the is to prefer those rules that match less examples whose results of classification. Our fitness function is defined as target values are different from rule’s consequent. High following: accuracy does not implicitly guarantee the entropy measure Fitness = Error rate + Entropy measure + Rule consistency is good because the final classification conclusion of a + Hole ratio (2) certain training example is based on the comprehensive We will elaborate each part in the fitness function in the results of a number of rules. It is very possible that each rule following. in the individual has a bad entropy measure but the whole 1) Error rate rule set still gives the correct classification. Keeping low It is well known that accuracy is the most important and entropy value of an individual will be helpful to get better commonly used measure in the fitness function as the final predicting results for untrained examples. goal of data mining is to get good prediction results. Since 3) Rule consistency our objective function here is minimization, we use error As stated in the above sections, the final predicted rate to represent this information. It is calculated as: classification of a training example is the majority Error rate = percent of misclassified examples (3) classification made by rules in an individual. Let’s consider If a rule matches a certain example, the classification it the following classifications made by two individuals on an gives is its consequent part. If it doesn’t match, no example: classification is given. An individual consists of a set of Individual a: six rules +, four rules -, final rules, the final classification predicted by this rule set is classification: + based on the voting of those rules that match the example. Individual b: nine rules +, one rule -, final The classifier gives all matching rules equal weight. For classification: + instance, in an individual (which has ten rules here), one We will prefer the second individual since it is less rule doesn’t match, six rules give positive classification and ambiguous. To address this rule consistency issue, we add three rules give negative classification on a training another measure in the fitness function. The calculation is example, then the final conclusion given by this individual similar to the entropy measure. Let Pcorrect be the proportion on that training example is positive. If a tie happens (i.e., of rules in one individual whose consequent is same with four positive classifications and four negative the target value of the training example, then classifications), the final conclusion will be the majority Rule consistency (individual ) = − p correct log 2 p correct − classification in the training examples. If none of the rules in (6) the individual matches that example, the final conclusion (1 − p correct ) log 2 (1 − p correct ) will also be the majority classification in the training We should notice that this formula will give the same rule examples. The error rate measure of this individual is the consistency value when pcorrect and (1-pcorrect) switch each percent of misclassified examples among all training other. Therefore a penalty will be given when pcorrect is examples. smaller than 0.5. In this case Ruleconsistency = 2 - Ruleconsistency. 2) Entropy measure The above calculation is based on the predicting results for Entropy is a commonly used measure in information theory. one training example. The complete measure of rule Originally it is used to characterize the (im)purity of an consistency of an individual should be averaged by the arbitrary collection of examples. In our implementation number of training examples. entropy is used to measure the homogeneity of the examples 4) Hole ratio: that a rule matches. The last element in the fitness function is the hole ratio. It is Given a collection S, containing the examples that a a measure of rule’s coverage on training examples. certain rule R matches, let Pi be the proportion of examples Coverage is not a problem for traditional inductive learning in S belonging to class i, then the entropy Entropy(R) related methods like decision trees, since the process of creating to this rule is defined as: trees guarantees that all the training examples will be
- 4. covered in the tree. However, this also brings a new next generation and calculate the average fitness of the new problem that it may be sensitive to noise. GA approach does generation. not guarantee that the generated rules will cover all the 3) If the fitness is better (value is smaller), continue on training examples. This allows flexibility and may be this direction and the amount of change is: potentially useful for future prediction. In real ∆p = max{0.05, (1- fitnessnew / fitnessold) * 0.1} (8) implementation we still hope the coverage should reach a If the fitness is worse (value is larger), reverse the certain point. For instance, if a rule only matches one direction and the amount of change is: training example and its consequent is correct, the accuracy ∆p = max{0.05, (fitnessnew / fitnessold - 1) * 0.05} (9) and entropy measure of this rule are both excellent but we Use the new probability to produce the next generation do not prefer this rule because its coverage is too low. and calculate the average fitness of the new generation. In our fitness function the hole ratio equals to 1-coverage, Repeat step 3 until the program ends. in which the latter is calculated by the union of examples that are matched and also correctly predicted by the rules in 4. Results and discussions an individual. Totally misclassified examples (not classified correctly by any rule in the individual) will not be included We tested our approach on three real databases. We even though they are matched by some rules. The following compared our approach with four other traditional data is the formula to calculate the hole ratio for binary mining techniques. This section will present the testing classification problem (positive, negative). results. 4.1 The information of databases P i i + + N i i − (7) 1) Credit database Hole = 1 − This database concerns credit card applications. All attribute S names and values have been changed to meaningless + where Pi stands for those examples whose target value is symbols to protect confidentiality of the data. This database is interesting because there is a good mix of attributes ---- positive and classified as positive by at least one rule in the continuous, nominal with small numbers of values, and individual, N i− stands for those examples whose target nominal with larger numbers of values. There are 15 value is negative and classified as negative by at least one attributes plus one target attribute. Total number of rule in the individual. S is the total number of training instances is 690. 2) Voting database examples. This database saves 1984 United States Congressional 3.3 Adaptive asymmetric mutation Voting Records. The data set includes votes for each of the In our reproduction step, traditional bit inversion is used on U.S. House of Representatives Congressmen on the 16 key mutation. However, we found many examples will not be votes identified by the CQA. The CQA lists nine different matched if we keep number of 1’s and 0’s approximately types of votes: voted for, paired for, and announced for equivalent in an individual (i.e., the inversion probability (these three simplified to yea), voted against, paired against, from 1-0 and 0-1 are both 50%). The learning process will and announced against (these three simplified to nay), voted become a majority guess if there are too many unmatched present, voted present to avoid conflict of interest, and did examples. Therefore, we put forward a strategy of adaptive not vote or otherwise make a position known (these three asymmetric mutation in which the inversion probability simplified to an unknown disposition). There are 16 from 1-0 (0-1) is self-adaptive during the process of run. attributes plus one target attribute. Total number of The asymmetric mutation biases the population toward instances is 435 (267 democrats, 168 republicans). generating rules with more coverage on training examples. 3) Heart database The self-adaptation of inversion probability makes the This database concerns heart disease diagnosis. The data optimal mutation parameter be automatically adjusted. was provided by V.A. Medical Center, Long Beach and We presented an adaptive simplex genetic algorithm Cleveland Clinic Foundation: Robert Detrano, M.D., Ph.D. before (Yang, 2000) in which the percentage of simplex There are 14 attributes plus one target attribute. Total operator is self-adaptive during the process of run. Similar number of instances is 303. idea is adopted here that average of fitness is used as a 4.2 The description of non-GA approaches feedback to adjust the inversion probability. The process of We used four well-known methods from machine learning, self-adaptation is described as following: namely Induction Decision Trees (ID3) (Quinlan, 1986), 1) An initial inversion probability is set (e.g., 0.5 for 1-0). Decision Trees with Boosting (Quinlan, 1996), Neural Use this probability on mutation to produce a new Networks, and Naïve Bayes (Mitchell, 1997), to compare generation. Calculate the average fitness of this generation. the performance of our improved GA. Appropriate state-of- 2) Randomly select the direction of changing this the-art techniques have been incorporated in most of the probability (increase, decrease). Modify the probability non-GA methods to improve their performance. The along that direction with a small amount (0.02 in our current implementation). Use the new probability to produce the
- 5. following is a description of the non-GA approaches we instance class is selected from the maximum weighted used for the performance comparison studies. average of the predicted class over all decision trees. 1) Induction Decision Trees 3) Neural Networks The construction of a decision tree is divided into two Inspired in part by biological learning systems, neural stages. First, creating an initial, large decision tree using a networks approach is built from a densely interconnected set set of training set. Second, pruning the initial decision tree, of simple units. Since this technique offers many design if applicable, using a validation set. Given a noise-free selections, we fixed some of them to the ones that had been training set, the first stage will generate a decision tree that well proven to be good or acceptable design choices. In can classify correctly all examples in the set. Except that the particular, we use a network architecture with one hidden training set covers all instances in the domain, the initial layer, the back-propagation learning algorithm (Rumelhart, decision tree generated will over fit the training data, which Hinton & William, 1986), the delta-bar-delta adaptive then reduce its performance in the test data. The second learning rates (Jacobs, 1988), and the Nguyen-Widrow stage helps alleviate this problem by reducing the size of the weight initialization (Nguyen & Widrow, 1990). Discrete- tree. This process has an effect in generalizing the decision valued attributes fed into the networks input layer is tree that hopefully could improve its performance in the test represented as 1-of-N encoding using bipolar values to data. denote the presence (e.g., value 1) and the absence (e.g., During the construction of an initial decision tree, the value –1) of an attribute value. Continuous-valued attribute selection of the best attribute is based on either the is scaled into a real value in the range [–1, 1]. We varied the information gain (IG) or gain ratio (GR). A binary split is design choices for batch versus incremental learning, and 1- applied in nodes with continuous-valued attributes. The best of-N versus single network output encoding. cut-off value of a continuous-valued attribute is locally 4) Naive Bayes selected within each node in the tree based on the remaining Naive Bayes classifier is a variant of the Bayesian learning training examples. The tree’s node expansion stops either that manipulates directly probabilities from observed data when the remaining training set is homogeneous (e.g., all and uses these probabilities to make an optimal decision. instances have the same target attribute values) or when no This approach assumes that attributes are conditionally attribute remains for selection. The decision on a leaf independent given a class. Based on this simplifying resulting from the latter case is determined by selecting the assumption, the probability of observing attributes’ majority of the target attribute value in the remaining conjunction is the product of the probabilities for the training set. individual attributes. Given an instance with a set of Decision tree pruning is a process of replacing sub-trees attribute-value pairs, the Naive Bayes approach will choose with leaves to reduce the size of the decision tree while a class that maximizes the conditional probability of the retaining and hopefully increasing the accuracy of tree's class given the conjunction of attributes values. Although in classification. To obtain the best result from the induction practice the independence assumption is not entirely correct, decision tree method, we varied the use of pruning it does not necessarily degrade the system performance algorithm to the initial decision tree. We considered using (Domingos & Pazzani, 1997). the following decision trees pruning algorithms: critical- We also assume that the values of continuous-valued value pruning (Mingers, 1987), minimum-error pruning attributes follow Gaussian distribution. Hence, once the (Niblett & Bratko, 1986), pessimistic pruning, cost- mean and the standard deviation of these attributes are complexity pruning and reduced-error pruning (Quinlan, obtained from the training examples, the probability of the 1987),. corresponding attributes can be calculated from the given 2) Induction Decision Trees with Boosting attribute values. To avoid zero frequency count problem that Decision Tree with Boosting is a method that generates a can dampen the entire probabilities calculation, we use an sequence of decision trees from a single training set by re- m-estimate approach in calculating probabilities of discrete- weighting and re-sampling the samples in the set (Quinlan, valued attributes (Mitchell, 1997). 1996; Freund & Schapire, 1996). Initially, all samples in the 4.3 Comparison results training set are equally weighted so that their sum is one. For each database, k-fold cross-validation method is used Once a decision tree has been created, the samples in the for evaluation. In this method, a data set is divided equally training set are re-weighted in such a way that misclassified into k disjoint subsets. k experiments are then performed examples will get higher weights than the ones that are using k different training-test set pairs. A training-test set easier to classify. The new samples weights are then pair used in each experiment is generated by using one of renormalized, and next decision tree is created using higher- the k subsets as the test set, and using the remaining subsets weighted samples in the training set. In effect, this process as the training set. Given k disjoint subsets, for example, the enforces the more difficult samples to be learned more first experiment takes the first subset for the test set, and frequently by decision trees. The trees generated are then uses the second subset through the kth subset for the training given weights in accordance with their performance in the set. The second experiment uses subset 1 and subset 3 training examples (e.g., their accuracy in correctly through subset k for the training set; and takes subset 2 for classifying the training data). Given a new instance, the new
- 6. the test set, and so on. All results from a particular database boosting, we present only the best experimental results after are averaged along with its variance over k experiment runs. varying the data splitting methods as well as the pruning Based on their size, the credit database and voting algorithms described earlier. Similarly, results from neural database are partitioned into 10 disjoint subsets, the heart networks approach are selected from the ones that provide database is partitioned into 5 subsets. Table 1, 2, 3 show the the best performance after varying the use of different performance comparison results of different approaches on network output encoding and batch versus incremental these three databases. For decision trees with and without learning methods. Table 1 The comparison results on the prediction accuracy and standard deviation (%) of credit database. Our GA Decision trees Decision trees Neural networks Naive Bayes approach (IG, Min-Err) with boosting (1-of-N, batch (RG, Cost-Com, learning) 21 trees) Run 1 90.77 87.69 89.23 89.23 66.15 Run 2 89.23 84.62 86.15 86.15 78.46 Run 3 89.23 89.23 90.77 90.77 84.62 Run 4 92.31 90.77 90.77 89.23 81.54 Run 5 86.15 81.54 81.54 84.62 75.38 Run 6 89.23 87.69 87.69 87.69 80.00 Run 7 84.62 81.54 84.62 84.62 73.85 Run 8 87.69 86.15 87.69 86.15 83.08 Run 9 90.77 86.15 89.23 87.69 76.92 Run 10 86.76 88.24 91.18 86.76 75.00 Average 88.68 86.36 87.89 87.29 77.50 Standard deviation 2.37 3.06 3.08 2.03 5.36 In above table, the decision tree is generated using data splitting and cost-complexity pruning algorithm. Batch information gain for data splitting and minimum-error learning and 1-of-N output encoding are used in neural pruning. Decision trees with boosting generates 21 different networks. decision trees, each is constructed by using gain ratio for Table 2 The comparison results on the prediction accuracy and standard deviation (%) of voting database. Our GA Decision trees Decision trees Neural networks Naive Bayes approach (IG, Red-Err) with boosting (1-of-N, batch (IG, Red-Err, learning) 3 trees) Run 1 95.35 95.35 95.35 93.02 93.02 Run 2 97.67 93.02 97.67 97.67 90.70 Run 3 97.67 97.67 97.67 97.67 93.02 Run 4 95.35 95.35 97.67 97.67 88.37 Run 5 95.35 95.35 95.35 93.02 88.37 Run 6 97.67 97.67 97.67 97.67 88.37 Run 7 95.35 93.02 93.02 95.35 93.02 Run 8 97.67 95.35 95.35 95.35 90.70 Run 9 100.00 100.00 97.67 95.35 90.70 Run 10 95.83 93.75 97.92 95.03 85.42 Average 96.79 95.65 96.54 95.86 90.17 Standard deviation 1.59 2.23 1.67 1.46 2.53 In table 2, the best results from both decision trees with Only three decision trees are needed in the decision trees and without boosting are obtained from using information with boosting. gain for data splitting and reduced-error pruning algorithm.
- 7. Table 3 The comparison results on the prediction accuracy and standard deviation (%) of heart database. Our GA Decision trees Decision trees Neural networks Naive Bayes approach (IG, Red-Err) with boosting (1-of-N, incr (IG, Red-Err, learning) 21 trees)
- 8. Run 1 89.83 81.36 88.14 81.36 89.83 Run 2 83.05 77.97 84.75 84.75 79.66 Run 3 81.36 76.27 76.27 74.58 83.05 Run 4 88.14 76.27 83.05 89.83 83.05 Run 5 83.33 66.67 71.67 81.67 80.00 Average 85.14 75.71 80.77 82.44 83.12 Standard deviation 3.64 5.46 5.54 5.66 4.08 In table 3, the best results from neural networks are Domingos, P. and Pazzani, M. (1997) On the Optimality of obtained from applying incremental learning and 1-of-N the Simple Bayesian Classifier under Zero-One Loss. network output encoding. Machine Learning, 29, 103-130. From the above results we can see that our GA approach Freund, Yoav, and Schapire, R. E. (1996) Experiments with outperformed other approaches on both the average a new boosting algorithm. In Machine Learning: prediction accuracy and the standard deviation. The Proceedings of the Thirteen International Conference, pp. advantage of our GA approach becomes more obvious on 148-156. heart database, which is most difficult to learn among the Greene, D., P. and Smith, S. F. (1993) Competition-based three. During the process of running, we also found that the induction of decision models from examples. Machine training accuracy and testing accuracy of GA approach are Learning, 13, 229-257. basically in the same level, while the training accuracy is Jacobs, R.A. (1988) Increased Rates of Convergence often much higher than testing accuracy for other Through Learning Rate Adaptation. Neural Networks, 1(4): approaches. This proves that GA approach is less sensitive 295-307. to noise and might be more effective for future prediction. Janikow, C. Z. (1993) A knowledge-intensive genetic algorithm for supervised learning. Machine Learning, 13, 5. Conclusions and future work 189-228. Mingers, J. (1987) Expert Systems – Rule Induction with In this paper we put forward a genetic algorithm approach Statistical Data. Journal of the Operational Research for classification problem. An individual in a population is a Society, 38, 39-47. complete solution candidate that consists of a fixed number Mitchell, Tom. (1997) Machine Learning. New York: of rules. Rule consequent is not explicitly encoded in the McGraw-Hill. string but determined by the match situation on training Niblett, T. (1986) Constructing Decision Trees in Noisy examples of the rule. To consider the performance affecting Domains. In I. Bratko and N. Lavrac (Eds). Progress in factors as complete as possible, four elements are included Machine Learning. England: Sigma Press. in the fitness function which are predicting error rate, Noda, E., Freitas, A. A. and Lopes, H. S. (1999) entropy measure, rule consistency and hole ratio, Discovering interesting prediction rules with a genetic respectively. Adaptive asymmetric mutation and two-point algorithm. In Proceedings of 1999 Congress on crossover are adopted in reproduction step. The inversion Evolutionary Computation (CEC’ 99), pp. 1322-1329. probability of 1-0 (0-1) in mutation is self-adaptive by the Nguyen, D. and Widrow, B. (1990) Improving the Learning feedback of average fitness during the run. The generated Speed of Two-Layer Networks by Choosing Initial Values classifier after evolution is voting-based. Rules are not of the Adaptive Weights. International Joint Conference on disjoint but allowed to overlap. Classifier gives all rules the Neural Networks, San Diego, CA, III:21-26. equal weight for their votes. We tested our algorithm on Quinlan, J.R. (1986) Induction of Decision Trees. Machine three real databases and compared the results with four Learning, 1, 81-106. other traditional data mining approaches. It is shown that Quinlan, J.R. (1987) Simplifying Decision Trees. our approach outperformed other approaches on both International Journal of Man-Machine Studies, 27, 221-234. prediction accuracy and the standard deviation. Quinlan, J. R. (1996) Bagging, Boosting, and C4.5. In Further testing on various databases is in progress to test Proceedings of the Thirteenth National Conference on the robustness of our algorithm. Splitting continuous Artificial Intelligence, pp. 725-730. attribute into multiple intervals rather than just two intervals Rumelhart, D. E., Hinton, G.E., and William, R. J. (1986) based on a single threshold is also considered to improve the Learning Representations by Back-Propagation Error. performance. Nature, 323:533-536. Yang, L. and Yen, J. (2000) An adaptive simplex genetic Bibliography algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2000), July 2000, pp. De Jong, K. A., Spears, W. M., and Gordon, D. F. (1993) 379. Using genetic algorithms for concept learning. Machine Learning, 13, 161-188.