Network Intrusion Detection Analysis using Random Forest Algorithm on Apache Mahout
1 like860 views
The document discusses using Apache Mahout, an Apache library for scalable machine learning, to classify network traffic data using decision trees. It describes preprocessing the NSL-KDD dataset, building a random forest model on Hadoop to classify the training data, and testing the model on separate test data while computing classification metrics.
![The dataset we are using is the NSL-KDD
dataset [2]. It is an improvement on the
KDD
99 dataset [1]. The KDD 1999 Data Set
was a set of data of simulated computer
network intrusion.](https://image.slidesharecdn.com/apachemahoutfinalproject-141211025851-conversion-gate01/85/Network-Intrusion-Detection-Analysis-using-Random-Forest-Algorithm-on-Apache-Mahout-7-320.jpg?cb=1676442024)
Network Intrusion Detection Analysis using Random Forest Algorithm on Apache Mahout
- 2. Group 7 Dec 08/12/2014 Prof Shanoan Tian
- 3. • Uses Decision trees as a base. Normally one tree is used Tree is explored in depth - many branches; many leaves It has to be monitored and tailored by a trained statistician
- 4. •Many decision trees •Shallow exploration •Slightly different dataset for each •Specified questions
- 6. • Apache Mahout is a library implemented on top of Apache Hadoop • Scalable machine-learning algorithms • Using the MapReduce paradigm. • Data Mining tools for the data stored on a Hadoop system • Clustering • Classification • Batch based collaborative filtering
- 7. The dataset we are using is the NSL-KDD dataset [2]. It is an improvement on the KDD 99 dataset [1]. The KDD 1999 Data Set was a set of data of simulated computer network intrusion.
- 8. An Apache Software Foundation project to create scalable machine learning libraries http://mahout.apache.org Why Mahout? Scalable Machine Learning Algorithms Map Reduce Implementations on Apache Hadoop
- 9. Apache Mahout has several classification algorithms implementations Naïve Bayes Complementary Naïve Bayes - Random Forest Hidden Markov Models Logistic Regression
- 10. Most algorithms have a Driver program Shell script in $MAHOUT_HOME/bin helps with most tasks Prepare the Data Different algorithms require different setup Run the algorithm Single Node Hadoop Print out the results Several helper classes: ClusterDumper, etc.
- 13. Make directory in HDFS $ hadoop fs -mkdir testdata Load data to HDFS $ hadoop fs -put ./downloads/data/* testdata Verify data loading $ hadoop fs -ls testdata
- 14. hadoop jar mahout-examples-0.9-job.jar org.apache.mahout.classifier.df.tools.Describ e -p testdata/KDDTrain+_20Percent.arff -f testdata/KDDTrain+_20Percent.info -d N 3 C 2 N C 4 N C 8 N 2 C 19 N L The "N 3 C 2 N C 4 N C 8 N 2 C 19 N L” describes all the attributes of the data. 1 numerical(N) attribute, followed by 3 Categorical(C) attributes, ...L indicates the label
- 16. 14/12/08 07:37:53 INFO mapreduce.BuildForest: Build Time: 0h 0m 25s 949 14/12/08 07:37:53 INFO mapreduce.BuildForest: Forest num Nodes: 62706 14/12/08 07:37:53 INFO mapreduce.BuildForest: Forest mean num Nodes: 627 14/12/08 07:37:53 INFO mapreduce.BuildForest: Forest mean max Depth: 14
- 17. hadoop jar mahout-examples-0.9-job.jar org.apache.mahout.classifier.df.mapreduce.T estForest -i testdata/KDDTest+.arff -ds testdata/KDDTrain+_20Percent.info -m nsl-forest -a -mr -o predictions Predicts on "KDDTest+.arff" dataset (-i argument) using the same data descriptor generated for the training set (-ds) and the decision forest built previously (-m) computes the confusion matrix (-a) Passing the (-mr) parameter to use Hadoop Mapreduce framework
- 18. Correctly Classified Instances 17639 78.2425% In correctly Classified Instances 4905 21.7575% Total Classified Instances 22544 100%
- 19. Confusion Matrix A = normal B = anamoly Total 9454 257 9711 4648 8185 12833
- 20. hadoop jar mahout-examples-0.9-job.jar org.apache.mahout.classifier.df.mapreduce.TestForest - i testdata/KDDTest+.arff -ds testdata/KDDTrain+_20Percent.info -m nsl-forest -a -mr -o predictions Predicts on "KDDTest+.arff" dataset (-i argument) using the same data descriptor generated for the training set (- ds) and the decision forest built previously (-m) computes the confusion matrix (-a) Passing the (-mr) parameter to use Hadoop Mapreduce framework http://nsl.cs.unb.ca/NSL-KDD/