Device status anomaly detection
Download as PPTX, PDF5 likes1,261 views
This document discusses using k-means clustering in Spark to detect device anomalies based on device feature data. It provides an example of device data with attributes like battery percentage and RAM usage. It also shows example Scala code to perform k-means clustering on this data, including normalizing the data first before clustering. The results show data points clustered and predictions assigned.
) ((a,b) => a.zip(b).map(t => t._1 + t._2*t._2))
val stdevs = sumSquares.zip(sums).map { case(sumSq,sum) => sqrt(n*sumSq - sum*sum)/n }
val means = sums.map(_ / n)
val normalizedData = data.map(
(_,means,stdevs).zipped.map((value,mean,stdev) =>
if (stdev <= 0) (value-mean) else
(value-mean)/stdev)).cache()
val kScores = (50 to 120 by 10).par.map(k => (k, clusteringScore(normalizedData, k)))](https://image.slidesharecdn.com/devicestatusanomalydetection-140905182944-phpapp02/85/Device-status-anomaly-detection-6-320.jpg)
![Result
([70.0,15.0,70.0,89.0,3.0],2)
([33.0,49.0,79.0,90.0,3.0],1)
([75.0,16.0,68.0,90.0,4.0],2)
([33.0,49.0,79.0,90.0,3.0],1)
([60.0,19.0,67.0,90.0,3.0],2)
([33.0,49.0,79.0,98.0,3.0],1)
([65.0,19.0,67.0,90.0,3.0],2)
([43.0,49.0,79.0,90.0,3.0],1)
([67.0,17.0,67.0,90.0,3.0],2)
([53.0,49.0,78.0,90.0,3.0],1)
([68.0,19.0,69.0,90.0,3.0],2)
([38.0,49.0,79.0,90.0,3.0],1)
([68.0,19.0,69.0,90.0,3.0],2)
([38.0,49.0,89.0,90.0,3.0],1)
([68.0,19.0,69.0,90.0,3.0],2)
([68.0,19.0,69.0,90.0,3.0],2)
([68.0,19.0,89.0,80.0,4.0],0)](https://image.slidesharecdn.com/devicestatusanomalydetection-140905182944-phpapp02/85/Device-status-anomaly-detection-7-320.jpg)
Device status anomaly detection
- 1. Device anomaly detection using Spark k-means 9/3/2014
- 2. Introduction Detect device anomaly based on the device information ( feature property vector ) ● battery % ● cpu % ● RAM % ● wifi strength ● build number ( in numerical ) ● exception ● charging ● gps long ● gps lat ● bundle version
- 3. K-means clustering Clustering is an unsupervised learning problem whereby we aim to group subsets of entities with one another based on some notion of similarity. Clustering is often used for exploratory analysis and/or as a component of a hierarchical supervised learning pipeline (in which distinct classifiers or regression models are trained for each cluster). MLlib supports k-means clustering, one of the most commonly used clustering algorithms that clusters the data points into predefined number of clusters.
- 4. Example Data battery, cpu, RAM, wifi, exception, charging count 70.00 15.00 70.00 89.00 3.00, 3 75.00 16.00 68.00 90.00 4.00, 0 60.00 19.00 67.00 90.00 3.00, 0 65.00 19.00 67.00 90.00 3.00, 0 67.00 17.00 67.00 90.00 3.00, 0 68.00 19.00 69.00 90.00 3.00, 0 68.00 19.00 69.00 90.00 3.00, 0 68.00 19.00 69.00 90.00 3.00, 0 68.00 19.00 89.00 80.00 4.00, 0 33.00 49.00 79.00 90.00 3.00, 0 33.00 49.00 79.00 90.00 3.00, 0 33.00 49.00 79.00 98.00 3.00, 0 43.00 49.00 79.00 90.00 3.00, 0 53.00 49.00 78.00 90.00 3.00, 0 38.00 49.00 79.00 90.00 3.00, 0 38.00 49.00 89.00 90.00 3.00, 0 68.00 19.00 69.00 90.00 3.00, 0
- 5. Example Scala code import org.apache.spark.mllib.clustering.KMeans import org.apache.spark.mllib.linalg.Vectors val data = sc.textFile("data/device_anomaly.txt").map { line => Vectors.dense(line.split(' ').map(_.toDouble))}.cache() val K = 3 val maxIteration = 20 val runs =20 val clusters= KMeans.train(data, K, maxIteration, runs) val vectorsAndClusterIdx = data.map{ point => val prediction = clusters.predict(point) (point.toString, prediction) } vectorsAndClusterIdx.foreach ( k => printf(k.toString()))
- 6. Normalize data.unpersist(true) val numCols = data.take(1)(0).length val n = data.count val sums = data.reduce((a,b) => a.zip(b).map(t => t._1 + t._2)) val sumSquares = data.fold(new Array[Double](numCols)) ((a,b) => a.zip(b).map(t => t._1 + t._2*t._2)) val stdevs = sumSquares.zip(sums).map { case(sumSq,sum) => sqrt(n*sumSq - sum*sum)/n } val means = sums.map(_ / n) val normalizedData = data.map( (_,means,stdevs).zipped.map((value,mean,stdev) => if (stdev <= 0) (value-mean) else (value-mean)/stdev)).cache() val kScores = (50 to 120 by 10).par.map(k => (k, clusteringScore(normalizedData, k)))
- 7. Result ([70.0,15.0,70.0,89.0,3.0],2) ([33.0,49.0,79.0,90.0,3.0],1) ([75.0,16.0,68.0,90.0,4.0],2) ([33.0,49.0,79.0,90.0,3.0],1) ([60.0,19.0,67.0,90.0,3.0],2) ([33.0,49.0,79.0,98.0,3.0],1) ([65.0,19.0,67.0,90.0,3.0],2) ([43.0,49.0,79.0,90.0,3.0],1) ([67.0,17.0,67.0,90.0,3.0],2) ([53.0,49.0,78.0,90.0,3.0],1) ([68.0,19.0,69.0,90.0,3.0],2) ([38.0,49.0,79.0,90.0,3.0],1) ([68.0,19.0,69.0,90.0,3.0],2) ([38.0,49.0,89.0,90.0,3.0],1) ([68.0,19.0,69.0,90.0,3.0],2) ([68.0,19.0,69.0,90.0,3.0],2) ([68.0,19.0,89.0,80.0,4.0],0)
- 8. Heatmap ( sample ) Venue MAC Time CPU Battery Sneakers 0A-94-05- F7-93 9/1/2014 7:30:20 89% 13% McCoverys 0A-94-05- F7-76 9/3/2014 5:30:20 73% 10% ...
- 9. References ● https://www.youtube.com/watch?v=TC5cKYBZAeI ● https://www.youtube.com/watch?v=FjhRkfAuU7I ● http://www.ebaytechblog.com/2014/05/28/using-spark- to-ignite-data-analytics/#.VAc0PWRdXCw ● http://stanford.edu/~rezab/sparkworkshop/slides/xia ngrui.pdf ● http://stanford.edu/~rezab/sparkworkshop/slides/xia ngrui.pdf