Anomaly Detection Via PCA
Download as PPTX, PDF3 likes8,166 views
This document describes an online over-sampling principal component analysis (osPCA) algorithm for detecting outliers in large datasets. Unlike prior PCA approaches, osPCA does not store the entire data matrix or covariance matrix, making it suitable for online or large-scale problems. It works by duplicating potential outlier instances instead of removing them to amplify their effect on the principal components. This allows osPCA to identify outliers by observing variations in the principal directions with and without each data point. The approach can also detect new outliers in an online setting by quickly updating the principal directions for new data.
Anomaly Detection Via PCA
- 1. Anomaly Detection via Online Over- Sampling Principal Component Analysis
- 2. Guide NAME USN Kumara BG 1NT11CS408 Mahesha GR 1NT11CS409 Mallikarjun S 1NT11CS410 Deepak Kumar 1NT10CS129 Ms.Nirmala Senior lecturer Dept of CSE
- 3. Problem Statement We propose an online over-sampling principal component analysis (osPCA) algorithm and it is detecting the presence of outliers from a large amount of data. Unlike prior PCA based approaches, we do not store the entire data matrix or covariance matrix, and thus our approach is especially of interest in online or large- scale problems.
- 4. Introduction We are drowning in the deluge of data that are being collected world-wide, while starving for knowledge at the same time. Anomalous events occur relatively infrequently
- 5. What are Anomalies? Anomaly is a pattern in the data that does not conform to the expected behaviour Also referred to as outliers, exceptions, peculiarities, surprise, etc. Anomalies translate to significant (often critical) real life entities ◦ Credit card fraud ◦ An abnormally high purchase made on a credit card
- 6. Motivation National / International Journals
- 7. Objectives The aim for this project is to detect the presence of outliers in a very large sampled data by finding the : ◦ Covariance matrix ◦ EigenValues ◦ EigenVectors, which are the direction of principal component ◦ Find Coordinates of each point in the direction of principal component
- 8. Hardware Specification: Processor - Pentium –IV RAM - 256 MB(min) Hard Disk - 20 GB Key Board - Standard Windows Keyboard Mouse - Two or Three Button Mouse
- 9. Software Specification Operating System : Windows XP Programming Language : JAVA Java Version : JDK 1.6 & above. IDE tool : ECLIPSE
- 10. Literature Survey: Research Paper Referred : Anomaly Detection Via Online Oversampling Principal Component Analysis by Yuh-Jye Lee, Yi-Ren Yeh and Yu-Chiang Frank Wang Other References: A Survey on Intrusion Detection Using Outlier Detection Techniques by V. Gunamani, M. Abarna
- 11. Design Of the Project :
- 12. Algorithm- Principal Component Analysis : PCA is a dimension reduction method. PCA is sensitive to outliers and we only need few principal components to represent the main data structure. An outlier or a deviated instance will cause a larger effect on these principal directions. With PCA outliers are detected by means of “Leave One Out” procedure .
- 13. We explore the variation of the principal directions with removing or adding a data point and use this information to identify outliers and detect new arriving deviated data The effect of LOO with a particular data may be diminished when the size of the data is large. An outlier via LOO strategy, we duplicate the target instance instead of removing it. Finally, we duplicate the target instance many times (10% of the whole data in our experiments) and observe how much variation do the principal directions
- 14. Implementation: It includes two steps : Data Cleaning Phase On-line Anomaly Detection Phase Data Cleaning Phase :The osPCA is applied for the data set for finding the principal direction. In this method the target instance will be duplicated multiple times, and the idea is to amplify the effect of outlier rather than that of normal data. After that using Leave One Out (LOO) strategy, the angle difference will be identified. In which if we add or remove one data instance, the direction will be changed.
- 15. On-line Anomaly Detection Phase : In the on-line anomaly detection phase, the goal is to identify the new arriving abnormal instance. The quick updating of the principal directions given in this approach can satisfy the on-line detecting demand. A new arriving instance will be marked .
- 16. Snapshots :
- 19. Outcomes We have explored the variation of principal directions in the leave one out scenario. We demonstrated that the variation of principal directions caused by outliers indeed can help us to detect the anomaly. The over-sampling PCA to enlarge the outlierness of an outlier.
- 20. Conclusion : This project has attempted to establish the significance of anomaly detection using osPCA technique. Our method does not need to keep the entire covariance or data matrices during the online detection process. Compared with other anomaly detection methods, our approach is able to achieve satisfactory results while significantly reducing computational costs and memory requirements.
- 21. Future Enhancement : In this Project we are working on a particular data set that we got from an online website but in future we’ll work on any data set to detect the anomalies.
- 22. Thank You