Overview of Data Mining
Download as PPT, PDF3 likes1,366 views
This document discusses data mining and provides several examples of its applications: 1) Data mining can be used to build customer profiles, analyze credit ratings, detect fraud and intrusions, and make recommendations on websites. 2) Environmental data mining can discover rules in geo-spatial data, analyze weather impacts, and find patterns in Earth science variables. 3) Common data mining techniques include classification, clustering, and association rule mining, which are used to organize data, group similar items, and discover relationships.
Overview of Data Mining
- 1. Overview of Data Mining Meeting of WP Data Mining April 28, 2008 Bowo Prasetyo http://www.scribd.com/prazjp http://www.slideshare.net/bowoprasetyo This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation In Slide Show, click on the right mouse button Select “Meeting Minder” Select the “Action Items” tab Type in action items as they come up Click OK to dismiss this box This will automatically create an Action Item slide at the end of your presentation with your points entered.
- 2. Contents What Is Data Mining? Does It Differ To Statistics? Why Uses Data Mining? What Can Data Mining Do? Methods of Data Mining Contoh Kasus - Toserba Mining Environmental Data Conclusion
- 3. What Is Data Mining? The exploration and analysis of large quantities of data in order to discover meaningful patterns and rules 1) . 1) Berry and Linoff, Data Mining Techniques for Marketing, Sales and Customer Support (Book), 1997
- 4. Does It Differ To Statistics? Data mining is a blend of statistics, artificial intelligence, and database research 16) . 16) D. Pregibon, Data Mining: Statistical Computing and Graphics , p. 7-8, 1997 Statistics Artificial Intelligence Database Data Mining
- 5. Statistics, AI, Database Statistics Distribution, mean, median, standard deviation Artificial Intelligence (AI) Neural network, fuzzy theory, genetic algorithm, particle swarm optimization Database Relational, object-oriented, spatial, temporal
- 6. Why Uses Data Mining? Data explosion Automated data collection Log data of large organizations 2) : 44% 1 terabyte per month 11% 10 terabytes per month World’s digital data on PCs, digital cameras, servers, sensors, etc. 3) : in 2006 161 billion gigabytes In 2010 988 billion gigabytes (predicted) Large amounts of data, but small amounts of knowledge Data mining to discover the knowledge 2) ESG Research, New ESG Research Finds Large Organizations Experiencing Explosive Growth in Log Data Collection, Analysis, and Storage , 2007 ( http://www.enterprisestrategygroup.com/_documents/NewsEvent/NewsEvent439.pdf ) 3) EMC — IDC Research, The Expanding Digital Universe: A Forecast of Worldwide Information Growth Through 2010 , 2006 ( http://www.emc.com/about/destination/digital_universe/ )
- 7. What Can Data Mining Do? Examples
- 8. On Business and Network Security Builds customer profiles based on his/her transactional histories 4) Analyzes corporate credit ratings using public financial statements, such as financial ratios 5) Detects credit card fraud by analyzing customer transaction database 6) Detects network intrusion based on system program behavior such as sendmail and tcpdump 7) 4) G. Adomavicius and A. Tuzhilin, Using data mining methods to build customer profiles , in Computer magazine p. 74-82, 2001 5) Z. Huang, H. Chen, C. Hsu, W. Chen, S. Wu, Credit rating analysis with support vector machines and neural networks: a market comparative study , in Journal of Decision Support Systems p. 543-558, 2004 6) T. Fawcett and F. Provost, Adaptive Fraud Detection , in Journal of Data Mining and Knowledge Discovery p. 291-316, 2004 7) W. Lee and S. J. Stolfo, Data Mining Approaches for Intrusion Detection , in Proceedings of the 7th USENIX Security Symposium, 1998
- 9. On The Web Discovers useful patterns from log files, contents, and links of websites 8) Ranks the web pages on the internet using link structure analysis 9) Personalizes a website based on log files, contents, and profile data 10) Supports on-line recommendation to customers by analyzing e-commerce transaction records 11) 8) R. Cooley, B. Mobasher, J. Srivastava, Web Mining: Information and Pattern Discovery on the World Wide Web , in Proceedings of 9th International Conference on Tools with Artificial Intelligence (ICTAI) p. 0558, 1997 9) Larry Page, Sergey Brin, R. Motwani, T. Winograd, The PageRank Citation Ranking: Bringing Order to the Web , 1998 ( http://citeseer.ist.psu.edu/page98pagerank.html ) 10) M. Eirinaki and M. Vazirgiannis, Web mining for web personalization , in ACM Transactions on Internet Technology (TOIT) p. 1- 27, 2003. 11) S. W. Changchien and T. Lu, Mining association rules procedure to support on-line recommendation by customers and products fragmentation , in Journal of Expert Systems with Applications v. 20-4 p. 325-335, 2001
- 10. On Environment Discovers rules in geo-spatial database 12) Analyzes weather impacts on airspace system 13) Discovers interesting patterns on Earth Science variables (soil moisture, temperature, precipitation) along with ecosystem data (Net Primary Production) 14) Finds Ocean Climate Indices based on pressure and temperature data 15) 12) J. Han, K. Koperski, N. Stefanovic, GeoMiner: a system prototype for spatial data mining, in Proceedings of ACM SIGMOD international conference on Management of data p. 553 - 556, 1997 13) Z. Nazeri and J. Zhang, Mining aviation data to understand impacts of severe weather on airspace system performance , in Proceedings of International Conference on Coding and Computing p. 518- 523, 2002. 14) V. Kumar, M. Steinbach, P. Tan, S. Klooster, C. Potter, A. Torregrosa, Mining Scientific Data: Discovery of Patterns in the Global Climate System , in Proceedings of the Joint Statistical Meetings p. 5--9, 2001 15) M. Steinbach, P. Tan, V. Kumar, S. Klooster, C. Potter , Data Mining for the Discovery of Ocean Climate Indices , in Proceedings of the 5th Workshop on Scientific Data Mining p. 7-16, 2002
- 11. Methods in Data Mining Basic Methods
- 12. Classification, Clustering, Association Rules Data mining consists of several basic methods: Classification Places items into groups based on a training set of previously labeled items (supervised) Clustering Places items into groups based on some defined distance measure (unsupervised) Association Rules Discovers items that co-occur frequently within a data set and also their rules, such as implication or correlation
- 13. Classification Naive Bayesian classifier Spam/Non-spam classification Spam if 17) http ://en.wikipedia.org/wiki/Naive_Bayes_classifier
- 14. Clustering K-means algorithm 18) Partitions items into k clusters Calculates mean of each cluster as centroid Associates each items to the closest centroid using defined distance Back to 2 until convergence 18) J. A. Hartigan and M. A. Wong, A k-means clustering algorithm, in Applied Statistics, 28 (1) p. 100-108, 1979
- 15. Association Rules If a customer buys bread and butter, then she will likely buy milk too with 90% confidence Algorithm 19) : Finds frequent itemsets whose support >= minsup Finds interesting rules from frequent itemsets above whose confidence >= minconf 19) R. Agrawal, R. Srikant, Fast Algorithms for Mining Association Rules , in Proc. 20th Int. Conf. Very Large Data Bases, VLDB, 1994
- 16. Association Rules Apriori algorithm to find frequent itemsets L in database D 19) : Find frequent set L k −1 Join step C k is generated by joining L k −1 with itself Prune step Any ( k −1)-itemset that is not frequent cannot be a subset of a frequent k -itemset, hence should be removed ( C k : Candidate itemset of size k ) ( L k : frequent itemset of size k whose support >= minsup )
- 17. Association Rules Apriori algorithm to find rules R from frequent itemsets L 19) : For each l L generate S = non-empty subsets of l For each s S generate rule s ( l - s ) if confidence >= minconf
- 18. Visualization Of Mining Results Problem of mining results Too much results to display Difficult to find important rules Difficult to understand the rules Needs good visualization tools Chart for statistical results Graph (node & edge) for association rules Globe map for geo-spatial results Animation for temporal results Utilizes colors, styles, thickness etc.
- 19. Contoh Kasus Aturan Asosiasi di Toserba
- 20. Item dan Transaksi Pembelian Pak Joko bulan Januari: beras, minyak goreng, daging sapi gula pasir, minyak goreng, telur ayam beras, gula pasir, minyak goreng, telur ayam gula pasir, telur ayam transaksi item
- 21. Frequent Item (Item Sering) “ Sering”: pembelian >= 2 daging sapi = 1 kali bukan sering support minimum support
- 22. n -Length Item ( n -Item) n > 1 2-length item 3-length item
- 23. Aturan Asosiasi Kustomer yang membeli beras akan membeli juga minyak goreng. “ jika beras maka minyak goreng" beras => minyak goreng support(minyak goreng & beras) support(beras) = 2/2 = 1 confidence antecedent consequent
- 25. Mining Environmental Data Examples
- 26. Explosion in Environmental Data Temperature, humidity, pressure, precipitation, sound, light, shock Weather & rainfall trends, river height & flows, air & water quality, pollution levels, salinity, emissions, FPAR, NPP Earth science, oceanography, meteorology, ecology Sensors, hand-held/wireless devices, remote sensing (satellites), other automated logging devices
- 27. Geo-spatial Database Discovers rules in geo-spatial database 12) Given Western Canada, describe the weather patterns Given temperature, precipitation, etc., describe the regions Show the differences in weather patterns between British Columbia and Alberta If a Canadian town is large and is adjacent to large water body, then it is close to the U.S. border, with the possibility of 78% GeoMiner
- 28. Earth Science Interesting patterns on Earth Science 14) Regions that are covered by the highly correlated pattern, FPAR-Hi NPP-Hi Shrubland regions FPAR: Fractional Intercepted Photosynthetically Active Radiation NPP : Net Primary Production
- 29. Earth Science Interesting patterns on Earth Science 14) Two clusters for NPP (land) and two clusters for SST (ocean). The clusters approximate the northern and southern hemispheres, for land and ocean. SST: sea surface temperature
- 30. Earth Science Interesting patterns on Earth Science 14) Clusters of ocean near the Philipines (SST) and lands of Eastern Brazil, Southern Africa, and a bit of Australia (NPP) is highly correlated (0.47). In particular, this sea region is highly correlated (0.66), with SOI, which is a climate index related to El Niño, and it is known that parts of Southern Africa and Australia experience droughts related to El Nino.
- 31. Conclusion Today’s data repository is huge and collected in enormous speed Traditional statistical methods are no longer sufficient to analyze data. Data mining is very important to discover knowledge hidden in data Helps decision making in broad range of fields: business, network security, web, environment etc. Good visualization tool is needed to understand mining results easily