Integrating compression technique for data mining
Download as PPT, PDF1 like284 views
This document outlines a presentation on data mining techniques. It discusses data compression methods like null compression and run length encoding. It also discusses association rule mining and the Apriori algorithm limitations. The problem statement proposes a method for compressing databases that can be decompressed while also improving data mining performance. The proposed work involves compressing data into groups, generating frequent itemsets using Apriori on the compressed data, then decompressing and generating association rules. The implementation environment and conclusions are also outlined. References on related work are provided at the end.
Integrating compression technique for data mining
- 1. DR MANMOHAN SINGH Assistant professor ITM UNIVERSE VDODARA GUJARAT INDIA
- 2. Presentation Outline Introduction Compression Technique Association Rule Mining Limitation Of Apriori Literature Survey Problem Statement Proposed Work Implementation Enviroment Conclusion References
- 3. What Is Data Mining Data mining is used to help users discover interesting and useful knowledge more easily. Data compression is one of good solutions to reduce data size. Data pre-process transforms the original database into a new data representation. It generates a new transaction database at the end of the data pre-process step.
- 4. What Is Data Mining The figure shows data mining as a step in an iterative knowledge discovery process.
- 5. Why Data Mining? Data is scattered over network. so it is difficult to find the actual data. Data mining helps to find that data. A business man wants to grow up his business. For that he needs smart data, techniques ,models , tools etc. Data mining helps how we get, use & understand that data. . There is a need to extract useful information from the data and to interpret the data.
- 6. Application Financial Data Analysis Retail Industry Telecommunication Industry Biological Data Analysis Other Scientific Applications Intrusion Detection
- 7. Issues Mining Methodology User Interaction Performance Issues Diverse Data Types Issues
- 8. Compression technique? Make optimal use of limited storage space. It reduces the size of the data and improves I/O performance. Compression has also been recently applied for reading large scientific files in parallel file systems. Compression decrease bandwidth consumption on networks, and reduce energy consumption in hardware. Compression has been used extensively in wireless networks.
- 9. Types Of Compression Techniques Null Compression: Replaces a series of blank spaces with a compression code. Run length Compression:- Expands on the null compression, by compressing a series of four repeating characters. Keyword Encoding:- Creates a table with values that represent common sets of character. Adaptive Huffman Coding:-Assign fewer bits to symbols that occur more frequently and more bits to symbols appear less often. Lempel Ziv Compession:- Building an indexed dictionary Compressing a string of symbols
- 10. Association Rule Mining It is a method for discovering interesting relations between variables in large databases. Intended to identify strong rules discovered in databases using different measures of interestingness. Many Algorithms had been proposed for finding the strong association between the data sets. In which Apriori was the most well known association rule algorithm which was developed in 1994, having some major issues.
- 11. Limitations of Apriori Needs several iterations for the scanning of the data. Difficulties to find rarely occuring events. Works for small set of data. Costly wasting of time to hold a vast number of candidate sets.
- 12. Sr No Reference Paper Methodology Used Future Work 1 Integrating Compression and Execution in ColumnOriented Database Systems by Daniel J. Abadi,Samuel R. Madden,Miguel & C.Ferreira. Column-Oriented Database system architecture NIL 2 Integrating Online Compression To Accelerate Large-Scale Data Analytics Application. By Tekin Bicer, Jian Yin,. David Chiu,Gagan Agrawal,& Karen Schuchardt Chunk Resource Allocation , Parallel Compressioon Engine NIL 3 Efficient Mining Frequent Itemsets Algorithms.By Marghny H. Mohamed, & Mohammed M. Darwieesh. Count Table , Binary Count Table Extend the algorithms to mine other kinds of patterns, such as sequential patteern mining problem, 4 A Transaction Mapping Algorithm For Frequent Itemsets Mining By Mingjun Song, & Sanguthevar Rajasekaran. Transaction Mapping Algorithm To Improve the implementation of the TM algorithm and make a fair comparison with FP-growth.
- 13. Sr No Reference Paper Methodology Used Future Work 5. Compact Transaction Database For Efficient Ffrequent Pattern Mining By Qian Wan & Aijun An. Compact Tree Structure Called CT- tree NIL 6. A New Association Rules Mining Algorithm Based On Vector By xin Zhang, Pin Liao & Huiyong Wang. Association rule mining algorithm based on vector. NIL
- 14. Problem Statement They all lack the ability to decompress the data to their original state and improve the data mining performance.. It is even a bigger challenge to maintain the compressed database in the future It spends too much time to check candidate itemsets in the data mining step. Unable to enter the data set at runtime
- 15. Original database Sorted database Sorted database Group1 Sorted database Group2 Sorted database Group3 Compressed dataset and generate merged group Compressed transaction dataset Generate frequent item set by simple apriori algorithms Now generate association rules and uncompressed dataset
- 16. Proposed Work The main criteria of research are related to the followings:- (a) The compressed database can be decompressed to the original form. (b) Reduce the process time of association rule mining by using a quantification table. (c) Reduce I/O time by using only the compressed database to do data mining. (d) Allow incremental data mining.
- 17. Implementation Enviroment Minimum Hardware Requirement: 1. 3 GHZ Pentium PC Machine. 2. 512 Megabytes Main Memory 3. Screen Resolution needs to be between 800*600 & 1200*800. Minimum Software Requirement: 1. Operating system microsoft windows XP. 2. Microsoft Visual Studio.net(C#).
- 18. Conclusion Rapid Increase of large data become a point of concern. i.e, time required for data pre-process. Hence, the proposed algorithm can be benificial while dealing with such large data. As, it can decompressed the data also after compression. It can also reduce the I/O time by using only compressed database.
- 19. References 1. Xin Zhang, Pin Liao and Huiyong Wang ”A New Association Rules Mining Algorithm Based On Vector”, 2009 Third International Conference on Genetic and Evolutionary Computing 2. Qian Wan And Aijun An” Compact Transaction database For Efficient Frequent Pattern Mining” Department of Computer Science and Engineering York University, Toronto, Ontario, M3J 1P3, Canada 3. Jis-Yu Dai, Don-lin Yang, Jungpin Wu, And Ming-Chuan Hung-” An Efficient Data Mining Approach on Compressed Transactions.” International Journal of Electrical and Computer Engineering 3:2 2008
- 20. References 4. Wael Ahmad AlZoubi, Khairuddin Omar, Azuraliza Abu Bakar” An Efficient Mining of Trasactional Data Using Graph-Based Technique” 2011 3rd Conference on Data Mining and Optimization (DMO) 28-29 June 2011, Selangor, Malaysia 5. Mingjun Song And Sanguthevar Rajasekaran, “A Transaction Mapping Algorithm For Frequent Itemsets Mining” IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, October 2005. 6. Marghny H. Mohamed, Mohammed M. Darwieesh,”Efficient Mining Frequent Itemsets Algorithm”. Revised: 7 March 2012/Accepted 29 April 2013 Springer- Verlag Berlin Heidelberg 2013.
- 21. References 7. Fan Zhang, Yan Zhang Jason Bakos,” GP Apriori: GPU-Accelerated Frequent Itemset Mining”. 2011 IEEE International Conference On Cluster Computing 8. Tekin Bicer, Jian Yin, David Chiu, Gagan Agrawal And Karen Schuchardt“ Integrating Online Compression To Accelerate large-Scale Data Analytics Application”. 2013 IEEE 27th International Sympoosium on parallel & distributed processing. 9. Daniel J. Abadi, Samuel R. Madden, Miguel C. Ferreira”Integrating Compression And Execution In Column-Oriented Database Systems”, SIGMOD 2006, June 27–29, 2006, Chicago, llinois, USA.Copyright 2006 ACM 1595932569/06/0006.
- 22. References 10. Shalini Dutt, Naveen Choudhary & Dharm Singh, “ An Improved Apriori Algorithm Based On Matrix Data Structure”, Global Journal Of Computer Science And Technology : C Software & Data Engineering, Vol. 14 Issues 5/Version 1.0 Year 2014. 11. Wael A.ALZoubi, Azuraliza Abu Bakar, Khairuddin Omar, “Scalable And Efficient Method For Mining Association Rules, ”2009 International Conference On Electrical Engineering And Infrmatics 5-7 August 2009, Selangor Malaysia. 12. Loan T.T.Nguyen, Bay Vo, Tzung-Pei Hong,Hoang Chi Thanh,“CAR-Miner: An Efficient Algorithm For Mining Class-Association Rules,”Expert system With Applications 40(2013) 2305-2311, 2012@Elsevier Ltd. All Rights.
- 23. References 10. Mohammed Al-Maolegi, Bassam Arkok, “An Improved Apriori Algorithm For Association Rules ,” International Journal On Natural Language Computing(IJNLC) Vol. 3, N.1, Feburary 2014.
- 24. ANY QUERY?