Web Access Log Management
2 likes542 views
This document discusses optimizing database management of large-scale web access logs. It proposes using a pre-processor to hash and sort logs in memory before writing to the database. An experiment compares the performance of using the pre-processor versus directly writing to the database. The results show the pre-processor is 18-20 times faster for input time and memory usage is twice as high but run time is much better compared to only using the database. The document concludes the proposed approach of using a pre-processor for in-memory processing before database storage provides better performance and optimization than traditional approaches.
Web Access Log Management
- 1. Database Optimization for Large Scale Web Access Log Management
- 2. Catalogue • Overview • Introduction • Performance Evaluation • Experiment Data Related Concepts & Terms • Worst Case Web Access Log Data • Log File • Hash • • Experiment Environment • • • Bucket • • • • • • JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) Run Time • Transaction Processing Sorting Frame • Log Analysis Algorithm Creation Of Statistical Information • Pre-Processor Writing Data to File • System Analysis & Design Data Input Time • Optimization Of Database Design • Experiment Result Memory Usage Conclusion 2
- 3. Overview • We will be talking about an optimal method of commercial DBMS for analysing largescale web access log. Also, we develop the pre-processor in memory structure to improve performance and the user management tool for user friendliness. • We have three stages for this research. The first stage in research is data collection. In this stage, we study characteristics of large-scale web access log data and investigate commercial DBMS tuning techniques. The next stage, we design the system model. The proposed system has three components including pre-processor, DBMS and management tool. To improve the effectiveness the system, Pre-processor hash and sort log data in memory. And we perform DBMS tuning to improve performance. The final stage, we implement system and evaluate the performance. And we develop system management tool for user friendliness. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 3
- 4. Introduction • Log generally collects some data for analysis, such as, click rates, type of users, how many users access each web page, and time-of-use . • Commercial DBMS with high performance and high scalability is used by many companies in the industry, such as Oracle, Microsoft, IBM, and etc. • In order to optimize the performance of DBMS with high performance, it needs to widely and deeply understand OS, hardware system, DMBS, and application etc. because the inappropriate optimization cause the performance degradation. • The main point of this study is about how to optimize DBMS to efficiently store and to manage web access log data. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 4
- 5. Web Access Log Data The system using only DB JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) The proposal system using Pre-Processor 5
- 6. A. Hash • Hash is a value that is stored on a specific address using hash function. After a value is processed by hash function, it is mapped to a hash table. A hash table is composed of a bucket that store data and a hash address that corresponds to a key value. When a hash address is formed by hash function, each different key might have the same hash address, which is called Hash Collision. • ADVANTAGE : it’s possible to get specific data with only one access using a key . B. Bucket • A value is mapped to a hash table. A hash table is composed of n buckets. Each bucket has m slots and each slot can store only one value. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 6
- 7. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 7
- 8. Optimization Of Database Design The System Structure JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 8
- 9. B. Pre-Processor: • Pre-Processor remove redundant data by reading the collected web log data and is the role of preprocessing for storing data in Database. Pre-Processor transfers collected data through log collection server and read data is stored in a hash table using a key. At the moment, if redundant data about URL is in a hash table, then increase a count value and make statistical information about collected URL. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 9
- 10. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 10
- 11. C. System Analysis and Design: • With key/value pair, data is divided into a key and a value. Time and source IP are stored into a key, on the other hand, other information and a count value are stored into a value. Basically, we compare a key using key/value pair and increase a count value if the key exits in a hash table. Because the hash table in this study uses 32-bit hash value, collision probability is 1/4.284.967.296. D. Transaction Processing: • Statistical information made by Pre-Processor has mostly a lot of URL that access frequency is 1. In case of URL that access frequency is high is because it brings information related to a web page without a direct request, such as, images, advertisement, etc. • Transaction processing is a way to sort out some irrelevant data as one transaction is considered all data related to a request page. Also, it processes log in n seconds, which is considered a unit of one transaction. After processing, the processed log is split among two results, such as, a result list and a filter list. The result list is a list without filtering, that is, an aggregate of unfiltered URL in the whole log. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 11
- 12. Access Log JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 12
- 13. Filter list & Result List JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) Transaction Filter 13
- 14. Performance Evaluation A. Experiment Environment B. Experiment Data 1. Zipf Distribution: • Zipf’s Law is that the frequency of a specific item is proportional to a priority item as creating the optimized data for an experiment. That is, the frequency of n priority item is that a proportion of some element is proportional to 1/priority with the law that the frequency is 1/n. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 14
- 15. C. Worst Case: • Worst case (WC) is that collected web access log is not duplicated and there is no constant distribution for web access. We expect the worst performance because all collected data are stored in DB and data de-duplication is not carried out. D. Log File: • A sample data for creating experiment data uses URL Snooper and is composed of collected time, source IP, destination IP, and domain, URL. We set up to collect over a million data in a server. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 15
- 16. EXPERIMENT RESULT A. • Data Input Time Zipf parameter 0.2, which includes the most redundant data, takes 437.5520 seconds to store 376,934 logs to DB is and the worst case, includes no redundant data, takes 1044.4392 seconds. Input Time JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 16
- 17. B. Writing Data to a File File Write Time JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) C. Creation of Statistical Information: Data input time to Hash Table 17
- 18. CONCLUSION: • We understand the architecture and the feature of DBMS through how to store, index structure, and data storage structure of commercial DMBS. And, using the feature of DBMS, we make the optimized DBMS and the tool for handling web access log data and providing fundamental knowledge to develop software. • The input time from Pre-Processor to DB depends on how often de-duplication occurs in a hash table. The memory usage is Pre-Processor uses the memory the twice time more than DB when creating statistical information. However, run time is Pre-Processor, which operate in-memory, is from 18 to 20 times better than DB • In summary, using Pre-Processor, works in-memory, than only using DB is expected to better performance, run-time, and the result data than the traditional system. Furthermore, this study is expected to be used from other area like network security. JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 18
- 19. References • Database Optimization for Large Scale Web Access Log Management - Minjae Lee, Jaehun Lee, Dohyung Kim, Taeil Kim, Sooyong Kang - Hanyang University {Dept. of Electronics & Computer Engineering} JAY J. PATEL (UDIT, UNIVERSITY OF MUMBAI) 19