IEJoin and Big Data Cleansing
0 likes535 views
This document summarizes Zuhair Khayyat's work on fast and scalable inequality joins for data cleansing. It describes an approach called IEJoin that sorts data by key columns and uses a bit array to efficiently identify violations of inequality rules, such as a rule that people with higher salaries must pay higher taxes. The approach runs in O(n log n) time and scales well in distributed systems. Experimental results show IEJoin outperforms database systems on inequality joins and can process billions of rows efficiently on a cluster. The work was presented at VLDB 2015 and will be presented at VLDB 2016.
IEJoin and Big Data Cleansing
- 1. Fast and Scalable Inequality Joins -- for Data Cleansing on Scale -- Zuhair Khayyat PhD Candidate @ InfoCloud group King Abdullah University of Science and Technology (KAUST)
- 2. ● Two customers having the same zip cannot be in different cities Data Cleansing Name Zip City Winnie 91340 San Francisco Robbert 91340 New York Emma 91340 San Francisco
- 3. ● Two customers having the same zip cannot be in different cities ● “inaccurate data has a direct impact ... the average company losing 12% of its revenue” -- Ben Davis (Econsultancy) Data Cleansing Name Zip City Winnie 91340 San Francisco Robbert 91340 New York Emma 91340 San Francisco
- 4. ● Two customers having the same zip cannot be in different cities ● “inaccurate data has a direct impact ... the average company losing 12% of its revenue” -- Ben Davis (Econsultancy) ● “This is the digital universe. It is growing 40% a year into the next decade” -- EMC2 Big Data Cleansing Name Zip City Winnie 91340 San Francisco Robbert 91340 New York Emma 91340 San Francisco
- 5. Data Cleansing System Data (Dirty) Quality Rules Violations
- 6. Data Cleansing System Data (Dirty) Quality Rules Violations Repair AlgorithmsFixes
- 7. Data Cleansing System Data (Partially Clean) Quality Rules Violations Repair AlgorithmsFixes
- 8. Big Data Cleansing System Data (Partially Clean) Quality Rules Violations Repair AlgorithmsFixes Big
- 9. BigDansing: A System for Big Data Cleansing In SIGMOD 2015 BigDansing Quality rules Repair Algorithms Dirty datasets Clean datasets
- 10. BigDansing: A System for Big Data Cleansing In SIGMOD 2015 BigDansing Quality rules Repair Algorithms Dirty datasets Clean datasets
- 11. BigDansing: A System for Big Data Cleansing Functional dependencies Inclusion dependencies Denial constraints Entity resolution Domain Specific Language Optimized execution plan In SIGMOD 2015
- 12. BigDansing: A System for Big Data Cleansing In SIGMOD 2015 Violations S1 S2 S3 Repair Algorithm Repair Algorithm Repair Algorithm
- 13. In SIGMOD 2015 10M 20M 40M 0 100 200 300 400 500 600 700 BigDansing on Spark Spark SQL Dataset Size Time(Sec) Two customers having the same zip cannot be in different cities (FD: Zip → City) BigDansing: A System for Big Data Cleansing
- 14. Complex Quality rule: Inequality joins ● If a person has a higher salary, he must pay more taxes compared to others
- 15. Complex Quality rule: Inequality joins ● If a person has a higher salary, he must pay more taxes compared to others ● Select * from D t1 JOIN D t2 on t1.Salary > t2.Salary AND t1.Tax < t2.Tax; ● Processed as a Cartesian product: O(n2 )
- 16. Lightning Fast and Space Efficient Inequality Joins ● Sort on Salary: ● Sort on Tax: ● Bit-array: In VLDB 2015 Tuple Salary Tax t1 100 5 t2 90 9 t3 150 15 t4 120 10 t3(150) t4(120) t1(100) t2(90) t3(15) t4(10) t2(9) t2(5) 0 1 2 3 0 1 3 2 0 0 0 0 Permutation array:
- 17. Lightning Fast and Space Efficient Inequality Joins ● Sort on Salary: ● Sort on Tax: ● Bit-array: In VLDB 2015 Tuple Salary Tax t1 100 5 t2 90 9 t3 150 15 t4 120 10 t3(150) t4(120) t1(100) t2(90) t3(15) t4(10) t2(9) t2(5) 0 1 2 3 0 1 3 2 0 0 0 0 Permutation array:
- 18. Lightning Fast and Space Efficient Inequality Joins ● Sort on Salary: ● Sort on Tax: ● Bit-array: In VLDB 2015 Tuple Salary Tax t1 100 5 t2 90 9 t3 150 15 t4 120 10 t3(150) t4(120) t1(100) t2(90) t3(15) t4(10) t2(9) t1(5) 0 1 2 3 0 1 3 2 0 0 0 0 Permutation array: O(n log n)
- 19. IEJoin vs. DBMS (Single machine) 10K 50K 100K 0.01 0.1 1 10 100 1000 10000 PG-IEJoin Postgres MonetDB DBMS-X Dataset size Runtime(Sec)
- 20. IEJoin vs. Spark SQL (Distributed) Salary-Tax Range Intersection 0 4 8 12 16 20 24 IEJoin on Spark SQL Spark SQL Runtime(Hours) 100M rows on 6 machines
- 21. IEJoin on 8B Rows ● A cluster of 16 workers ● 8B rows, 287 GB on Disk ● Runtime in 13 hours ● Close to the ideal speedup 0 2 4 6 8 10 12 14 16 18 0 20 40 60 80 100 IEJoin on Spark SQL Ideal Speedup Cluster Size Hours
- 22. Visit us! ● Zuhair Khayyat – cloud.kaust.edu.sa ● SIGMOD 15 – BigDansing paper ● VLDB 15 – IEJoin Paper -----> to be presented in VLDB 16 ● SIGMOD 16 – Demo Paper