Data preprocessing
Download as PPTX, PDF4 likes1,706 views
Data preprocessing is a crucial and challenging process that enhances data quality to make it suitable for analysis. It involves tasks such as data cleaning, normalization, reduction, and integration, addressing issues like noise, incompleteness, and inconsistency. The document emphasizes the importance of documentation and collaboration in preprocessing efforts, alongside utilizing various resources for best practices.
Data preprocessing
- 1. Data Preprocessing Likit Preeyanon, Ph.D. Center of Data Mining and Biomedical Informatics Faculty of Medical Technology Mahidol University
- 2. Real World Data • Noisy, incomplete, inconsistent • Unstructered (Web pages, Tweets, Emails etc.) • Big
- 3. “Garbage in, garbage out” Your analysis is as good as your data.
- 4. Definition • Data pre-processing is a process that improves quality of data and makes data more suitable for the analysis • Data pre-processing is the quality control
- 5. Data Analysis Workflow • Data Preprocessing 80% • Data Analysis 5% • Report/Presentation 5%
- 6. What To Do • Make data suitable (reformat) the analysis • Remove irrelevant data, discrepancies, errors or outliers (data cleaning) • Remove, replace or impute missing data • Reduce data size (data reduction) • Normalize, transform, convert or discretize data • Integrate data • Anonymize data
- 7. Data Preprocessing is Hard • Data are different (text data, sequencing data, image data, signal data, geographical data, time series data and etc.) • Data are massive • Data are from different sources • Some data are of low quality • Some data are not well documented (no metadata) • Different tools require data in different formats • Some decisions have to be made
- 8. Examples stuid score1 score2 age gender 1 3 76 16 M 2 33 56 15 F 3 45 88 75 M 4 90 32 - - 5 45 37 17 M 5 45 37 17 M 6 55 - 17 female 7 45 99 16 male
- 9. Examples stuid math biology age gender 2 33 56 15 F 3 45 88 16 M 4 90 32 16 - 5 45 37 17 M 6 55 - 17 F 7 45 99 16 M
- 10. Examples stuid math biology age gender 2 F P 15 F 3 F P 16 M 4 P F 16 - 5 F F 17 M 6 P F 17 F 7 F P 16 M
- 11. Examples stuid math biology age gender 2 F P 15 F 3 F P 16 M 4 P F 16 - 5 F F 17 M 6 P F 17 F 7 F P 16 M stuid city state country 2 London Kentucky US 3 Seattle WA US 4 L.A. California US 5 Denver Colorado US 6 Beijing - China 7 - - India stuid income 2 100000 3 13400 4 20000 5 430000 6 23000 7 5400
- 12. Take home message • Data preprocessing is hard and not taught in class • Use Google and web forum to learn more about available tools, guidelines and tricks to preprocess data • Trial and error is necessary (it’s research!) • You should consult collaborators or experts, if needed • Data preprocessing must also be documented! Both raw and preprocessed data must be available. • Well-designed data collection method is helpful
- 13. Resources • stackoverflow.com • rseek.org • google.com • Lots of R books