Checkpoints for data_quality
Download as PPTX, PDF0 likes56 views
The document discusses checking various aspects of data quality, including file type, text content type, table data structure, tidiness, and wholeness. It recommends using common file types like text that can be read by various applications, and suggests JSON as an all-round format for text content. Table data should have each column as a variable and each row as an observation to be considered tidy. Data quality checks aim to gather related data while maintaining tidiness.
Checkpoints for data_quality
- 1. Check points for data quality Soonmok Kwon Naver Last update 2019.2.22
- 2. Data pipeline and data quality Sometimes, it is as serious as code quality. app data
- 3. Introduction • Let’s check following topics for data quality: • File type • Text content type • Table data • Tidiness • Wholeness
- 4. File types • text, parquet, hadoop sequence-file, … • + various compression codecs • Characteristics of those types? • space efficiency • processing speed (including compression/decompression times) • human visibility (simplicity) • language dependency • Use general one. • Hadoop sequence file can be read by Python using some libraries. But it would be much happier if the original file were TEXT. • The content should decide the file type not the application logic. • Because, some day, applications with different logics can access the data.
- 5. Text content types • TSV (CSV), JSON, XML, … • Characteristics of those types? • processing speed • space efficiency • human visibility(simplicity) • self-explanation power • expandability • JSON is all-round. Good for most cases. • Don’t mix (JSON in XML, JSON in TSV, …) ! • You will have hard time generalizing your code for processing those data. • You will meet annoying character escaping issues. • You will lost support of developer community such as validation code. • In general, when you mix, the sub-content with high-level format loses its functionality. • Put them in TEXT files. • Because they are text.
- 6. Table data • A table data is dataset that can be converted to 2-dimention table with mostly PRIMITIVE-type values in each cell. • This is what data scientists call DATASET. • Some data are TSV (CSV), JSON, XML, … but not table. • E.g. arbitrary number of items in TSV • Not-flat JSON • For data science, terms like TSV, CSV, JSON is used in following meaning: • TSV: table data in TSV • CSV: table data in CSV • JSON: table data in JSON • When Spark can’t read a CSV, it is not a table data in CSV.
- 7. Table data – Tidy data • Definition • A table data is TIDY when • Each column is VARIABLE, and • Each row is OBSERVATION. • In the definition, Variable and observation are statistics terminology. • Observation: Aspects values measured in a single experiments • Variable: Aspects to be measured in experiments. Features. • Some properties • No variable needs to be separated • Each variable has ONE type • The process to build tidy data has been systemized in large part, but many parts are remaining as art.
- 8. Table data – Tidy data Example data engineering process to create tidy data 1 2 3
- 9. Table data – Tidy data 1 2 3 Example data engineering process to create tidy data
- 10. Table data – Whole data • Data with the same topic scattered all around? • So bad. Lets gather them up. • But this move can result in un-tidy data. • When you pursue wholeness, consider tidiness. • Or, offer both parts and whole.
- 11. Conclusion • Mind following topics for data quality: • File type • Text content type • Table data • Tidiness • Wholeness