OSBot - Data transformation workflow (from GSheet to Jupyter)
1 like753 views
The document outlines a data transformation workflow from Google Sheets to Jupyter Notebook, detailing steps such as loading data, parsing the dataset, and merging data. It includes examples of merging datasets and highlights issues related to data integrity, particularly concerning user identification and email matches. The document emphasizes the importance of data cleaning before merging to ensure accurate results.
OSBot - Data transformation workflow (from GSheet to Jupyter)
- 1. Data Transformation Workflow (from gsheet to jupyter) July 2019, @DinisCruz OSBOT (OWASP Security Bot)
- 3. Jupyter Lab Dev Environment
- 4. Step 1 Loading Data from GSheet
- 6. Viewing Data in Slack
- 7. Viewing data in Jupyter Pandas DataFrame Jupyter QGrid
- 8. All code and data is auto-saved in Git
- 10. Explaining what we are doing in an Jupyter Notebook
- 12. Starting refactoring code into methods
- 13. Get 3 datasets as separate DataFrames
- 14. Helper function to transform raw data into objects
- 15. Refactored code that extracts data into 3 DataFrames
- 16. 3 DataFrames with data
- 18. Example 1 - Merge 3 datasets All fields from all 3 data sets (in this case there was only one exact match on email)
- 19. Example 2 - Merging data loses user with different email When Merging DF_1 with DF_3 we lose `Alan Lee` due to different email
- 20. Example 3 - Fixing data before merge (version 1)
- 21. Lost user due to bad data in Name Lost `Bruno Lyon` User Because `Name` value is wrong
- 22. Example 4 - Fixing data before merge (version 2) The merge of the two DataFrames now successfully finds all 4 users (including Alan who has two emails) By using email to find the first and last names values