Week 1: Programming for Data Analysis
0 likes261 views
This document outlines the topics to be covered in a programming for data analysis course at the Thai-Nichi Institute of Technology in Bangkok. The course will introduce Pandas, a data analysis library in Python. It will cover reading data from Excel/CSV files, calculating statistics, cleaning data, and visualizing data with Matplotlib. Specific topics will include reading and writing data, working with indexes, querying data, handling missing values, and correlating variables. Students will learn to import data, explore datasets, clean and transform data, and store results back into files.
![JSONfromfile
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
51
• Create a file person.json and store the following
{"name":"Bob", "languages":
["English", "French"]
}](https://image.slidesharecdn.com/week1-201207030440/85/Week-1-Programming-for-Data-Analysis-51-320.jpg)
Week 1: Programming for Data Analysis
- 1. Programming for Data Analysis Week 1 Dr. Ferdin Joe John Joseph Faculty of Information Technology Thai – Nichi Institute of Technology, Bangkok
- 2. Topics to cover Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 2
- 3. Today’s Lesson • Introduction to Pandas • Read Data from Excel/CSV • Store Data in Excel/CSV • Laboratory Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 3
- 4. Pandas • Pandas is the backbone of most of the data science projects • Derived from the term "panel data", an econometrics term for data sets that include observations over multiple time periods for the same individuals. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 4
- 5. Pandas • Explores datasets and read CSV and Excel sheets • Calculate Statistics • Clean the data by doing things like removing missing values and filtering rows or columns by some criteria • Visualize the data with help from Matplotlib. Plot bars, lines, histograms, bubbles, and more. • Store the cleaned, transformed data back into a CSV, other file or database Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 5
- 6. Pandas - Structure • Pandas is built on top of the NumPy package • Data in pandas is often used to feed statistical analysis in SciPy • Plotting functions from Matplotlib • Machine learning algorithms in Scikit-learn Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 6
- 7. Pandas - Installation In Command prompt pip install pandas or conda install pandas In jupyter notebook cell !pip install pandas Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 7
- 8. Pandas installation Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 8
- 9. Pandas version Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 9
- 10. Pandas version Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 10
- 11. Pandas read data Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 11
- 12. Pandas - Index Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 12
- 13. Pandas – Locate data query Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 13
- 14. Pandas read CSV Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 14
- 15. Pandas – Read CSV Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 15
- 16. Pandas – Read CSV from local drive Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 16
- 17. Read CSV and remove index Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 17
- 18. Pandas – Read JSON Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 18
- 19. Pandas head() Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 19
- 20. Pandas tail() • List last few data Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 20
- 21. Data frame variables Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 21
- 22. Size of dataset and data duplication Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 22
- 23. Drop duplicates Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 23
- 24. Rename Columns Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 24
- 25. Change case of all variables Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 25
- 26. Pandas – Is null Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 26
- 27. Pandas – Is null Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 27
- 28. Remove null values Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 28
- 29. Remove null by axis Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 29
- 30. Select a column based on index Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 30
- 31. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 31
- 32. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 32
- 33. Fill null with mean of duration Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 33
- 34. Describe function Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 34
- 35. Describe a column and statistics Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 35
- 36. Correlation of all variables Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 36
- 37. iloc function Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 37
- 38. Data between two values Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 38
- 39. Query in Pandas Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 39
- 40. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 40
- 41. Multiple Conditions Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 41
- 42. Excel Read - Pandas Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 42
- 43. Formsof JSON Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 43 • Objects • Arrays • Values • Strings • Whitespace
- 44. JSONProtocol Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 44 • ECMA-404 The JSON Data Interchange Standard. • Link of the protocol in canvas. For your reading
- 45. Usageof JSONinpython Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 45 • Demonstration in Jupyter
- 46. Prerequisite Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 46 • Prepare a JSON file using notepad and save it as filename.json • For easy access, save the file in the same directory as the ipynb notebook
- 47. Prerequisite Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 47
- 48. Libraries Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 48
- 49. Libraries Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 49
- 50. JSONinvariable Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 50
- 51. JSONfromfile Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 51 • Create a file person.json and store the following {"name":"Bob", "languages": ["English", "French"] }
- 52. JSONfileread Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 52
- 53. Dict to JSON Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 53
- 54. Readandwrite JSON Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 54
- 55. Addmorefields Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 55
- 56. Appendmoredata Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 56
- 57. DSA 207 • Import libraries • Get familiar with read csv • Read xlsx • Read json • Use numpy to access the values • Use pandas to remove null values Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 57