Dynamic weather data scraping
- 1. Dynamic Weather Data Scraping Solution Brief Requirement of a system through which we can fetch the historic temperature information on Day level with minimum or no manual efforts. Weather data should be contextual taking into consideration inputs like Date, City and State. The data would be accurate and will also give Day Minimum and Day Maximum Temperature which will be the average of Hourly Minimum and Hourly Maximum, also it handles if there is no weather data available for certain Hours. Solution Approach Our solution approach comprises several distinct components: 1. Finding the State ids from master data with state name provided in the data 2. Searching for all the Station ids and matching it with the right City Name 3. Fetching the Hourly temperature and averaging it to get daily minimum and maximum 4. Dealing with few NA issues 5. Having a master sheet for the State ids and Station ids for future scalability Future Goals: 1. API it to our Dashboard 2. Expanding the master data file to incorporate other countries (Currently it has all the major cities of United States) – this will also require UOM handling 3. Should fetch if the State Abbreviations are provided rather just full name 4. Much more validations need to be in place 5. Testing with false or abnormal inputs 6. Store master file on cloud so that it becomes platform independent 7. Search by only city if there is no state name mentioned (Issue with this is some city names are in multiple states) 8. Code Optimization as the current system will slower the dashboard 9. Including Wind Speed and Humidity data along with temperature
- 2. Sample Master Data: Sample Input Data:
- 3. Sample Output: Snapshot of R Environment: