Alex Liu Harvard Forest Presentation
- 1. Improving RDataTracker Accessibility and Functionality: R Markdown and Caching Siqing (Alex) Liu, Amherst College Mentors: Barbara Lerner, Emery Boose Harvard Forest, Summer 2016
- 2. Background: Provenance • A record of ownership of a work of art or an antique, used as a guide to authenticity or quality1 1Oxford English Dictionary The Beautiful Princess, allegedly a rediscovery from Leonardo da Vinci
- 3. Background: Data Provenance • Why is it Important for Science? • Validity • Reproducibility • Reproducibility Crisis
- 4. Data Provenance in R: RDataTracker Data R Script Data Derivation Graph (DDG)
- 5. Problem: Organizing Data Derivation Graphs (DDGs)
- 6. Solution: R Markdown • What is R Markdown? • File Format that allows simple creation of formatted and interactive output from R
- 8. Organizing Scripts with R Markdown
- 11. Problem: Repeated Execution • Intensive Calculations • Complex • Over large data sets • Takes a significant amount of time each run • Minor changes take a lot of time
- 12. ModifiedOriginal
- 13. Solution: Caching • What is Caching? • Storing intermediate values so they don’t need to be reprocessed • Problem: When to Re-execute? • Dependencies
- 14. Caching with Knitr: Chunk-by-Chunk Dependencies
- 17. Conclusion: Accessibility and Functionality • R Markdown • Accessibility: Directly read in R Markdown files • Functionality: Automatically organize nodes in DDG • Caching • Accessibility: Make repeated execution of compute-intensive scripts faster • Functionality: Accurate, command-by-command caching and tracing
- 18. Further Work • More complete caching • Compression vs. Speed • Side Effects • Creating a plot • Writing file • Printing results to console
- 19. Acknowledgements Emery Boose Harvard Forest Harvard University Barbara Lerner Dept. of Computer Science Mount Holyoke College