2013 ucar best practices
- 1. Best practices for scientific computing C. Titus Brown ctb@msu.edu Asst Professor, Michigan State University (Microbiology, Computer Science, and BEACON)
- 2. Best practices for scientific computing C. Titus Brown ctb@msu.edu Asst Professor, Michigan State University (Microbiology, Computer Science, and BEACON)
- 3. Towards better practices for scientific computing C. Titus Brown ctb@msu.edu Asst Professor, Michigan State University (Microbiology, Computer Science, and BEACON)
- 4. Who are we? Greg Wilson, D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H. D. Haddock, Katy Huff, Ian M. Mitchell, Mark Plumbley, Ben Waugh, Ethan P. White, Paul Wilson Authors of “Best Practices for Scientific Computing” http://arxiv.org/abs/1210.0530
- 5. Who am I? • “Computational scientist” • Worked in: – Evolutionary modeling – Albedo measurements (Earthshine) – Developmental biology & genomics – Bioinformatics • “Data driven biologist” – Data of Unusual Size + bio
- 6. Who am I? (Alternative version) • Open source / free software • Member of the Python Software Foundation • Developed a few different pieces of non-scientific software, mostly in testing world. => Open science, reproducibility, better practices.
- 7. What is this talk about? • Most scientists engage with computation in their science… • …but most are never exposed to good software engineering practices. • This is not surprising. – Computer science generally does not teach “practice” – Learning your scientific domain is hard enough.
- 8. A non-dogmatic perspective • There are few practices that you really need to use. – Version control. – Testing of some sort – Automation of some sort (builds, deployment, pipelines) • There are lots of practices that will consume your time and eat your science. – …but figuring out which practices are useful is often somewhat domain and project and person specific. • There are no silver bullets. (Sorry!)
- 9. What do scientists care about? 1. Correctness 2. Reproducibility and provenance 3. Efficiency
- 10. What do scientists actually care about? 1. Efficiency 2. Correctness 3. Reproducibility and provenance
- 11. Our concern • As we become more reliant on computational inference, does more of our science become wrong? • “Big Data” increasingly requires sophisticated computational pipelines… • We know that simple computational errors have gone undetected for many years – a sign error => retraction of 3 Science, 1 Nature, 1 PNAS – Rejection of grants, publications! http://boscoh.com/protein/a-sign-a-flipped-structure-and-a- scientific-flameout-of-epic-proportions
- 12. Our central thesis With only a little bit of training and effort, • Computational scientists can become more efficient and effective at getting their work done, • while considerably improving correctness and reproducibility of their code.
- 13. The paper • Code for people • Plan for mistakes • Automate repetitive tasks • Avoid premature • Record history optimization • Make incremental changes • Document design & • Use version control purpose of code, not details • Don’t repeat yourself • Collaborate
- 14. The subset of these I’ll discuss 1. Use version control 2. Plan for mistakes 3. Automate repetitive tasks 4. Document design & purpose of code
- 15. Use version control! 1. Any kind of version control is better than none. 2. Distributed version control (Git, Mercurial) is very different from centralized VCS (CVS, Subversion). 3. Sites like github and bitbucket are changing software development in really interesting ways. (see: www.wired.com/opinion/2013/03/github/, “The github revolution”)
- 16. Use version control • Version control enables efficient single-user work by “gating” changes into discrete chunks. • Version control is essential to multiperson collaboration on software. • Distributed version control enables remixing and reuse without permission, while retaining provenance.
- 17. Original github repo Change A "Fork" Change B Change 1 "Fork" Change C Change 2 Change AA Change D Change 3 Change BB Change E Change 4
- 18. Plan for mistakes! 1. Program defensively -- Use assertions to enforce conditions upon execution def calc_gc_content(dna): assert ‘N’ not in dna, “DNA is only A/C/G/T”
- 19. Plan for mistakes! 2. Write/run tests – def test_calc_gc_1(): gc = calc_gc(“AT”) assert gc == 0 def test_calc_gc_2(): gc = calc_gc(“”) asssert gc == 0
- 20. Plan for mistakes! 3. Black box regression tests: For fixed input, do we get the same (recorded) output as last day/week/month? (Very powerful when combined with version control.)
- 21. Plan for mistakes! Write/run tests – A few personal maxims: - simple tests are already very useful (if they don’t work…) - past mistakes are a guide to future mistakes - any tests are better than no tests - if they’re not easy to run, no one will run them
- 22. Automate repetitive tasks! Automate your builds, your test running, your analysis pipeline, and your graph production. 1. Augments reusability/reproducibility. 2. Encodes expert knowledge into scripts. 3. Decreases arguments about culpability :) 4. Excellent training mechanism for new students/collaborators! 5. Combined with version control => provenance of analysis results! 6. Improves ability to revise, reuse, remix.
- 23. IPython Notebook
- 24. Cloud computing/VMs • One approach my lab has been using is to make publication’s data, code, and instructions available for Amazon EC2 instances: ged.msu.edu/papers/2012-diginorm/ • Reviewers have been known to actually go rerun our pipeline… • More to the point, this enables others (including collaborators) to revise, reuse, remix.
- 25. Document design & purpose x = x + 1 # add 1 to x • vs # increase past possible fencepost boundary error range_end = range_end + 1
- 26. Document design & purpose More generally, - describe APIs - provide tutorials on use - discuss the design for domain experts & programmers, not for novices.
- 27. Anecdotes I need to remember to tell 1) A sizeable fraction of my “single-use” scripts were wrong, upon reuse. 2) New students in my lab run through at least one old paper’s execution pipeline before starting their work. 3) Students may develop for long time on own branch, while continually merging from main.
- 28. DVCS particularly facilitates long term branching. Original github repo Change A Change B "Fork" Change C Change AA Change D Change BB Change E Change CC
- 29. There are many, many practices I did not discuss. Testing: • TDD vs BDD vs SDD? • Functional tests vs unit testing vs … • Code coverage analysis. • Continuous integration! My view: be generally aware of what’s out there & focus on what addresses your pain points.
- 30. Software Carpentry http://software-carpentry.org • Invite us to run a workshop! • 2 days of training at appropriate/desired level: – Beginning/intro – Intermediate – Advanced (?) • Funded by Sloan and operated by Mozilla
- 31. Contact info Titus Brown, ctb@msu.edu http://ivory.idyll.org/blog/ @ctitusbrown on Twitter This talk will be on slideshare shortly; google “titus brown slideshare” Best Practices for Scientific Computing http://arxiv.org/abs/1210.0530 Git can facilitate greater reproducibility… (K. Ram) http://www.scfbm.org/content/8/1/7/abstract
Editor's Notes
- #24: Increasingly being connected via SOA to backend db