![Case Studies
Variable Stars in Stripe 82
SDSS
Y N
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-55-320.jpg)
![Case Studies
Variable Stars in Stripe 82
SDSS
Y N
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-56-320.jpg)
![Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1 i=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-57-320.jpg)
![Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1 i=1
Runs Runs
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-58-320.jpg)
![Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1 i=1
Runs Runs
⇤ ⇤
pgood ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
var
“Constant & Good” “Variable & Good”
⇤
pbad ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
bad
“Bad”
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-59-320.jpg)
![Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1
Npars = Nstars + Nruns + 6
i=1
Runs ~ 0 , ~ ⇤ , , ⌘, ⌃2 , PRuns , Pbad }
2
⇥ = {f f var var , ⌃bad
⇤ ⇤
pgood ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
var
“Constant & Good” “Variable & Good”
⇤
pbad ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
bad
“Bad”
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-60-320.jpg)
![Case Studies
Variable Stars in Stripe 82
SDSS
N
Y
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-63-320.jpg)
![Case Studies
Variable Stars in Stripe 82
SDSS
N
X
f ⇤ (t) = A0 + [An sin(!t) + Bn cos(!t)]
n=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com](https://image.slidesharecdn.com/pygotham-110919092240-phpapp02/85/Python-and-MongoDB-in-Astronomy-64-320.jpg)
Python and MongoDB in Astronomy
- 1. Python and MongoDB in Astronomy Dan Foreman-Mackey Center for Cosmology and Particle Physics Department of Physics @ NYU In collaboration with: David W. Hogg (NYU), Larry Widrow (Queen’s), Dustin Lang (Princeton), Jonathan Sick (Queen’s), Micha Gorelick (NYU) and many others...
- 2. Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 3. Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Andromeda Case Studies The Internet The Milky Way Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 4. The Universe Galaxies Stars Planets Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 5. What is the The Universe Universe Made of? Galaxies Stars Planets Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 6. What is the The Universe Universe Made of? Galaxies Stars Are there other Earth- Planets like planets? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 7. What is the The Universe Universe Made of? Galaxies Stars Are there other Earth- Planets like planets? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 8. What is the Universe Made of? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 9. What is the Universe Made of? Source: Wikipedia (Adam Evans) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 10. What is the Universe Made of? Rotational Speed Radius Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 11. What is the Universe Made of? Observed Rotational Speed Radius Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 12. What is the Universe Made of? Observed Rotational Speed WTF? Radius Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 13. What is the Universe Made of? Rotational Speed ? Observed WTF? Radius Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 14. What is the Universe Made of? Rotational Speed ? Observed WTF? Radius ? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 15. What is the Universe Made of? Observed Rotational Speed WTF? Radius ? ? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 16. What is the Universe Made of? Dark Matter Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 17. What is the Universe Made of? Size of the Universe PyGotham b le rva obse Time Source: NASA / WMAP Science Team Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 18. What is the Universe Made of? Dark Energy 73% Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 19. What is the Universe Made of? Dark Energy 73% Source: DFM & Widrow (in prep) Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 20. What is the Universe Made of? Source: http://apod.nasa.gov Dark Energy 73% Source: DFM & Widrow (in prep) Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 21. What is the Universe Made of? Source: http://apod.nasa.gov Dark Energy 73% Source: DFM & Widrow (in prep) Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 22. Dan Foreman-Mackey CCPP@NYU Credit: The Millennium Simulation Project dfm.github.com
- 23. Dan Foreman-Mackey CCPP@NYU Credit: The Millennium Simulation Project dfm.github.com
- 24. Dan Foreman-Mackey CCPP@NYU Credit: The Millennium Simulation Project dfm.github.com
- 25. Data in Astronomy Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 26. Data in Astronomy Credit: Jonathan Sick jonathansick.ca Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 27. Data in Astronomy Credit: Jonathan Sick jonathansick.ca Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 28. Data in Astronomy MegaCam: 340 MegaPixels Credit: Jonathan Sick jonathansick.ca Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 29. Data in Astronomy MegaCam: 340 MegaPixels Credit: NASA Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 30. Data in Astronomy Imaging Source: NASA / ESA Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 31. Data in Astronomy Imaging Spectroscopy Source: NASA / ESA Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 32. Data in Astronomy Source: Riaud & Schneider (2007) Imaging Spectroscopy Spectroscopy Source: NASA / ESA Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 33. Data in Astronomy Source: Riaud & Schneider (2007) Imaging Spectroscopy Spectroscopy Source: NASA / ESA Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 34. a Data in Astronomy lot is Open of Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 35. a Data in Astronomy lot is Open of ! and there’s a lot of it Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 36. Data in Astronomy is Open 1990– 2000– 1997–2001 SDSS Hubble archive.stsci.edu/hst sdss.org 2MASS www.ipac.caltech.edu/2mass GAIA Pan-STARRS LSST Planned Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 37. Data in Astronomy is Open 1990– 2000– 1997–2001 SDSS Hubble archive.stsci.edu/hst sdss.org 2MASS www.ipac.caltech.edu/2mass GAIA Pan-STARRS LSST Planned Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 38. Data in Astronomy is Open 1990– 2000– 1997–2001 SDSS Hubble archive.stsci.edu/hst sdss.org 2MASS www.ipac.caltech.edu/2mass GAIA Pan-STARRS LSST Planned Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 39. Where does Python fit in? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 40. Where does Python fit in? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 41. Where does Python fit in? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 42. Where does Python fit in? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 43. Where does Python fit in? Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 44. Where does Python fit in? + Scientific Python Stack Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 45. Where does Python fit in? + Scientific Python Stack Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 46. Where does Python fit in? + Scientific Python Stack Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 47. What about ? Easy Flexible Pythonic Scalable Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 48. Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 49. Case Studies Variable Stars in Stripe 82 SDSS Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 50. Case Studies Variable Stars in Stripe 82 SDSS 728 SESAR ET AL. Ses S07 Labela Ntot A 84 B 144 C 54 D 8 E 11 F 11 Source: Sesar et al. (2010) G 10 H 7 I 4 J 26 K 8 Dan Foreman-Mackey CCPP@NYU L 3 dfm.github.com M 5
- 51. Case Studies Variable Stars in Stripe 82 SDSS Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 52. Case Studies Variable Stars in Stripe 82 SDSS Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 53. Case Studies Variable Stars in Stripe 82 SDSS 800k “Fields” ~ 12TB Imaging data > 1M “Target Stars” Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 54. Case Studies Variable Stars in Stripe 82 SDSS Photons/Brightness Time Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 55. Case Studies Variable Stars in Stripe 82 SDSS Y N p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 56. Case Studies Variable Stars in Stripe 82 SDSS Y N p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 57. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Y pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 i=1 Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 58. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Y pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 i=1 Runs Runs Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 59. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Y pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 i=1 Runs Runs ⇤ ⇤ pgood ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) var “Constant & Good” “Variable & Good” ⇤ pbad ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) bad “Bad” Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 60. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Y pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 Npars = Nstars + Nruns + 6 i=1 Runs ~ 0 , ~ ⇤ , , ⌘, ⌃2 , PRuns , Pbad } 2 ⇥ = {f f var var , ⌃bad ⇤ ⇤ pgood ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) var “Constant & Good” “Variable & Good” ⇤ pbad ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) bad “Bad” Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 61. Case Studies Variable Stars in Stripe 82 SDSS Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 62. Case Studies Variable Stars in Stripe 82 SDSS Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 63. Case Studies Variable Stars in Stripe 82 SDSS N Y p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 64. Case Studies Variable Stars in Stripe 82 SDSS N X f ⇤ (t) = A0 + [An sin(!t) + Bn cos(!t)] n=1 Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 65. Case Studies CFHT 4.7 Gigapixel mosaic of M31 Source: Jonathan Sick (Queen’s University) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 66. Dan Foreman-Mackey CCPP@NYU dfm.github.com Jonathan Sick Source: (Queen’s University)
- 67. Dan Foreman-Mackey CCPP@NYU dfm.github.com Jonathan Sick Source: (Queen’s University)
- 68. Dan Foreman-Mackey CCPP@NYU dfm.github.com Jonathan Sick Source: (Queen’s University)
- 69. Case Studies CFHT Cosmic ray removal Flat-fielding Sky subtraction ... Mosaic making MongoDB Persistent Metadata + GeoSpatial Indexing img1.fits img2.fits ... img4000.fits Source: Jonathan Sick (Queen’s University) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 70. Astrometry.net Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 71. Astrometry.net Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 72. Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 73. Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 74. Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 75. Crowdsourcing Comet Holmes Case Studies Lang & Hogg (2011) ~2500 JPGs Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 76. Crowdsourcing Comet Holmes Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 77. Crowdsourcing Comet Holmes Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 78. Crowdsourcing Comet Holmes Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 79. Crowdsourcing Comet Holmes Case Studies Source: Lang & Hogg (2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 80. Crowdsourcing Comet Holmes Case Studies Source: Lang & Hogg (2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 81. Crowdsourcing Comet Holmes Case Studies Source: Lang & Hogg (2011) github.com/dfm/MarkovPy Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 82. Crowdsourcing Comet Holmes Case Studies Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 83. Growing Datasets User Interaction Number Data And Much Crunching Management More... ( ) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 84. Growing Datasets User Interaction Easy! Number Data And Much Crunching Management More... ( ) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 85. Growing Big Questions Datasets User Interaction Easy! Number Data And Much Crunching Management More... ( ) Dan Foreman-Mackey CCPP@NYU dfm.github.com
- 86. Dan Foreman-Mackey Center for Cosmology & Particle Physics (NYU) dfm.github.com @__dfm__ Dan Foreman-Mackey CCPP@NYU dfm.github.com