DockerCon EU 2015: Finding a Theory of the Universe with Docker and Volunteer Computers
7 likes8,294 views
The document discusses the use of Docker and volunteer computing in cosmology research, specifically through the project cosmology@home, which analyzes data from the Planck satellite to understand neutrino mass. It outlines the benefits of Docker for managing computational tasks and improving efficiency in running simulations of the universe. The author invites contributions to their open-source projects and emphasizes the advantages of utilizing volunteer computing resources for scientific advancement.
![What running jobs now looks like
• Create the Docker image which runs my
calculation and push to Docker hub
docker build -t cosmohome/mytheory .
docker push cosmohome/mytheory
...
FROM debian:wheezy
RUN apt-get update &&
apt-get -y install libgfortran3 libgomp1 &&
rm -rf /var/lib/apt/lists/*
COPY ["camb", "HighLExtrapTemplate_lenspotentialCls.dat", "/root/"]](https://image.slidesharecdn.com/dockercon20151-151202150608-lva1-app6892/85/DockerCon-EU-2015-Finding-a-Theory-of-the-Universe-with-Docker-and-Volunteer-Computers-37-320.jpg)
DockerCon EU 2015: Finding a Theory of the Universe with Docker and Volunteer Computers
- 1. Finding a Theory of the Universe with Docker and Volunteer Computers Marius Millea Lagrange Postdoctoral Fellow Institut Lagrange de Paris
- 2. About Me • I'm a cosmologist • ...but I've always been really into computers • I work on analyzing data from a satellite called Planck • I'm the main developer at Cosmology@Home
- 3. Thanks to... Ben Wandelt Rom Walton Cam Boehmer Original creator of Cosmology@Home A BOINC developer Introduced me to Docker ...and many others
- 4. • Cosmology@Home is a volunteer-based computer cluster Active computers: (as of July 2015)
- 5. • Our aim is to better understand our universe (in particular the mass of the neutrino particle) Server Volunteers Possible universes
- 6. Outline • Measuring the mass of the neutrino particle with Planck • Cosmology@Home and how Docker has helped • Future Docker use at Cosmology@Home and in the field of physics in general
- 7. Why the science? • Show you a definite example of the type of calculations we do • Give you a sense of their ever-changing nature
- 8. “For the discovery of neutrino oscillations, which shows that neutrinos have mass” So what is the mass?
- 9. neutrinos Created in nuclear power plants Created in the sun Created by the big bang nasa.gov wikipedia.org Known particles
- 10. ● Neutrinos interact very weakly ● Makes measuring their mass difficult Neutrinos emitted from the sun, as seen through the earth Super-Kamiokande
- 12. The History of the Universe (backwards)
- 13. The History of the Universe (forwards)
- 18. Cold (-270.3999˚C) Hot (-270.4001˚C)
- 19. Cold (-270.3999˚C) Hot (-270.4001˚C)
- 21. Summary thus far • We measure the distortion of the CMB due to neutrinos released from the big bang to infer the mass of the neutrino particle • Our volunteer computers simulate different hypothetical universes with different neutrino masses to tell us how much distortion we would expect, then we compare to data
- 22. Outline • Measuring the mass of the neutrino particle with Planck • Cosmology@Home and how Docker has helped • Future Docker use at Cosmology@Home and in the field of physics in general
- 23. From the user's POV
- 30. From our POV
- 31. Regular apps • Compile executable for each platform • Either statically link or include all necessary libraries • Link against the BOINC libraries and manage user requests for suspend/checkpoint/resume, etc… • BOINC takes care of sending it each host, running it, etc...
- 33. VM apps • Just create one VM image, works on all platforms • BOINC handles suspend/checkpoint/resume • Still some downsides – Its clunky to create a VM image – New app version, user has to redownload whole thing?
- 35. Docker apps +Debian base image (~40Mb compressed) +our app (~3Mb compressed) BOINC BOINC Virtualbox wrapper “boinc2docker” ISO Docker container handles pause/resume generally “BOINC things...” lightly modified to interact with BOINC
- 36. Persistence Logging Startup script boot2docker ISO modifications #!/bin/sh /etc/init.d/docker stop tar cf /root/scratch/boinc2docker_persistence.tar /var/lib/docker/* /etc/init.d/docker start #!/bin/sh docker inspect $IMAGE || { docker pull $IMAGE && save_docker.sh; } docker run --rm -v /root/shared:/root/shared $IMAGE via Virtualbox guest-logging, logs are uploaded to our DB
- 37. What running jobs now looks like • Create the Docker image which runs my calculation and push to Docker hub docker build -t cosmohome/mytheory . docker push cosmohome/mytheory ... FROM debian:wheezy RUN apt-get update && apt-get -y install libgfortran3 libgomp1 && rm -rf /var/lib/apt/lists/* COPY ["camb", "HighLExtrapTemplate_lenspotentialCls.dat", "/root/"]
- 38. What running jobs now looks like • Submit workunit on Cosmology@Home server pointing to my image ./create_work --app boinc2docker boinc_app params e.g. neutrino mass = X... A short script like you saw before which does a docker run cosmohome/mytheory
- 39. End result • Its easy! • I can spend my time being a physicist, not a programmer Previously Planck
- 40. Outline • Measuring the mass of the neutrino particle with Planck • Cosmology@Home and how Docker has helped • Future Docker use at Cosmology@Home and in the field of physics in general
- 41. Docker and volunteer computing
- 42. “BOINC is the #5 super-computer in the world. What if it ran Docker?” 4 2
- 43. Docker and volunteer computing Cosmology@Home: • “Higher fidelity” simulations of the universe Generally: • An “IPython Parallel”-like interface to BOINC • “Service to scientists” • Swarm, Tutum..?
- 44. • From the top: the most computer savvy physicists know of and may use Docker • From the grassroots: more work needed – Good news: physicists develop, ship, and modify code – Docker and MPI (e.g. see “Shifter” at NERSC) Docker and scientific computing
- 45. Summary • Cosmology@Home, a cluster of volunteer computers, now runs Docker • Docker allows us to spend our time being physicists • If anyone is looking for open source projects to contribute to, we welcome helpers, find me or our github.com/marius311/cosmohome