ieee cloud 2015 keynote talk
5 likes1,167 views
The document discusses the evolution of research fields into data science domains, emphasizing the integration of theory, experiment, and simulation using large datasets. It highlights various programming tools, frameworks, and the challenges of sustainability, sharing, and reproducibility in data science. Furthermore, it outlines the importance of collaboration in data acquisition, modeling, and analysis.
ieee cloud 2015 keynote talk
- 6. Every research field is now a data science field
- 7. Last few decades Thousand years ago Today and the FutureLast few hundred years 2 2 2. 3 4 a cG a a Simulation of complex phenomena Newton’s laws, Maxwell’s equations… Description of natural phenomena Unify theory, experiment and simulation with large multidisciplinary Data Using data exploration and data mining (from instruments, sensors, humans…) Distributed Communities
- 10. The Long Tail of Science
- 11. Programming tools: Scala, IPython, Azure ML, … Frameworks: Spark, Hadoop, Yarn, HDInsight, Reef, Twister, Brisk Software Defined Storage Software Defined Networks Hardware Abstraction/Virtualization Container and Distributed Cluster OS (Docker, Mesosphere, K8S)
- 17. The IPython notebook deployed on a VM. 1. Create a coreos VM and open the https port 443 2. Login and issue this one command $docker run -d -p 443:8888 -e "PASSWORD=****" ipython/scipyserver 3. Go to https://yourVMaddress and log in.
- 29. • Many Examples • Challenges: • I only talked about the Analysis … but it there is more • Sustainability • Sharing • Reproducible Science Data Acquisition & modelling Collaboration and visualisation Analysis & data mining Dissemination & sharing Archiving and preserving