Cloud hpc-bigdata-challenges
1 like625 views
The document discusses data science and distributed computing. It includes details about the author's contact information and areas of expertise including locations, programming tools, and frameworks. It then discusses perspectives on data science and how fields are now data science fields. It provides examples of using data exploration and data mining from various sources. It also includes information about genetic causes of disease research conducted on Azure cloud using many cores. Finally, it lists various data analytics and machine learning tools and frameworks.
Cloud hpc-bigdata-challenges
- 4. IT PAC
- 6. Melbourne Sydney Brazil Beijing
- 7. Programming tools: Scala, IPython, Azure ML, … Frameworks: Spark, Hadoop, Yarn, HDInsight, Reef, Twister, Brisk Software Defined Storage Software Defined Networks Hardware Abstraction/Virtualization
- 11. Every research field is now a data science field
- 12. Last few decades Thousand years ago Last few Today and the Future hundred years 2 2 2 . 3 4 a G c a a Simulation of complex phenomena Newton’s laws, Maxwell’s equations… Description of natural phenomena Unify theory, experiment and simulation with large multidisciplinary Data Using data exploration and data mining (from instruments, sensors, humans…) Distributed Communities
- 13. Video Link
- 19. Inputs (training data) Labels Hidden layers Input data Detected features Mona Lisa
- 21. •The Genetic Causes of Disease (David Heckerman) •WellcomeTrust for a GWAS for a large population •Looking for causes for seven common diseases (bipolar, r. arthritis, coronary, hypertension, ….) •Confounding is a problem. Needed a new algorithm. •Ran on Azure cloud using 35,000 cores in 3 weeks.
- 24. Chameleon Cloud SDN NIH data commons
- 25. Mesos Tachyon Docker Spark Data Analytics and ML programming tools Reef Twister
- 29. •Many Examples •The Challenge: sustainability Data Acquisition & modelling Collaboration and visualisation Analysis & data mining Dissemination & sharing Archiving and preserving