Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Nandez
1 like888 views
The document discusses the analysis of Andromeda Galaxy data using Spark, highlighting SHARCNET's role in providing high-performance computing services to researchers. It describes the significance of Andromeda, including its size, star count, and relevance in astrophysics, while presenting data challenges regarding new celestial sources found through infrared observations. The preliminary findings indicate a total of 1,391 known sources, with recommendations for machine learning approaches to classify additional unidentified objects.
Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Nandez
- 1. Analyzing Andromeda Galaxy data using Spark Jose Nandez SHARCNET – University of Western Ontario jnandez@sharcnet.ca
- 2. What is ? • Shared Hierarchical Academic Research NETwork, • A consortium of 18 Ontario academic institutions, lead by University of Western Ontario • Partner of Compute Canada that oversees funding and distribution of equipment. • Sysadmins and HPC specialist, 20 in total, distributed across 6 institutions.
- 3. What does SHARCNET do? • Provides service and support to all SHARCNET researchers in High Performance Computing. • Researchers are part of partner universities across Ontario. • Starting to provide service for large data needs: – With storage and processing of large data sets – Data processing using Spark, Hadoop, etc – Data mining and Machine Learning
- 4. What is the Andromeda Galaxy? • Known as M31, or Messier 31 • Spiral galaxy • 2.5 million light-years • Closest galaxy • Bigger galaxy than ours
- 5. Why Andromeda galaxy? • Cool wallpaper • t-shirts, • Mugs … • Science?
- 6. Andromeda Galaxy in Science • It has a ~ trillion stars • 2.5 times longer than our galaxy • Thought to have merged with another galaxy • It contains about 26 known black holes • It can be used as a galaxy laboratory for extragalactic astronomy • Our galaxy will collide with it (in about 4 billion years)
- 8. Particularly… • It has been recognized the extension of Andromeda. • The area shows the extension of the galaxy, further than thought before. • M. Rafiei Ravandi et al 2016.
- 9. Extended Andromeda • They were taken from Spitzer-IRAC which is an Infrared telescope. • It has 426,529 new sources. • Extends observations for disc and halo.
- 10. Classification of these objects • Do all these sources (426,529) are part of Andromeda? • Are they all known from previous catalogs? • What type of object (such as Black holes, galaxies, etc) are those new sources? • What can we learn from these new objects?
- 11. Which catalogs? • Astronomical databases : – SIMBAD (39,022) – NED (126,862) – MAST (118,854,914) • Sources only around M31, sources are in different wavelengths (IR, Optical, UV) • Then compare them with the observed objects.
- 12. How hard could it be? 𝑊𝑒 𝑑𝑒𝑓𝑖𝑛𝑒𝑑 2)) 𝑜𝑟 2 arcsec as a good match. Arcsec = 1/3600∘ , angular measurement, not linear measurement (such as miles/km).
- 13. 𝑘𝑒𝑦 − 𝑣𝑎𝑙𝑢𝑒 = 𝑅𝐴, 𝐷𝐸𝐶 , 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 → join(), groupByKey(), filter(), map(), sortByKey()
- 14. NED+SIMBAD+IRAC
- 15. Counts? • 613 Stars • 70 Globular Cluster • 63 X-rays sources • 62 Galaxies • 52 Star clusters • Total known sources: 1,391
- 16. And the rest? • They are not part of SIMBAD, NED or MAST • What about other catalog? • Can we classify them? • Can we use machine learning?
- 17. Conclusions • MAST has a higher resolution than IRAC-catalog, SIMBAD and NED. • Only 1,391 known sources from a matched between NED + SIMBAD + IRAC-catalog. • The rest could be classified using ML using the known object features in order to give a classification. • We need more data for a better classification.
- 18. Thank You!Collaborator: Prof. Pauline Barmby, Department of Physics, University of Western Ontario Photos: Mainly from NASA, ESO, EarthSky, MacOS.