Processing of raw astronomical data of large volume by map reduce model
1 like432 views
The document discusses the processing of large volumes of raw astronomical data using a MapReduce model, highlighting the development of a configurable image data pipeline at Moscow State University and the Space Research Institute of the Russian Academy of Sciences. It outlines various steps involved in processing and analyzing this data, including machine learning applications for astrophysical tasks such as object classification and distance estimation. The research indicates the practical utility of the MapReduce pipeline for astrophysicists, with further experiments planned for scaling and efficiency improvements.
Processing of raw astronomical data of large volume by map reduce model
- 1. Processing of raw astronomical data of large volume by MapReduce model Gerasimov S.V., Kolosov I.Y., Glotov E.S., Popov I.S. Faculty of Computational Mathematics and Cybernetics Lomonosov Moscow State University Meshcheryakov A.V. Space Research Institute of the Russian Academy of Sciences
- 2. Digital Sky Surveys pipeline Telescope Raw images Catalogue
- 4. Data volumes 2010-2014 (PS1) ● 1400 mega-pixels ● 0,4 PB / year ● 2PB totally 1998-2009 ● 120 mega-pixels ● 0.08PB totally
- 5. Data volumes 2022-2032 ● 3200 mega-pixels ● 15 TB data every night ● 60PB of raw data in 10 years ● 15PB catalogue 2013-2018 ● 570 mega-pixels ● 0.1 PB/year ● 0.5 PB in total
- 6. The astronomical science of big data sets Biggest questions ● nature of dark energy ● nature of dark matter Small effects ● requires large volumes (all-sky and high depth imaging) ● systematics are important Small teams ● require ability to analyse big data sets
- 7. Our research ● Research of big data technologies to process and store raw astrophysical data (current report) ● Creation of experimental prototype of configurable astronomical image data pipeline based on MapReduce (current report) ● Research & developement of machine learning algorithms and their “big” versions to solve actual astrophysical tasks: ○ extragalactic objects distance (and other properties) estimation ○ star/galaxy/quasar classification ○ transient sky objects detection and classification ○ exploration of hidden structure in the sky objects distribution … to help astrophysicists to do their job better!
- 8. Pipeline steps Step Task Co-addition Projection Background estimation and subtraction Stacking Catalogue creation Background estimation and subtraction Image areas filtering Image segmentation (object groups extraction on filtered images) Deblending (object extraction inside groups) Artifacts removal (cleaning) Basic objects features measurement Star/galaxy classification Creation of PSF-model on stars Accurate measurement of all features of objects taking into account PSF-model Step Description Raw processing Removal of CCD noise and artefacts Astronomical calibration Detection of World Coordinate System (WCS) Photometric calibration Objects intensity callibration Telescope / sky survey side steps Intelligent steps
- 9. Co-addition depth effect + 53 more images
- 10. Pipiline tools: astromatic.org Image co-addition Objects extraction, measurement of their features Point Spread Function (PSF) -modelling
- 11. Impact of telescope and atmospheric effects on astronomical images Courtesy of LSST PhotoSIM PSFEx uses stars detected on image for empirical PSF modeling.
- 12. Distributed pipelines ● Astromatic-Wrapper https://github.com/fred3m/astromatic_wrapper ● Wiley et al. Astronomy in the Cloud: Using MapReduce for Image Coaddition ● Farivar et al. Cloud Based Processing of Large Photometric Surveys ● Montage: an astronomical image mosaic engine http://montage.ipac.caltech.edu/ ● LSST www.lsst.org http://confluence.lsstcorp.org/
- 13. Proposed approach pipeline machine learning, data miningcatalogue
- 14. Proposed approach MapReduce + HDFS head node SWarp, SExtractor PSFEx config files
- 15. Co-addition step 1,1 2,1 2,2 1,2 map(filename, image): if doesn’t belong to target sky fragment return image’=projection_and_background_subtraction (image) # SWarp return ((i,j), image’) reduce((i,j), list<image>): cell=stacking(list<image>) # SWarp return ((i,j), cell)
- 16. Catalogue creation step map((i,j), stacked_image): basic_object_features, object_icons=basic_extract(stacked_image) #SExtractor psf_model=create_psf_model(basic_object_features, object_icons) #PSFEx object_features=extract(stacked_image, psf_model) #SExtractor return ((i,j), objects_features)
- 17. Objects on cell frontier 1,1 2,1 2,2 1,2
- 18. Experiments Stripe82 SDSS DR 12 ● stripe ● run (north / south)
- 19. Experiments D12 (4 cores, 28GB RAM, 200GB SSD) x 12 Co-addition cell-size: 0,7o x0,7o (0,5o +0,2o ) cell-count: 60x5 mapreduce.map.memory.mb=5GB mapreduce.reduce.memory.mb=5GB Catalogue creation mapreduce.map.memory.mb=5GB Source FITS-files (size of each ~12MB) were initially converted to SequenceFile format to suite HDFS block size.
- 22. Co-addition: MapReduce metrics Volume Processing time, min. map shuffle sort reduce total 14 GB 16 17 1 10 27 21 GB 17 17 1 14 33 33 GB 23 27 1 21 49
- 23. Results & next steps ● Current numbers showed MapReduce pipeline is practically useful for astrophysicists ● Further experiments scheduled to measure scalability on larger data volumes (1+TB) ● Some enhancements in the MapReduce pipeline will be implemented and their impact on performance will be researched ● 2016-2017: research and development of “big” versions of machine learning and data mining algorithms for astrophysics
- 24. Acknowledgment The project is supported by RFBR grant #15-29-07085 офи_м Cloud resources were provided as grant “Microsoft Azure for Research”