SlideShare a Scribd company logo
Implementing a VO archive for
datacubes of galaxies
José Enrique Ruiz
Instituto de Astrofísica de Andalucía – CSIC
Sprint 2014 IVOA Interop
May 20th 2014 - ESAC
Discovering Cubes
And doing something…
Collections
SMA B0DEGA
Below 0 Degrees Galaxies
•  30 FITS Files
•  Single Object / Single Line Emission
•  2D Spatial + 1D Velocity + 1D Polarization
•  12 CO 21 Molecular Transition
WSRT WHISP
Westerbork observations of neutral Hydrogen in Irregular and SPiral galaxies
•  33 FITS Files
•  Single Object / Single Line Emission
•  2D Spatial + 1D Frequency + 1D Polarization
•  HI 21cm Transition
Collections
SMA B0DEGA
Below 0 Degrees Galaxies
•  30 FITS Files
•  Single Object / Single Line Emission
•  2D Spatial + 1D Velocity + 1D Polarization
•  12 CO 21 Molecular Transition
WSRT WHISP
Westerbork observations of neutral Hydrogen in Irregular and SPiral galaxies
•  33 FITS Files
•  Single Object / Single Line Emission
•  2D Spatial + 1D Frequency + 1D Polarization
•  HI 21cm Transition
Technologies
•  Linux Ubuntu 12.04 LTS Server
•  MySQL Community Server
•  Apache HTTP Server
•  Django Web Framework 1.6
•  MySQL Workbench
•  IPython Notebooks Framework
•  Python Libraries and Software
•  Kapteyn Package
•  AstroPy
•  APLPy
•  PVExtractor
•  PySpeckit
•  yt project
•  GIPSY
Data ModelImage DM, ObsCore, Spectral DM
FieldID – Type – UType – UCD – Description – FITS
Group Name
Data Model
Single-line / Single-object velocity data cubes
Velocity associated to emission line
Better Flux Characterization
Data Model
Instrumental and Post processing Provenance
Derived properties associated to the Target
Database
….
Database
….
Database
….
Database
….
Database Filling
IPython Notebook Executable Recipe
•  Web tool executable in the browser
•  Imports a single Collection each time
•  Single-target single-line velocity data cubes of galaxies
•  Configuration of parameters for each Collection
•  Type, SubType, File Format, Calibration Level
•  Collection,Access Rights, Publisher, Contact, Creator, CreationType
•  Facilicity, Instrument, Bandpass, DataSource, Proposal
•  UCD/Unit Axis Characterization, Line Characterization
•  Fills a specific MySQL BDD structure
•  DataSets are FITS files placed in a specified folder
•  Configurable FITS headers mapping
•  Produces a SQL Script
•  Addition of Derived Metadata for Targets in a second step
Web Interface
Database Admin
Django Web Framework Integrated Database Admin
Discovery
Response List
Record
Target and Derived Properties
Spatial and Spectral Characterization
Line Velocity Characterization
Polarization and Time
Instrumental and Post Processing Provenance
Dataset Curation and Characterization
Access and Thumbnails
Access and Thumbnails
Access and Thumbnails
VOTables
3D Exploration /Access Data Use Cases
Collapsing the cube: Moment Zero
Integrated Emission
Range in velocity
Range in flux
Beam
3D Exploration /Access Data Use Cases
Collapsing the cube: Moment One
Velocity Weighted Integrated Emission
v
Range in velocity
Range in flux
Beam
3D Exploration /Access Data Use Cases
Channel Maps and Contour Overlay
Channel Slicing
Range in velocity
Range in flux
Beam
Channel Slicing
3D Exploration /Access Data Use Cases
Range in flux
Position Velocity at Fixed RA
3D Exploration /Access Data Use Cases
Position Velocity at Fixed Dec
Range in velocities
Range in flux
Position Velocity at Defined Position Angle
3D Exploration /Access Data Use Cases
Range in velocities
Range in flux
Position angle
Renzogram: 2D overlay of contour channels with different colors
3D Exploration /Access Data Use Cases
Range in velocities
Range in flux
Position angle
Aperture Velocity Profile
3D Exploration /Access Data Use Cases
Integrated Emission Radial Profile
Spatial aperture
Range in velocities
Range in flux
Range in velocities
Range in flux
Position angle
Elliptical rings
Visualization as a service
IPython implementation of yt-project software
3D Exploration /Access Data Use Cases
Conclusions
•  SIAv2 prototype/testbed evolving with in progress standards
•  Two different collections of single target /single line-emission velocity cubes of galaxies
•  Import from FITS into MySQL database with IPython executable notebook /tool
•  Web Interface for discovery and display of SIAv2/ImageDM UCDs and Utypes
•  accesData UseCases implemented as programmatic access services
•  Additions performed into DM to provide better knowledge of the dataSet:
•  Emission-line characterization (velocity and flux)
•  Post-processing provenance in radio interferometric cleaned datacubes
•  Archive-specific physical properties of Targets
LinksSIAv2 Prototype
http://amiga.iaa.es:9000
IPython Notebooks Recipes
Datafiller
http://nbviewer.ipython.org/gist/Bultako/690e88d3548745ca194e
Moments
http://nbviewer.ipython.org/gist/Bultako/6522a416d749701d1310
Slicing Datacubes
http://nbviewer.ipython.org/gist/Bultako/badb836e659599b104db
Contour Overlays on Channel Maps
http://nbviewer.ipython.org/gist/Bultako/73f82690e90e2dbb3d76
Renzograms /Contour Overlays on 2DImage
http://nbviewer.ipython.org/gist/Bultako/11427424
Visualization with yt-project software package
http://nbviewer.ipython.org/gist/bultako/11137094
jer@iaa.es
bultako

More Related Content

PDF
Velocity cubes of galaxies
PDF
Workflows to access and massage VOData
PDF
Research Objects in Wf4Ever
PDF
Workflows in the Virtual Observatory
PDF
Digital Science: Reproducibility and Visibility in Astronomy
PPTX
Virtual Science in the Cloud
PDF
A Recommender Story: Improving Backend Data Quality While Reducing Costs
PDF
IPython Notebooks - Hacia los papers ejecutables
Velocity cubes of galaxies
Workflows to access and massage VOData
Research Objects in Wf4Ever
Workflows in the Virtual Observatory
Digital Science: Reproducibility and Visibility in Astronomy
Virtual Science in the Cloud
A Recommender Story: Improving Backend Data Quality While Reducing Costs
IPython Notebooks - Hacia los papers ejecutables

What's hot (20)

PDF
Astronomical Data Processing on the LSST Scale with Apache Spark
PPTX
Taming Big Data!
PPTX
Accelerating Discovery via Science Services
PPTX
Learning Systems for Science
PPT
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
PPTX
Coding the Continuum
PDF
Geo exploration simplified with Elastic Maps
PPTX
ADASS XXV: LSST DM - Building the Data System for the Era of Petascale Optica...
PPTX
Data Automation at Light Sources
PPTX
Data Tribology: Overcoming Data Friction with Cloud Automation
PDF
Big Data Modeling Challenges and Machine Learning with No Code
PDF
04 open source_tools
PPTX
Scaling collaborative data science with Globus and Jupyter
PDF
07 data structures_and_representations
PDF
Big data visualization frameworks and applications at Kitware
PPTX
Materials Data Facility: Streamlined and automated data sharing, discovery, ...
PDF
Big Data Visualization
PDF
The Matsu Project - Open Source Software for Processing Satellite Imagery Data
PPTX
Big Process for Big Data @ PNNL, May 2013
PPTX
Research Automation for Data-Driven Discovery
Astronomical Data Processing on the LSST Scale with Apache Spark
Taming Big Data!
Accelerating Discovery via Science Services
Learning Systems for Science
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Coding the Continuum
Geo exploration simplified with Elastic Maps
ADASS XXV: LSST DM - Building the Data System for the Era of Petascale Optica...
Data Automation at Light Sources
Data Tribology: Overcoming Data Friction with Cloud Automation
Big Data Modeling Challenges and Machine Learning with No Code
04 open source_tools
Scaling collaborative data science with Globus and Jupyter
07 data structures_and_representations
Big data visualization frameworks and applications at Kitware
Materials Data Facility: Streamlined and automated data sharing, discovery, ...
Big Data Visualization
The Matsu Project - Open Source Software for Processing Satellite Imagery Data
Big Process for Big Data @ PNNL, May 2013
Research Automation for Data-Driven Discovery
Ad

Similar to Implementing a VO archive for datacubes of galaxies (20)

PDF
DataStax and Esri: Geotemporal IoT Search and Analytics
PDF
Multidimensional Data in the VO
PPTX
Multidimensional Scientific Data in ArcGIS
PPT
The Homogenization and Reporting of Groundbased Atmospheric Datasets for the ...
PPTX
ArcGIS and Multi-D: Tools & Roadmap
PPTX
WOTS2E: A Search Engine for a Semantic Web of Things
PDF
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
PDF
Apache CarbonData+Spark to realize data convergence and Unified high performa...
PDF
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
PDF
Suburface 2021 IBM Cloud Data Lake
PDF
LoCloud Geocoding Application, Runar Bergheim, Asplan Viak Internet
PDF
20160922 Materials Data Facility TMS Webinar
PDF
The Ariadne Project
PPT
DA_MAP
PPT
PPT
WebServices_Grid.ppt
PPTX
Spatial Data in SQL Server
PDF
Scalable Preservation Workflows
PDF
Scalable Data Analytics and Visualization with Cloud Optimized Services
DataStax and Esri: Geotemporal IoT Search and Analytics
Multidimensional Data in the VO
Multidimensional Scientific Data in ArcGIS
The Homogenization and Reporting of Groundbased Atmospheric Datasets for the ...
ArcGIS and Multi-D: Tools & Roadmap
WOTS2E: A Search Engine for a Semantic Web of Things
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Apache CarbonData+Spark to realize data convergence and Unified high performa...
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
Suburface 2021 IBM Cloud Data Lake
LoCloud Geocoding Application, Runar Bergheim, Asplan Viak Internet
20160922 Materials Data Facility TMS Webinar
The Ariadne Project
DA_MAP
WebServices_Grid.ppt
Spatial Data in SQL Server
Scalable Preservation Workflows
Scalable Data Analytics and Visualization with Cloud Optimized Services
Ad

More from Jose Enrique Ruiz (16)

PDF
Jupyter notebooks on steroids
PDF
Open Science and Executable Papers
PDF
Digital Science: Towards the executable paper
PDF
Curation and Characterization of Web Services
PDF
Digital Science
PDF
Wf4Ever: Workflow Preservation
PDF
Use of CharDM in an archive of velocity cubes
PDF
Workflow Preservation
PDF
VO web-services-based astronomy workflows
PDF
Web services based workflows to deal with 3D data
PDF
Curating and Preserving Collaborative Digital Experiments
PDF
Collaborative Digital Experiments
PDF
SVO Activities - SEA 2008
PDF
El Observatorio Virtual - eCA
PDF
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
PDF
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Jupyter notebooks on steroids
Open Science and Executable Papers
Digital Science: Towards the executable paper
Curation and Characterization of Web Services
Digital Science
Wf4Ever: Workflow Preservation
Use of CharDM in an archive of velocity cubes
Workflow Preservation
VO web-services-based astronomy workflows
Web services based workflows to deal with 3D data
Curating and Preserving Collaborative Digital Experiments
Collaborative Digital Experiments
SVO Activities - SEA 2008
El Observatorio Virtual - eCA
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i

Recently uploaded (20)

PDF
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
PDF
diccionario toefl examen de ingles para principiante
PPTX
ECG_Course_Presentation د.محمد صقران ppt
PPTX
famous lake in india and its disturibution and importance
PPTX
Classification Systems_TAXONOMY_SCIENCE8.pptx
PPTX
Cell Membrane: Structure, Composition & Functions
PPTX
neck nodes and dissection types and lymph nodes levels
PDF
Biophysics 2.pdffffffffffffffffffffffffff
PDF
AlphaEarth Foundations and the Satellite Embedding dataset
DOCX
Q1_LE_Mathematics 8_Lesson 5_Week 5.docx
PDF
bbec55_b34400a7914c42429908233dbd381773.pdf
PPTX
INTRODUCTION TO EVS | Concept of sustainability
PDF
IFIT3 RNA-binding activity primores influenza A viruz infection and translati...
PDF
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
PPTX
DRUG THERAPY FOR SHOCK gjjjgfhhhhh.pptx.
PPTX
Taita Taveta Laboratory Technician Workshop Presentation.pptx
PPTX
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
PDF
Phytochemical Investigation of Miliusa longipes.pdf
PDF
The scientific heritage No 166 (166) (2025)
PDF
Unveiling a 36 billion solar mass black hole at the centre of the Cosmic Hors...
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
diccionario toefl examen de ingles para principiante
ECG_Course_Presentation د.محمد صقران ppt
famous lake in india and its disturibution and importance
Classification Systems_TAXONOMY_SCIENCE8.pptx
Cell Membrane: Structure, Composition & Functions
neck nodes and dissection types and lymph nodes levels
Biophysics 2.pdffffffffffffffffffffffffff
AlphaEarth Foundations and the Satellite Embedding dataset
Q1_LE_Mathematics 8_Lesson 5_Week 5.docx
bbec55_b34400a7914c42429908233dbd381773.pdf
INTRODUCTION TO EVS | Concept of sustainability
IFIT3 RNA-binding activity primores influenza A viruz infection and translati...
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
DRUG THERAPY FOR SHOCK gjjjgfhhhhh.pptx.
Taita Taveta Laboratory Technician Workshop Presentation.pptx
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
Phytochemical Investigation of Miliusa longipes.pdf
The scientific heritage No 166 (166) (2025)
Unveiling a 36 billion solar mass black hole at the centre of the Cosmic Hors...

Implementing a VO archive for datacubes of galaxies

  • 1. Implementing a VO archive for datacubes of galaxies José Enrique Ruiz Instituto de Astrofísica de Andalucía – CSIC Sprint 2014 IVOA Interop May 20th 2014 - ESAC
  • 4. Collections SMA B0DEGA Below 0 Degrees Galaxies •  30 FITS Files •  Single Object / Single Line Emission •  2D Spatial + 1D Velocity + 1D Polarization •  12 CO 21 Molecular Transition WSRT WHISP Westerbork observations of neutral Hydrogen in Irregular and SPiral galaxies •  33 FITS Files •  Single Object / Single Line Emission •  2D Spatial + 1D Frequency + 1D Polarization •  HI 21cm Transition
  • 5. Collections SMA B0DEGA Below 0 Degrees Galaxies •  30 FITS Files •  Single Object / Single Line Emission •  2D Spatial + 1D Velocity + 1D Polarization •  12 CO 21 Molecular Transition WSRT WHISP Westerbork observations of neutral Hydrogen in Irregular and SPiral galaxies •  33 FITS Files •  Single Object / Single Line Emission •  2D Spatial + 1D Frequency + 1D Polarization •  HI 21cm Transition
  • 6. Technologies •  Linux Ubuntu 12.04 LTS Server •  MySQL Community Server •  Apache HTTP Server •  Django Web Framework 1.6 •  MySQL Workbench •  IPython Notebooks Framework •  Python Libraries and Software •  Kapteyn Package •  AstroPy •  APLPy •  PVExtractor •  PySpeckit •  yt project •  GIPSY
  • 7. Data ModelImage DM, ObsCore, Spectral DM FieldID – Type – UType – UCD – Description – FITS Group Name
  • 8. Data Model Single-line / Single-object velocity data cubes Velocity associated to emission line Better Flux Characterization
  • 9. Data Model Instrumental and Post processing Provenance Derived properties associated to the Target
  • 14. Database Filling IPython Notebook Executable Recipe •  Web tool executable in the browser •  Imports a single Collection each time •  Single-target single-line velocity data cubes of galaxies •  Configuration of parameters for each Collection •  Type, SubType, File Format, Calibration Level •  Collection,Access Rights, Publisher, Contact, Creator, CreationType •  Facilicity, Instrument, Bandpass, DataSource, Proposal •  UCD/Unit Axis Characterization, Line Characterization •  Fills a specific MySQL BDD structure •  DataSets are FITS files placed in a specified folder •  Configurable FITS headers mapping •  Produces a SQL Script •  Addition of Derived Metadata for Targets in a second step
  • 16. Database Admin Django Web Framework Integrated Database Admin
  • 20. Target and Derived Properties
  • 21. Spatial and Spectral Characterization
  • 24. Instrumental and Post Processing Provenance
  • 25. Dataset Curation and Characterization
  • 30. 3D Exploration /Access Data Use Cases Collapsing the cube: Moment Zero Integrated Emission Range in velocity Range in flux Beam
  • 31. 3D Exploration /Access Data Use Cases Collapsing the cube: Moment One Velocity Weighted Integrated Emission v Range in velocity Range in flux Beam
  • 32. 3D Exploration /Access Data Use Cases Channel Maps and Contour Overlay Channel Slicing Range in velocity Range in flux Beam
  • 33. Channel Slicing 3D Exploration /Access Data Use Cases Range in flux
  • 34. Position Velocity at Fixed RA 3D Exploration /Access Data Use Cases Position Velocity at Fixed Dec Range in velocities Range in flux
  • 35. Position Velocity at Defined Position Angle 3D Exploration /Access Data Use Cases Range in velocities Range in flux Position angle
  • 36. Renzogram: 2D overlay of contour channels with different colors 3D Exploration /Access Data Use Cases Range in velocities Range in flux Position angle
  • 37. Aperture Velocity Profile 3D Exploration /Access Data Use Cases Integrated Emission Radial Profile Spatial aperture Range in velocities Range in flux Range in velocities Range in flux Position angle Elliptical rings
  • 38. Visualization as a service IPython implementation of yt-project software 3D Exploration /Access Data Use Cases
  • 39. Conclusions •  SIAv2 prototype/testbed evolving with in progress standards •  Two different collections of single target /single line-emission velocity cubes of galaxies •  Import from FITS into MySQL database with IPython executable notebook /tool •  Web Interface for discovery and display of SIAv2/ImageDM UCDs and Utypes •  accesData UseCases implemented as programmatic access services •  Additions performed into DM to provide better knowledge of the dataSet: •  Emission-line characterization (velocity and flux) •  Post-processing provenance in radio interferometric cleaned datacubes •  Archive-specific physical properties of Targets
  • 40. LinksSIAv2 Prototype http://amiga.iaa.es:9000 IPython Notebooks Recipes Datafiller http://nbviewer.ipython.org/gist/Bultako/690e88d3548745ca194e Moments http://nbviewer.ipython.org/gist/Bultako/6522a416d749701d1310 Slicing Datacubes http://nbviewer.ipython.org/gist/Bultako/badb836e659599b104db Contour Overlays on Channel Maps http://nbviewer.ipython.org/gist/Bultako/73f82690e90e2dbb3d76 Renzograms /Contour Overlays on 2DImage http://nbviewer.ipython.org/gist/Bultako/11427424 Visualization with yt-project software package http://nbviewer.ipython.org/gist/bultako/11137094 jer@iaa.es bultako