Real-Time Analysis of Streaming Synchotron Data: SCinet SC19 Technology Challenge Winner
Download as PPTX, PDF1 like254 views
This document describes a real-time workflow for analyzing streaming synchrotron data using high-performance computing resources. Synchrotron experiments produce large amounts of data that need to be reconstructed in real-time. The workflow includes data acquisition from multiple simulated beamlines, distribution to compute nodes, tomographic reconstruction using TraceX, denoising using TomoGAN, and visualization of results. A demonstration of this workflow is being run on Argonne Leadership Computing Facility's Theta supercomputer to process streaming data from 16,000 cores in real-time and provide reconstructed volumes and feedback to experiments.
Real-Time Analysis of Streaming Synchotron Data: SCinet SC19 Technology Challenge Winner
- 1. TUESDAY, NOVEMBER 19, 2019 REAL-TIME ANALYSIS OF STREAMING SYNCHROTRON DATA SC’19 TECHNOLOGY CHALLENGE DEMOerhtjhtyhy TEKIN BICER Assistant Computer Scientist Data Science and Learning Division, CELS X-Ray Science Division, APS Argonne National Laboratory Denver, Colorado US COLLABORATORS StarLight & Northwestern University Argonne Leadership Computing Facility, ANL Advanced Photon Source, ANL University of Chicago Northern Illinois University
- 2. SYNCHROTRON EXPERIMENTS Synchrotron light sources help scientific experiments of many fields – Studying internal morphology of materials/samples with very high spatial and temporal resolutions Real-time analysis of synchrotron experiments – Change data acquisition for dynamic systems – Adjust experimental parameters on the fly – Detect errors early in experiments – Enables smart and efficient experimentation High performance network and compute resources are necessary 2
- 3. TOMOGRAPHIC DATA ACQUISITION AND ITERATIVE RECONSTRUCTION 3 Data AcquisitionIterative Tomographic Reconstruction advanced object and data models and can provide better image quality than does FBP on a limited number of projections. Although many variations exist, the basic iterative recon- struction method involves three major steps, as depicted in Fig. 2. First, an initial guess of the volume object, which might simply be an empty volume, is used to calculate the simulated data through a forward model. Second, the simulated data are compared with the measured data. Third, an update of each model estimate is performed based on the employed algorithm. Reconstruction of an object might require hundreds of iterations, depending on the experiment and sample. Theearliest and most basic form of iterativereconstruction is the algebraic reconstruction technique (ART), which involves solving a sparse linear system of equations in the form of Af = p where p is the projection data, A is the forward projection operator, and f is the unknown 3D object to be determined. While ART provides satisfactory images and has a fast convergence rate, the iterations must be stopped before a deteriorating “salt and pepper” or “checkerboard” effect begins to degrade the object estimate. A variation of the ART method is the simultaneous iterative reconstruction technique (SIRT) in which the updates to the solution are computed by taking into account all rotation angles simultaneously in one iteration, as follows: f k+ 1 = f k + λAT (p − Af k ). (3) As in the ART method, a relaxation parameter λ can be used to control convergence in certain cases. SIRT typically produces better quality reconstructions than does ART and is more robust to outliers in the measurement data. In our system, we use more advanced iterative reconstruction algorithms. However, the main computational steps remain the same. Fig. 3: Reconstructed image of a shale sam streamed projections: (a) fixed angle, offset=1 offset=5◦ ; (c) optimized interleaved. The ra [0,180)◦ . processes; (2) the analysis system, which i analysis and reconstruction of streaming d controller, which analyzes reconstructed data subsections, we explain each of these compo A. Data Acquisition and Distribution Data acquisition at current tomography b cally performed with one of two methods: fix or interleaved. With fixed-angle rotation, acquisition sta starting point and then increments by a spec to a specified ending point. If, for example, ending angles are 0◦ and 180◦ , respectively, ◦
- 4. HIGH-PERFORMANCE TOMOGRAPHIC IMAGE RECONSTRUCTION 11,283 *M. Hidayetoglu, T. Bicer et al., Supercomputing 2019 *T. Bicer, D. Gursoy et al., Advanced Structural and Chemical Imaging 2017 Dataset: Dyer et al., Society for Neuroscience (eNeuro) 2017
- 6. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION WORKFLOW (DATA ACQUISITION) * V. De Andrade et al., Nanoscale 3D imaging at the Advanced Photon Source, SPIE’16 Continuous vs. Interleaved DAQ
- 7. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION WORKFLOW (DISTRIBUTOR) . .. Normalization Distribution
- 8. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION WORKFLOW (TRACE-X) * T. Bicer et al., Advanced Structural and Chemical Imaging, 2017 * T. Bicer et al., eScience, 2017 • * TraceX: A High-Throughput Tomographic Reconstruction Engine for Large-Scale Datasets • Sliding window with adjustable runtime params. • Length (w), iteration (i), func. trigger freq (s). • Reduction-based processing model • Highly scalable and efficient • Replicated reduction objects • 32K cores on Mira, 64K cores on Theta
- 9. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION WORKFLOW (TOMOGAN: DENOISER) * Z. Liu, T. Bicer et al., Deep Learning on Supercomputer, SC’19 * Z. Liu, et al., JOSA A (Under review)
- 10. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION WORKFLOW (VISUAL OUTPUTS) Measurement Normalized Measurement Reconstructed Image (3D vol.) Denoized Image (3D Vol.)
- 11. DEMO SETUP 16K Cores100GigE Conn. * 100GigE network enables simulation of 10 beamlines each with 10GigE detector Gateway Node(s) Argonne Leadership Computing Facility Compute Nodes (Theta) ... Starlight / TC Booth @ SC Floor Data Acquisition Simulator and Data Transfer 100 GigE... Visualization and Feedback Data Analysis Reconstructed Volume TomoGAN Denoised Volume ... ... Sinogram Distribution ... Normalized Measurement Data ... Volume Gather Measurement Data ... *Multiple Beamline Simulation Volume Rendering (Cooley) Reconstructed Volumes ØMQ
- 12. DEMO
- 13. THANKS QUESTIONS? Acknowledgement • DSL & MCS: Zhengchun Liu, Raj Kettimuthu, Joaquin Chung, Stefan Wild, Ian T. Foster • ALCF: Venkatram Vishwanath, Mike Papka, Suport & Allocation Teams • APS: Doga Gursoy, Junjing Deng, Jeff Klug, Vincent De Andrade, Pavel Shevchenko, Francesco De Carlo, Stefan Vogt • StarLight & Northwestern: Se-Young Yu, Jim Chen, Fei Yeh, Joe Mambretti • Many others… Papers at SC’19 • Mert Hidayetoglu et al., “MemXCT: Memory- Centric X-Ray CT Reconstruction with Massive Parallelization”, Technical Paper • Zhengchun Liu et al., “Deep Learning Accelerated Light Source Experiments”, Deep Learning on Supercomputers