A Buffering Approach to Manage I/O in a Normalized Cross-Correlation Earthquake Detection Code for Large Seismic Datasets
- 1. A Buffering Approach to Manage I/O in a Normalized Cross-Correlation Earthquake Detection Code for Large Seismic Datasets Dawei Mu, Pietro Cicotti, Yifeng Cui, Enjui Lee, Po Chen
- 2. Outlines 1. Introduction of cuNCC code 2. Realistic Application 3. Performance Analysis 4. Memory Buffer Approach and I/O Analysis 5. Future Work
- 3. 1. Introduction of cuNCC what is cuNCC ? CUDA based software designed to calculate the normalized cross- correlation coefficient (NCC) between a collection of selected template waveforms and the continuous waveform recordings of seismic instruments to evaluate the waveform similarity among the waveforms and/or the relative travel-time differences. Feb 05, 2016 M6.6 Meinong aftershocks detection • more uncatalogued aftershocks were detected • contributed to earthquake location detection and earthquake source parameters estimation
- 4. 2. Realistic Application M6.6 Meinong aftershocks hypocenter re-location. •traditional method using short term long term to detect events and using 1-D model to locate hypocenter. fewer aftershock detected, the result contains less information due to inaccuracy. •3-D waveform template matching detect events, using 3D model and waveform travel-time differences to re-locate the hypocenter, the result shows more events and more clustered hypocenters, which give us detailed fault geometry •over 4 trillion NCC calculations involved
- 5. 3. Performance Analysis optimization scheme • The cuNCC is bounded by memory bandwidth • The constant memory is used to stack multiple events into single computational kernel • The shared memory is used to improve the memory bandwidth utilization 1. Compute, Bandwidth, or Latency Bound The first step in analyzing an individual kernel is to determine if the performance of the kernel is bounded by com bandwidth, or instruction/memory latency. The results below indicate that the performance of kernel "cuNCC_04 limited by memory bandwidth. You should first examine the information in the "Memory Bandwidth" section to d limiting performance. 1.1. Kernel Performance Is Bound By Memory Bandwidth For device "GeForce GTX 980" the kernel's compute utilization is significantly lower than its memory utilization levels indicate that the performance of the kernel is most likely being limited by the memory system. For this kern factor in the memory system is the bandwidth of the Shared memory.
- 6. 3. Performance Analysis performance benchmark This cuNCC code can achieve high performance without high-end hardware or expensive clusters, optimized CPU based NCC code needs 21 hours for one E7-8867 CPU (all 18 cores) to finish mentioned example while a NVIDIA GTX980 only costs 53 minutes. Hardware Runtime (ms) SP FLOP (×1e11) Achieved GFLOPS Max GFLOPS Achieved Percentage Speedup E7-8867 (18 cores) 2968 1.23 41.36 237.6 17.4% 1.0x C2075 (Fermi) 495 1.8 363.83 1030 35.3% 6.0x GTX980 (Maxwell) 116 1.8 1552.80 5000 31.0% 25.6x M40 (Maxwell) 115 1.8 1569.86 7000 22.4% 25.8x P100 (Pascal) 62 1.8 2911.84 10600 27.5% 47.9x
- 7. 4. Memory Buffer Approach and I/O Analysis the I/O bottleneck After improving the computational performance with GPU acceleration, I/O efficiency became the new bottleneck of the cuNCC’s overall performance. The output file of cuNCC is an 1-D vector of similarity coefficients saved in binary format, which size is equal to seismic data file. CPU NCC I/O operations cost roughly 10% of the total runtime, while the GPU code I/O cost more than 75% of total runtime. 0 125 250 375 500 NCC(CPU) cuNCC I/O Compute
- 8. 4. Memory Buffer Approach and I/O Analysis test environment The SGI UV300 system has 8 sockets 18 core Intel Xeon E7-8867 V4 processors 16 DDR4 32GB DRAM run at 1600 MHz. 4 TB of DRAM in Non-Uniform Memory Access (NUMA) via SGI’s NUMALink technology. 4x PCIe Intel flash cards for a total of 8TB configured as a RAID 0 device and mounted as “/scratch” with a 975.22 MB/s achieved I/O bandwidth (with IOR). 2x 400GB Intel SSDs configured as a RAID 1 device and mounted as “/home” with a 370.07 MB/s achieved I/O bandwidth. The software we used were GCC-4.4.7 and CUDA-7.5 along with MPI package MPICH-3.2.
- 9. 4. Memory Buffer Approach and I/O Analysis use CPU memory as a buffer •Most GPU-enabled computers have more CPU memory than GPU memory. (in our case 48GB << 4TB) •Fixed data chuck size (120 days’) with different total workloads •on ”/scratch” partition, for every data size, the buffering technique costs more overall runtime than the no-buffering •on ”/home” partition, buffering starts to help after reaching the 2400-day’s total workload •the high I/O bandwidth filesystem, the improvement brought by the buffering cannot cover up the overhead of the memory transfer.
- 10. 4. Memory Buffer Approach and I/O Analysis use shared memory virtual filesystem as a buffer •we set 2 TB of DRAM as a shared memory virtual filesystem, and measured the I/O bandwidth achieved 2228.05 MB/s. •on the ”/dev/shm” partition, the high bandwidth of shared memory improves performance greatly by reducing the time used for output. •we gathered the runtime result without buffering scheme from all three storage partitions, and the shared memory partition obtains the best performance.
- 11. 4. Memory Buffer Approach and I/O Analysis I/O test conclusion •for machines support shared memory virtual filesystem, we recommend to use the shared memory as buffer to output cuNCC result, especially when the similarity coefficients are the median result for the following computation. •for those machines do not have shared memory with high bandwidth I/O device, we recommend to directly output the result to storage without the buffering scheme. •for those machines do not support shared memory with low bandwidth I/O device, we should consider to use CPU memory as a buffer to reduce disk access frequency.
- 12. 5. Future Work • further optimize the cuNCC code on the Pascal GPU platform. • implement our cuNCC code with “SEISM-IO” library, which interface allows user to switch among “MPI-IO”, “PHDF5”, “NETCDF4”, and “ADIOS” as low level I/O libraries.
- 13. Thank you for your time !