Speed up your GIS server - run GIS software on solid-state drives (SSD)
5 likes2,861 views
The document discusses the performance advantages of using solid-state drives (SSDs) over traditional hard disk drives (HDDs) for Geographic Information Systems (GIS) applications. It covers comparisons between SSDs and HDDs, testing methodologies, and benchmarks to highlight the potential for increased data access speeds, particularly for geospatial data processing. The content also addresses the needs for both read and write access in various use cases, such as tile servers and databases.
Speed up your GIS server - run GIS software on solid-state drives (SSD)
- 1. Speed up your GIS server GIS software on solid-state drives (SSD) FOSS4G2011 - #1 GIS software on solid-state drives (SSD)
- 2. Speed up your GIS server GIS software on solid-state drives (SSD) Seongbong Kim Daniel Kastl KimSeongbong@next-group.jp daniel@georepublic.de FOSS4G2011 - #2 GIS software on solid-state drives (SSD)
- 3. Motivation ● Involved in pgRouting project ● Lots of questions about performance ● pgRouting's bottleneck is loading data from hard disks into memory ● How to be as fast as Google? There is no simple answer ... FOSS4G2011 - #3 GIS software on solid-state drives (SSD)
- 4. … what about running PostgreSQL/pgRouting on SSD? FOSS4G2011 - #4 GIS software on solid-state drives (SSD)
- 5. Who is using SSD with a laptop? FOSS4G2011 - #5 GIS software on solid-state drives (SSD)
- 6. Who is using SSD on a server? FOSS4G2011 - #6 GIS software on solid-state drives (SSD)
- 7. What is a solid-state drive? A solid-state drive (SSD) is a data storage device that uses solid-state memory to store persistent data with the intention of providing access in the same manner of a traditional block i/o hard disk drive. SSDs are distinguished from traditional hard disk drives (HDDs), which are electromechanical devices containing spinning disks and movable read/write heads. In contrast, SSDs use microchips which retain data in non- volatile memory chips and contain no moving parts. http://en.wikipedia.org/wiki/Solid-state_drive FOSS4G2011 - #7 GIS software on solid-state drives (SSD)
- 8. SSD vs. HDD (1) Characteristic Solid-state drive Hard disk drive Spin-up time short long Random access time ~ 0.1 ms 5-10 ms Read latency time short long Read performance consistent varying Moving parts no yes Weight/size smaller larger Write longevity type dependent unlimited FOSS4G2011 - #8 GIS software on solid-state drives (SSD)
- 9. SSD vs. HDD (2) Characteristic Solid-state drive Hard disk drive Parallel operation possible no Shock, altitude, resistant not resistant vibration Magnetic no possible susceptibility Software encryption limited yes Costs (energy) lower high Costs (capacity) high “low” FOSS4G2011 - #9 GIS software on solid-state drives (SSD)
- 10. MLC vs. SLC Multi-Level-Cell Single-Level-Cell ● shorter lifespan ● longer lifespan (10.000 write cycles) (100.000 write cycles) ● higher data density ● lower data density ● slower throughput ● higher throughput ● less efficient writing ● more efficient writing ● less expensive ● more expensive FOSS4G2011 - #10 GIS software on solid-state drives (SSD)
- 11. Test Server Specification Model Name Intel(R) Xeon(R) CPU - E5620 @ 2.40GHz Processors 6 CPU cores 4 Cache Size 12.288 KB Total Memory 24.675.980 KB (24 GB) RAID 0 Total SSD 750 GB (SLC) OS CentOS 5.6 FOSS4G2011 - #11 GIS software on solid-state drives (SSD)
- 12. Benchmark FOSS4G2011 - #12 GIS software on solid-state drives (SSD)
- 13. Data & Tools ● Geodata ● Raster: Landsat imagery North America ● Vector: OpenStreetMap Planet ● Processing ● PostgreSQL import with Imposm ● Image processing with GDAL ● Routing topology with osm2po ● Image rendering with Mapserver (Fast CGI) FOSS4G2011 - #13 GIS software on solid-state drives (SSD)
- 14. Ambitious Plans ● File system benchmarks ● WMS map rendering from raster data ● WMS map rendering from vector data (DB) ● Tile seeding from raster data ● Tile seeding from vector data (DB) ● Serving from tile cache FOSS4G2011 - #14 GIS software on solid-state drives (SSD)
- 15. Reality ● File system benchmarks ● WMS map rendering from raster data ● WMS map rendering from vector data (DB) ● Tile seeding from raster data ● Tile seeding from vector data (DB) ● Serving from tile cache FOSS4G2011 - #15 GIS software on solid-state drives (SSD)
- 16. Benchmarking Tools ● Apache JMeter for WMS ● pgBench for PostgreSQL ● FIO for file system ＝＞ WMS Shootout FOSS4G2011 - #16 GIS software on solid-state drives (SSD)
- 17. File I/O Benchmark 4000 3500 3000 2500 16k 32k 64k 2000 128k 256k 512k 1024k 1500 1000 500 0 Sequential Read Sequential Write Random Read Random Write FOSS4G2011 - #17 GIS software on solid-state drives (SSD)
- 18. FOSS4G2011 - #18 GIS software on solid-state drives (SSD)
- 19. FOSS4G2011 - #19 GIS software on solid-state drives (SSD)
- 20. FOSS4G2011 - #20 GIS software on solid-state drives (SSD)
- 21. FOSS4G2011 - #21 GIS software on solid-state drives (SSD)
- 22. Use Case - VOD or FTP Requirements ● Mainly reading access ● Fast read rate ● Serving data Example ● Tile server ● Data server http://www.maptiler.org/ FOSS4G2011 - #22 GIS software on solid-state drives (SSD)
- 23. Use Case - DB / FS only Requirements ● Combined writing and reading access ● Fast read rate Examples ● Geo-database ● Map rendering ● Data processing FOSS4G2011 - #23 GIS software on solid-state drives (SSD)
- 24. Lessons Learned ● Different field of expertize ● Relatively small amount of community resources ● Standards & best practices ● Comparability ● PostgreSQL/PostGIS – how to tune it right? … well, everyone asks for benchmarks FOSS4G2011 - #24 GIS software on solid-state drives (SSD)
- 25. FOSS4G2011 - #25 GIS software on solid-state drives (SSD)
- 26. Want to benchmark your Open Source software? FOSS4G2011 - #26 GIS software on solid-state drives (SSD)
- 27. Thank you! LSD Tech Korea KimSeongbong@next-group.jp daniel@georepublic.de FOSS4G2011 - #27 GIS software on solid-state drives (SSD)