Rubyslava + PyVo #48
Download as PPTX, PDF0 likes710 views
Skypicker is a flight ticket search and booking engine that processes hundreds of terabytes of airline data monthly and sells thousands of tickets daily in Europe, Russia, and China. It uses PostgreSQL extensively to power its API and databases. The document discusses Skypicker's PostgreSQL infrastructure, which includes 5 database clusters with over 0.5TB of memory each that handle 20 million updates per hour. It also covers optimizations made to scale the databases, such as table partitioning, cascading replication, and query tuning. Skypicker is currently preparing for a large increase in data and is hiring database masters and developers.
Rubyslava + PyVo #48
- 2. Skypicker • flight ticket search&booking engine • covering markets in Europe, Russia and China • hundreds of TBs of airline data processed monthly • selling thousands of tickets daily • covering LCCs only • LCs in progress!
- 3. API • millions of daily searches • average response time <1s (we are slowing down some queries) • Built on top of PostgreSQL • used worldwide =) • a huge home grown data processing framework running under the API
- 4. SP databases • 5 db clusters, >0.5TB memory each • the main one handling 20M updates / hour (this is caused be the airline tickets price changes) • main table has 1 billion+ rows • basically unlimited read scaling with the replication feature • managed mostly by Ansible and custom bash stuff for semi auto failover • there are tools like repmgr though
- 5. PostgreSQL • our silver bullet for everything • for low-cost Big data, there is no better combo than PG+Redshift, yet. • dont even try with Hadoop/Cassandra, they will make your wallet cry
- 6. HW tips • running on bare metal, because the SSD RAID 10 on Intel 3700 series • series 3500 will kill you. After few weeks of 24/7 load, there will be a huge performance drop, always • (its a feature!)
- 7. Why not cloud • AWS is not the best fit for a high performance PG cluster, the I/O is unstable and unpredictable • joyent.com cloud is fine (8k$/mth/instance) • bare metal from Rackspace works also well (1800$/mth). The traffic can get expensive here…
- 8. How we found out these things • …randomly…by fucking things up… • But! We multiplied our master db performance 5 times in the past 6 months • From 15M to nearly 75M
- 9. Prove it! • October 2014, 15M • April 2015, 75M
- 10. Replication for dummies • simple master-slave • a good way to die
- 12. Pros&cons • adds some replication delay (-) • nobody cares (+) • because it scales! (++)
- 13. How the data flows 1. new price for the flight pushed to a queue 2. picked up by a worker 3. inserted to db 4. (magic happens here, shitload of updates) 5. copied to slave servers for select statements 6. search query on slave server 7. …booking made…? 8. profit!
- 14. Queue over the db • our data processing framework is pushing the data to a redis queue • workers are picking up the data and inserting to DB • load can be easily balanced here • you wont loose any data if you need to restart your db (this can be achieved also with pgbouncer) • monitor the size of the queue and keep it near 0 =)
- 15. HaProxy • probably the most stable piece of software ever made. TCP balancer • has a health check for PG • if your slave will go down, nobody will notice • (just dont forget to have alert for it)
- 16. Pgbouncer • small shit • useful when you are doing thousands of connections to your db • lowered the server load to half • boosted the writes by 30%
- 17. Optimalizations steps (for dummies) 1. optimize your queries with Explain 2. do some pg config changes 3. buy better hw 4. goto 2
- 18. Little bit advanced! • table partitioning (this is the game changer) • partial indexes • turn off vacuum, use pg_repack for rebuilding the tables • run analyse often • turn on the genetic query planner
- 19. Redshift • Dummy PG database from Amazon made for science&some sql • its costly to download the data from AWS after they are processed • you should also try Snowflake, Vertica
- 20. Redshift flow • using the PG fdw feature to connect RS remotely to our slave database • download data • process it • push it to master db
- 21. Postgresql replication • no battle tested master-master solution, yet (9.4) • its async - dont forget to monitor the delay between your master and slaves • cascading replication for unlimited scaling
- 22. Postgresql config tuning • 12-Step Program for Scaling Web Applications on PostgreSQL from Wanelo.com • they cover every aspect of the config optimalization and we dont want to copy it here =)
- 23. What are our pains why we are here • our data will grow 10 times by adding legacy carriers in the next 2 months • we need DB masters and developers who will help us to manage this growth
- 24. We are hiring! • We offer • many money • skills Get in touch at jk@skypicker.com