![Couchbase – Map Reduce Views
{
"website": {
"published": true,
"id": "affd", ...
},
"office_id": "1",
"city_id": "AUHA",
"raw_data": [...]
"address":
"closing_time": "00:00",
"routenodeid": "9618",
"type": "office“
}](https://image.slidesharecdn.com/allyourbaseconfahannv3public-141020092130-conversion-gate02/85/SomeSQL-at-Skyscanner-Scaling-in-a-changing-world-of-databases-and-hardware-23-320.jpg)
SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware
- 1. SomeSQL Scaling in a changing world of databases and hardware Alistair Hann CTO, Skyscanner
- 2. Buzzwords
- 3. Web 2.0
- 5. Big Data
- 6. NoSQL
- 7. Scaling the live pricing cache
- 9. Website Native Apps APIs and White Labels Traditional Airlines Budget Airlines Online Travel Agencies
- 12. Prices +Timetables Data Collection Services 1) Which websites should we show? 2) What prices do we already have cached? 3) Live update what we still need. 5) Return the prices to the user. 4) Clean up and save the new data Live Pricing Service
- 14. Live Pricing Service Cached Prices (key/value)
- 15. 2000 quotes per second 2 bn itineraries and quotes 270 GB table 250 GB indices
- 16. What we really needed Consistency Horizontal Scaling Elasticity Persistence Speed Resilience Simplicity
- 17. Live Pricing Service Cached Prices (key/value)
- 21. Beyond key value
- 23. Couchbase – Map Reduce Views { "website": { "published": true, "id": "affd", ... }, "office_id": "1", "city_id": "AUHA", "raw_data": [...] "address": "closing_time": "00:00", "routenodeid": "9618", "type": "office“ }
- 24. What about the hardware?
- 25. Disk for VMs c.f. 250,000 iops Fusion I/O Standard $0.03 / GB Glacier $0.01 / GB
- 26. Quote Bus staging UK1 Thrift long-term archive GZIP queryable hierarchical LZO queryable flat filter Loader GZIP Quote Bus UK2 Thrift Loader
- 27. hierarchical flat Hadoop cluster or Elastic MapReduce analysts query load analytical tools feed export
- 28. The death of the data warehouse
- 29. A distributed database… Fluentd Kafka reporting Graphite Stitched events Fluentd Stitched events Operational metrics Raw JSON events Errors Elastic MapReduce Raw Events Trigger and view materialization Indexes on the data
- 30. Some things don’t change
- 31. CC Images courtesy of chaya760 on Flickr
- 32. We still face the same challenges RAM and Disk i/o concerns Administration Security Data insert and retrieval Monitoring and alerting Performance optimization
- 33. The report of my death was an exaggeration
- 34. Elastic Search
- 35. NoSQL
- 38. Edinburgh Quartermile One 15 Lauriston Place Edinburgh EH3 9EN Glasgow 5th floor, 151-155 St Vincent St, Glasgow G2 5NW Singapore No. 08-01&04 & 09- 04 8th floor, Robinson Point, 39 Robinson Rd, Singapore Beijing Level 19, Tower E2, Oriental Plaza, No. 1 East Chang An Avenue, Dong Cheng District, Beijing 100738 Miami 1395 Brickell Ave, Suite 900, Miami, Florida 33131 Barcelona Torre NN, Calle Tarragona, 157, 4a Planta, Barcelona, 08014 thank you
Editor's Notes
- #30: Approach heavily inspired by http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying