PostgreSQL - масштабирование в моде, Valentine Gogichashvili (Zalando SE)
1 like2,597 views
This document discusses how PostgreSQL helped Zalando, one of Europe's largest online fashion retailers, scale their operations. Some key points: - Zalando uses PostgreSQL to store data for over 13.7 million customers, 150,000+ products, and handles 640+ million visits annually. - They access data through stored procedures to maintain consistency and independence from schema changes. Stored procedures also allow easy data modeling and schema changes without downtimes. - Zalando shards their database without limits through their Java stored procedure wrapper which directs calls to the appropriate shard transparently. - They closely monitor PostgreSQL using custom tools like PGObserver and have a dedicated team to ensure high performance and availability.
PostgreSQL - масштабирование в моде, Valentine Gogichashvili (Zalando SE)
- 2. Scaling fashionably How PostgreSQL helped Zalando to become one of the biggest online fashion retailers in Europe
- 3. About me Valentine Gogichashvili Database Engineer @Zalando twitter: @valgog google+: +valgog email: valentine.gogichashvili@zalando.de
- 5. One of Europe's largest online fashion retailers 15 countries 3 fulfillment centers 13.7+ million active customers 1.8 billion € revenue 2013 150,000+ products 640+ million visits in first half-year 2014
- 8. Some more numbers 200+ deployment units (WARs) 1300+ production tomcat instances 80+ database master instances 90+ different database schemas 300+ developers and 200+ QA and PM 10 database engineers
- 9. Even more numbers ● > 4.0 TB of PostgreSQL data ● Biggest instances (not counted before) ○ eventlogdb (3TB) ■ 20 GB per week ○ riskmgmtdb (5TB) ■ 12 GB per day
- 10. Biggest challenges ● Constantly growing ● Fast development cycles ● No downtimes are tolerated
- 11. Agenda How we ● access data ● change data models without downtimes ● shard without limits ● monitor
- 12. Agenda How we ● access data ● change data models without downtimes ● shard without limits ● monitor
- 13. Accessing data - customer - bank account - order -> bank account - order position - return order -> order - return position -> order position - financial document - financial transaction -> order
- 14. Accessing data NoSQL ▶ map your object hierarchy to a document ▶ (de-)serialization is easy ▶ transactions are not needed
- 15. Accessing data NoSQL ▶ map your object hierarchy to a document ▶ (de-)serialization is easy ▶ transactions are not needed ▷ No SQL ▷ implicit schemas are tricky
- 16. Accessing data ORM ▶ is well known to developers ▶ CRUD operations are easy ▶ all business logic inside your application ▶ developers are in their comfort zone
- 17. Accessing data ORM ▶ is well known to developers ▶ CRUD operations are easy ▶ all business logic inside your application ▶ developers are in their comfort zone ▷ error prone transaction management ▷ you have to reflect your tables in your code ▷ all business logic inside your application ▷ schema changes are not easy
- 18. Accessing data Are there alternatives to ORM? Stored Procedures ▶ return/receive entity aggregates ▶ clear transaction scope ▶ more data consistency checks ▶ independent from underlying data schema
- 19. Java Application JDBC Database Tables
- 20. Java Application JDBC Stored Procedure API Database Database Tables Tables
- 21. Java Sproc Wrapper Java Application Java Application Sproc Wrapper JDBC Stored Procedure API Database Database Tables Tables
- 22. CREATE FUNCTION register_customer(p_email text, p_gender z_data.gender) RETURNS int AS $$ INSERT INTO z_data.customer (c_email, c_gender) VALUES (p_email, p_gender) RETURNING c_id $$ LANGUAGE 'sql' SECURITY DEFINER; SQL Java Sproc Wrapper
- 23. @SProcService public interface CustomerSProcService { @SProcCall int registerCustomer(@SProcParam String email, @SProcParam Gender gender); } JAVA CREATE FUNCTION register_customer(p_email text, p_gender z_data.gender) RETURNS int AS $$ INSERT INTO z_data.customer (c_email, c_gender) VALUES (p_email, p_gender) RETURNING c_id $$ LANGUAGE 'sql' SECURITY DEFINER; SQL Java Sproc Wrapper
- 24. @SProcService public interface CustomerSProcService { @SProcCall int registerCustomer(@SProcParam String email, @SProcParam Gender gender); } JAVA CREATE FUNCTION register_customer(p_email text, p_gender z_data.gender) RETURNS int AS $$ INSERT INTO z_data.customer (c_email, c_gender) VALUES (p_email, p_gender) RETURNING c_id $$ LANGUAGE 'sql' SECURITY DEFINER; SQL Java Sproc Wrapper
- 25. Java Sproc Wrapper CREATE FUNCTION find_orders(p_email text, OUT order_id int, OUT order_created timestamptz, OUT shipping_address order_address) RETURNS SETOF record AS $$ SELECT o_id, o_created, ROW(oa_street, oa_city, oa_country)::order_address FROM z_data."order" JOIN z_data.order_address ON oa_order_id = o_id JOIN z_data.customer ON c_id = o_customer_id WHERE c_email = p_email $$ LANGUAGE 'sql' SECURITY DEFINER; SQL @SProcCall List<Order> findOrders(@SProcParam String email); JAVA
- 26. Java Sproc Wrapper CREATE FUNCTION find_orders(p_email text, OUT order_id int, OUT order_created timestamptz, OUT shipping_address order_address) RETURNS SETOF record AS $$ SELECT o_id, o_created, ROW(oa_street, oa_city, oa_country)::order_address FROM z_data."order" JOIN z_data.order_address ON oa_order_id = o_id JOIN z_data.customer ON c_id = o_customer_id WHERE c_email = p_email $$ LANGUAGE 'sql' SECURITY DEFINER; SQL @SProcCall List<Order> findOrders(@SProcParam String email); JAVA
- 27. Stored Procedures for developers ▷ CRUD operations need too much code ▷ Developers have to learn SQL ▷ Developers can write bad SQL ▷ Code reviews are needed
- 28. Stored Procedures for developers ▷ CRUD operations need too much code ▷ Developers have to learn SQL ▷ Developers can write bad SQL ▷ Code reviews are needed ▶ Use-case driven ▶ Developers have to learn SQL ▶ Developers learn how to write good SQL
- 29. Horror story ▷ Never map your data manually ▷ Educate developers ▷ Educate yourself
- 30. Stored Procedure API versioning Database Tables search_path = api_v14_23, public; api_v14_23
- 31. Stored Procedure API versioning api_v14_23 api_v14_24 Database Tables search_path = api_v14_23, public;
- 32. Stored Procedure API versioning api_v14_23 api_v14_24 Database Tables search_path = api_v14_23, public; search_path = api_v14_24, public;
- 33. Stored Procedure API versioning search_path = api_v14_24, public; api_v14_23 api_v14_24 Database Tables
- 34. Stored Procedure API versioning search_path = api_v14_24, public; Database Tables api_v14_24
- 35. Stored Procedure API versioning ▶ Tests are done to the whole API version ▶ No API migrations needed ▶ Deployments are fully automated
- 36. Agenda How we ● access data ● change data models without downtimes ● shard without limits ● monitor
- 37. Easy schema changes PostgreSQL ▶ Schema changes with minimal locks with: ADD/RENAME/DROP COLUMN ADD/DROP DEFAULT VALUE ▶ CREATE/DROP INDEX CONCURRENTLY ▷ Constraints are still difficult to ALTER (will be much better in PostgreSQL 9.4)
- 38. Easy schema changes Stored Procedure API layer ▶ Can fill missing data on the fly ▶ Helps to change data structure without application noticing it
- 39. Easy schema changes ● Read and write to old structure ● Write to both structures, old and new. Try to read from new, fallback to old ● Migrate data ● Read from new, write to old and new
- 40. Easy schema changes Schema changes using SQL script files ● SQL scripts written by developers (DBDIFFs) ● registering DBDIFFs with Versioning ● should be reviewed by DB guys ● DB guys are rolling DB changes on the live system
- 41. Easy schema changes BEGIN; SELECT _v.register_patch('ZEOS-5430.order'); CREATE TABLE z_data.order_address ( oa_id int SERIAL, oa_country z_data.country, oa_city varchar(64), oa_street varchar(128), ... ); ALTER TABLE z_data."order" ADD o_shipping_address_id int REFERENCES z_data.order_address (oa_id); COMMIT; DBDIFF SQL
- 42. Easy schema changes BEGIN; SELECT _v.register_patch('ZEOS-5430.order'); i order/database/order/10_tables/10_order_address.sql ALTER TABLE z_data."order" ADD o_shipping_address_id int REFERENCES z_data.order_address (oa_id); COMMIT; DBDIFF SQL
- 43. Easy schema changes BEGIN; SELECT _v.register_patch('ZEOS-5430.order'); i order/database/order/10_tables/10_order_address.sql SET statement_timeout TO ‘3s’; ALTER TABLE z_data."order" ADD o_shipping_address_id int REFERENCES z_data.order_address (oa_id); COMMIT; DBDIFF SQL
- 46. Easy schema changes No downtime due to migrations or deployment since we use PostgreSQL
- 47. Easy schema changes One downtime due to migrations or deployment since we use PostgreSQL
- 48. Horror story ▷ Invest in staging environments ▷ Do not become a bottleneck for developers ▷ Educate developers ▷ Educate yourself
- 49. Agenda How we ● access data ● change data models without downtimes ● shard without limits ● monitor
- 50. One big database ▶ Joins between any entities ▶ Perfect for BI ▶ Simple access strategy ▶ Less machines to manage
- 51. One big database ▷ Data does not fit into memory ▷ OLTP becomes slower ▷ Longer data migration times ▷ Database maintenance tasks take longer
- 54. Sharded database ▶ Data fits into memory ▶ IO bottleneck wider ▶ OLTP is fast again ▶ Data migrations are faster ▶ Database maintenance tasks are faster
- 55. Sharded database ▷ Joins only between entities aggregates ▷ BI need more tooling ▷ Accessing data needs more tooling ▷ Managing more servers needs more tooling
- 56. Sharded database ▷ Need more tooling
- 57. Sharding without limits Java Application Sproc Wrapper Stored Procedure API Database Database Tables Tables
- 58. Sharding without limits Java Application ... Sproc Wrapper Shard 1 Shard 2 Shard 3 Shard N
- 59. @SProcCall int registerCustomer(@SProcParam @ShardKey CustomerNumber customerNumber, @SProcParam String email, @SProcParam Gender gender); JAVA Sharding with Java Sproc Wrapper @SProcCall Article getArticle(@SProcParam @ShardKey Sku sku); JAVA @SProcCall(runOnAllShards = true, parallel = true) List<Order> findOrders(@SProcParam String email); JAVA
- 60. Sharding with Java Sproc Wrapper Entity lookup strategies ● search on all shards (in parallel) ● hash lookups ● unique shard aware ID ○ Virtual Shard IDs (pre-sharding)
- 61. Agenda How we ● access data ● change data models without downtimes ● shard without limits ● monitor
- 62. Monitoring
- 63. pg_view
- 64. Monitoring ● Nagios/Icinga was replaced by ZMON2 ● Dedicated 24x7 monitoring team ● Custom monitoring infrastructure ZMON2
- 65. PGObserver
- 66. PGObserver
- 67. Links SProcWrapper – Java library for stored procedure access github.com/zalando/java-sproc-wrapper PGObserver – monitoring web tool for PostgreSQL github.com/zalando/PGObserver pg_view – top-like command line activity monitor github.com/zalando/pg_view
- 68. Thank you!