"Building data streams" Константин Евтеев (Avito)
- 1. Strictly ConfidentialStrictly Confidential Building data streams Константин Евтеев kevteev@avito.ru
- 2. 2Strictly Confidential 2Strictly Confidential 2 CRUD read DBcreate update delete read read read read read
- 3. 3Strictly Confidential 3Strictly Confidential 3 1 Оптимизируем процедуры выборки 2 Вертикальное и функциональное масштабирование 3 Вводим в бой стендбаи бинарной репликации 4 Денормализуем данные для чтения 5 Шардирование 6 Денормализованные данные выносим на отдельные машины Optimization steps
- 4. 4Strictly Confidential 4Strictly Confidential 4 C(R)UD & R read DBcreate update delete read read read read read DB-SB
- 5. 5Strictly Confidential 5Strictly Confidential 5 movies id name length_minutes rating_id movie_showtimes id movie_id theatre_id room start_time theatres id name phone_number purchased_tickets id confirmation_code purchase_price_cents auditoriums theatre_id room seats_available orders confirmation_code movie_showtime_id movie_id theatre_id auditorium_id room start_time Денормализуем данные для чтения mat_view movie_name movie_rating_id movie_length_minutes movie_id, theatre_id, room, start_time, theatre_name, seats_availble ... http://www.slideshare.net/pavlushko/sphinx-10460333 http://www.pgcon.org/2008/schedule/attachments/64_BSDCan2008-MaterializedViews-paper.pdf http://www.pgcon.org/2008/schedule/attachments/63_BSDCan2008-MaterializedViews.pdf
- 6. 6Strictly Confidential 6Strictly Confidential 6 It looks like 1 Command and Query Responsibility Segregation (CQRS) или Command-query separation (CQS) 2 Event Sourcing 3 Eventual consistency 4 Lambda Architecture 5 Kappa Architecture http://lambda-architecture.net/ http://milinda.pathirage.org/kappa-architecture.com/
- 7. 7Strictly Confidential 7Strictly Confidential 7 DB CUD (CRUD) mv1 Field1 - fieldN DB-SB R DBN- CUD (CRUD) DB-SB R mvN Field1 - fieldN DB-SB Rmv1 Field1 - fieldN DB-RE-01*2 mv2 Field1 - fieldN mv1 Field1 - fieldN DB-RE-N*2 mv6 Field1 - fieldN mv3 Field1 - fieldN DBN-SB R DBN-SB R node1 node2 node3 node3 PL/Proxy cluster dwh Data streams
- 8. 8Strictly Confidential 8Strictly Confidential 8 Выгрузка по времени + работает − есть вопросы
- 9. 9Strictly Confidential 9Strictly Confidential 9 Выгрузка данных batch-ами Источник Буфер ПриемникТранспорт транспорт #pg_current_xlog_insert_location() #Get current transaction log insert location master_pos=$(psql_ping_db c "select force_wal(pg_current_xlog_insert_location())") #pg_xlog_location_diff(location text, location text) numeric #Calculate the difference between two transaction log locations psql c "select pg_xlog_location_diff(pg_last_xlog_replay_location(), '${master_pos}')") https://github.com/eshkinkot/pgday2016/tree/master/ping_DB
- 10. 1Strictly Confidential 1Strictly Confidential 10 Ticker & pgq 2008 год! https://www.pgcon.org/2008/schedule/attachments/55_pgq.pdf https://www.postgresql.org/docs/9.4/static/functions-info.html#FUNCTIONS- TXID-SNAPSHOT https://github.com/markokr/skytools/tree/skytools_2_1_stable/python https://github.com/markokr/skytools/blob/skytools_2_1_stable/sql/pgq/functions/pg q.ticker.sql http://www.pgcon.org/2009/schedule/attachments/91_pgq.pdf
- 11. 1Strictly Confidential 1Strictly Confidential 11 MVCC (https://www.postgresql.org/docs/8.3/static/release-8-3.html) Add several txid_*() functions to query active transaction IDs (Jan) This is useful for various replication solutions.
- 12. 1Strictly Confidential 1Strictly Confidential 12 How to select txids that are between snapshots xmin1 xmax1 xmax2 xmax1 xmin2
- 13. 1Strictly Confidential 1Strictly Confidential 13 Все изменения данных под единой блокировкой объекта(row level) Item sCategory params values moderatio n doublets Fee packages … User sContact informatio n shops, company Afraud payments … User lock Item lock Payme nt data Items data Users data Под блокировкой понимаем цепочку блокировок: select item_id from items for update;
- 14. 1Strictly Confidential 1Strictly Confidential 14 Сессионные переменные “signal” 1. Достаточно дешево по ресурсам 2. Не нужно разбирать цепочку вызовов процедур, и добавлять входные/выходные параметры Достоинства:
- 15. 1Strictly Confidential 1Strictly Confidential 15 Сессионные переменные, групповые действия, подводные камни … В 1 транзакции может меняться несколько объектов, а событие относится не ко всем. Решение:Используем массив по unique key array ‘{}’
- 16. 1Strictly Confidential 1Strictly Confidential 16 Init remote mv or counter https://www.depesz.com/2016/06/14/incrementing-counters-in-database/ 1) Cоздать временную таблицу на стороне подписчика 2) Создать принимающую процедуру ● временную для инициализации ● реальную 3)Начинаем слать события 4)Запускаем инициализацию со стендбая(дождаться прихода события, с которого начали заполнять временную таблицу) 5)Переключаем на реальную процедуру приема 6)Пересчитываем под блокировкой через очередь
- 17. 1Strictly Confidential 1Strictly Confidential 17 Init remote mv or counter items item_id user_id category_id last_update_txtime CONSTRAINT TRIGGER items_delta AFTER INSERT OR UPDATE OR DELETE providerprovider subscriber pgq user_items_cnt user_id category_id cnt date tmp_user_cnt user_id 1 save data for recount after init 2 copy 3 switch accept to real table https://github.com/eshkinkot/pgday2016/tree/master/remote_cnt
- 18. 1Strictly Confidential 1Strictly Confidential 18 Table and trigger function on provider's side
- 19. 1Strictly Confidential 1Strictly Confidential 19 Tables on subscriber's side Accept function
- 20. 2Strictly Confidential 2Strictly Confidential 20 Provider init function
- 21. 2Strictly Confidential 2Strictly Confidential 21 Subscriber's init functions
- 22. 2Strictly Confidential 2Strictly Confidential 22 Maintenance
- 23. 2Strictly Confidential 2Strictly Confidential 23 Real time sphinx index Main DB mat_view_index Id field1 …. filedN DB-RE-IND mat_view_index Id field1 …. filedN sphinx londiste Select full reindex pgq pgq consumer
- 24. 2Strictly Confidential 2Strictly Confidential 24 Persistent queue table1 Id field1 …. filedN pgq Table1 1 Id field1 …. filedN tick_id call_id Table1 n Id field1 …. filedN tick_id call_id Lock free rotation complex object tbl 1 Id field1 …. filedN tick_id call_id complex object tbl n Id field1 …. filedN tick_id call_id Lock free rotation Complex object part 1 Id field1 …. Complex object part 1 Id field1 …. Complex object part 1 Id field1 …. deferred trg deferred trg
- 25. 2Strictly Confidential 2Strictly Confidential 25 Особенности согласования данных Eventual consistency 1 Read with ver num master Avito_delta Client Avito_delta reserv 3 (if 2) write due to delay possible in pgq pgq 2 consumers 2 write with check version
- 26. 2Strictly Confidential 2Strictly Confidential 26 Восстановление после аварий provider sb subscriber provider undo provider subscriber subscriber subscriber 1 crash 3 apply undo 2 promote 4 start consumer 3 copy 1 crash 2 stop consumer 4 Add new consumer 5 start consumer
- 27. White Gardens Business Center, 7 Lesnaya street, Moscow, 125047, www.avito.ruWhite Gardens Business Center, 7 Lesnaya street, Moscow, 125047, www.avito.ru Thank you for your attention! White Gardens Business Center, 7 Lesnaya street, Moscow, 125047, www.avito.ru Спасибо за внимание! Константин Евтеев kevteev@avito.ru https://hh.ru/vacancy/10795267 https://hh.ru/vacancy/11463461