Building a modern in-house analytics pipeline
- 1. Changing the way cities move Modern in-house analytics pipeline Sergey Burkov R&D Engineering Lead DATA STRATEGY:
- 2. 2 1. About Mobimeo 2. Importance of analytics 3. Buy or build choice 4. Vision vs. Reality 5. Scoping the requirments Agenda 6. Choice of tech stack 7. Implementation details 8. Big picture 9. Challenges
- 3. Mobimeo – Changing the way cities move Easy access to daily mobility Our technology empowers mobility providers to orchestrate existing and new modes of public transport. Together we create an effortless transport experience to make mobility service attractive to millions of users. More mobility. Less traffic. 3
- 4. 4 We help our mobility partners to boost customer happiness Develop white label products - together with our mobility partners Fast implementation of tailor-made solutions in the apps of our partners Continuous improvement of products – our customers are always the focal point Selection of current mobility partners
- 5. We are rethinking urban mobility 5 Personalized information Public transport connected with multimodal offers Step by step navigation Easy ticket purchase
- 6. Deloitte: “As organization investment in data modernization increases, value grows exponentially and changes from hindsight to insights to foresight.” Gartner Research: “Data and analytics leaders should augment or upgrade traditional BI platforms to modern platforms that improve business value and speed time to insight.” Data Strategy: BI and Analytics Is Not Just a Tool 6
- 7. Outsource Data Strategy: Buy or build In house + Speed of launch + Advanced UI reporting + Easy integration into app + Less maintenance overhead - Costs based on data/MAU volume - Tailored to APP analytics mainly - Privacy/GDPR concerns - Code libs dependency + Full privacy transparency + Flexibility + Speed of transition from insight to product + Complete understanding of user context - Steep implementation curve - Maintenance work - Time to launch 7
- 8. Data Strategy: Ideal analytics/data pipeline 8 Low Event Latency Scalability Interactive Querying Versioning Monitoring Testing
- 9. • No single owner for the system • Different flavours of events definitions between platforms and apps • Inconsistent names, non-heterogeneous structure • No strict schema, data sent as raw JSON (some > 500KB) • JSON BLOB as a field value • Data stored in S3 in one bucket as individual json files • … • Data discovery and query building is complicated and fragile process • New business event addition is a cross teams full blown project Data Strategy: starting point & reality 9
- 11. 11
- 12. • Ease of adoption • Extensibility • Re-usability • Streaming & Batch • Strict schema enforcement • Schema evolution • Multiple schema versions • Boiler Code generation Data Strategy: where do we want to be • Transparency of execution • Handling duplicates • Data flow monitoring • Robustness for recovery • Backfilling • Self-service • External data sources • Smoke tests 12
- 13. 13
- 14. Data Strategy: distributed streaming platform 14
- 15. @namespace("analytics.mobile") protocol ActiveTicketClickProtocol { import idl "../common/AnalyticsCommon.avdl"; import idl "../common/EventPropertiesCommon.avdl"; record ActiveTicketClickProperties { analytics.common.GenericClickEventProperties @inlineFields(true) clickEvent; } @register(true) record ActiveTicketClickEvent { analytics.common.CommonEventProperties @inlineFields(true) commonEventProperties; ActiveTicketClickProperties eventProperties; analytics.common.FrontendDeviceProperties deviceProperties; analytics.common.FrontendAppProperties appProperties; } } Tactics: Schema definitions reside in central repository 15
- 16. Tactics: safe schema change process 16
- 17. Tactics: documentation is always up-to-date 17
- 18. • Helper library for Kafka with generated classes for BE • DTO classes with JSON serializers for mobile apps • Custom data connectors (REST API source, S3 Parquet sink) • JSON REST API service (resolve json against schema registry) • Replay service from raw dead letter S3 • Monitoring/Alerting: injection rate, dead letter queue Tactics: data injection 18
- 19. • Access permissions • Catalogue of data assets • Events are stored in dedicated S3 buckets • Data in Avro, Parquet and raw JSON • Partitioned by origin, event type, tenant, date, hour • Dead-letter bucket contains events rejected by pipeline Tactics: data lake organization 19
- 20. CREATE EXTERNAL TABLE `ticketclickevent_avro`( analyticsid string, timestamp bigint, eventproperties struct<loggedin:boolean,screenname:string>, PARTITIONED BY (tenant string, date date, hour int) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' WITH SERDEPROPERTIES ( 'avro.schema.literal'='{"type":"record","name":"TicketClickEvent" ...') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION 's3://.../avro/event=analytics.mobile.TicketClickEvent' MSCK REPAIR TABLE Tactics: data access 20
- 22. • Open source monitoring and configuration tools • Overcoming AVRO Schema inheritance / re-usability limitations • Parsing multiple versions of events from JSON to AVRO • Schema -> Java, Scala, Kotlin, Swift, documentation generation • Tools / libs doesn’t support latest Avro specsifications or uses incompatible forks • Splitting classes according to domains • AWS Athena / Glue limitations Challenges 22
- 23. Sergey Burkov R&D Engineering Lead sergey.burkov@mobimeo.com linkedin.com/in/burkov Mobimeo GmbH Hallesches Ufer 60 10963 Berlin info@mobimeo.com