Data Science at Scale @ barricade.io
0 likes1,191 views
The document discusses the challenges of data science, emphasizing the need for data hacking, analysis, and expertise while maintaining focus on purpose. It outlines a pragmatic approach involving questioning, analyzing data, and sharing findings within a collaborative team structure, particularly in the context of fraud prevention. The author highlights the complexities of scaling data science at barricade.io, mentioning the use of lambda architecture and distributed messaging systems for effective data handling.
Data Science at Scale @ barricade.io
- 1. Data Science @ Scale
- 2. @davidcoallier Part of an amazing team at Barricade.io
- 7. Got all? Having the three is real hard!
- 9. Is that it? Well don’t forget your purpose.
- 10. You are not an economist. ɪˈkɒnəmɪst/: Someone with all the answers, and none of the questions.
- 12. Find a question.
- 13. Use the data you have
- 14. Features & Tests
- 15. Analyse Results You will be sad.
- 16. Conversate Talk about your findings.
- 17. Good Chats Imply egoless and collaborative data scientists.
- 18. Recap.
- 19. 1. Hacking 2. Maths & Stats 3. Expertise
- 20. And
- 21. 1. Question 2. Be Pragmatic 3. Features 4. Analyse 5. Share.
- 22. A team! Rarely a single-person effort.
- 23. An Example Fraud Prevention — Business Prevention
- 24. I knew better. Obviously… duh
- 25. We didn’t share. Science has historically been shared.
- 27. Empathise. Use human language, not lingo.
- 30. Doing this at scale is hard.
- 31. We’re still small About a billion data points a day.
- 32. Humble Beginnings Typically… an Queue and an API.
- 33. This had issues. Hard to scale, hard to decouple, etc.
- 39. Speed Layer
- 41. Batch Layer
- 43. Speed Layer: U new behaviour from new data Batch Layer: All classified behaviour since T
- 44. Serving Layer
- 46. Speed Layer: U new behaviour from new data Batch Layer: All classified behaviour since T Serve Layer: Batch layer U Speed Layer
- 47. Cache Layer
- 49. On Amazon AWS
- 53. Kafka Queue. Distributed messaging system Append-only log Consumers have offsets Partition for parallelism Replicate for redundancy Message order guaranteed, per-partition
- 61. Questions?