Datageeks
2 likes385 views
This document discusses plans to build an in-house user behavior tracking system to replace Google Analytics. It would track users across multiple platforms, store data in HBase with 1.5 billion entries, and have response times under 20ms. Data would be collected from nginx logs via Flume and written to HBase. The row key design and use of inverse timestamps are discussed to enable fast lookups by user or timespan. Cross-domain tracking techniques like cookies, fingerprinting, and HTML5 storage are also mentioned. The document touches on batch processing of the stored data and using a recommendation engine called Myrrix.
Datageeks
- 1. User Behaviour Tracking Track - Store - Process ! //Florian Pfeiffer - Head of Data&Infrastructure - gutefrage.net !
- 3. Vision „Let’s build our own Google Analytics“
- 4. Why analytics does sampling we want the (raw) data
- 5. Ideas,Thoughts&Goals fast / minimal impact on page loading time high availability track user over multiple platforms storage engine? -> hbase
- 7. Numbers! 10-20ms Response Time per pixel record for now: ~2500 concurrent reqs 1,5 billion entries in Hbase 10 Nodes in Hadoop Cluster
- 8. Serving Infrastructure Loadbalancers & RR DNS nginx with empty_gif module (~2ms) data is written to logfile
- 9. Storing Infrastructure every nginx node has flume-ng flume ingests logfile AsyncHBaseSink with custom Serializer direct writes to HBase
- 10. why flume? we had it already in production ;) Storm might be an interesting alternative
- 12. Why? You can scan through all data and use filters for selecting specific data But scanning with start & stop row speeds things up (a lot)
- 13. HBase rowkey design Do I need a fast user or a fast timespan lookup? User - clientid,ts<,connectionId> Timespan - ts,clientid<,connectionId>
- 14. Inverse Timestamps Data in HBase is stored lexicographicaly sorted Normal TS - scan would yield oldest results first Inverse TS - newer entries come first (and you can cancel the scan if you have enough data)
- 16. The olden times… or Cookies Easy to drop a 3rd party cookie with userId on different websites Gets more and more blocked (Safari, FF..)
- 17. Fingerprinting Yields interesting results on desktop, difficult on e.g. iPhone invisible to user Last resort if everything else fails?
- 18. Etag Quite new, based on browser cache sounds interesting
- 19. HTML5 Storage Store data in local HTML5 storage Retrieve data with Cross Domain Messaging
- 20. Store data e.g. UserId, SessionId, GeoIP, URL, action, data
- 21. Batch Processing Calculate how many users are active on platform A and also on B Get Traffic of all Questions belonging to Channel X sorted by Country
- 22. Now to something completely different…
- 23. demo
- 25. Myrrix Evolution: taste -> mahout -> myrrix (-> oryx) Recommender based on ALS
- 26. Recommendations @ GF.net User emit signals on questions view, like, gives answer, answer is voted best Application sends signals through RabbitMQ to recommendation servers
- 27. YEAH but what happens, when a new user signs up?
- 28. ?
- 29. Fetch data from tracking and feed it into myrrix
- 30. Collecting&Storing data works great using & processing is another thing ;)