Matthias Kricke_Martin Grimmer_Michael Schmeißer - Building a real time Tweet map with Flink in six weeks
1 like881 views
The document outlines the development of a real-time tweet map, Ostmap, using Apache Flink over six weeks to demonstrate big data technologies to students and potential customers. It details the development milestones, including stream and batch processing techniques, indexing methods, and the application of geospatial data to visualize geotagged tweets. The project aims to enhance trust in big data technologies while also serving educational and industrial collaboration purposes.
![Geotemporal Index for OSTMap
Geo index – querying?
9z db dc df dg
9x d8 d9 dd de
9r d2 d3 d6 d7
9p d0 d1 d4 d5
3z 6b 6c 6f 6g
partitioned by geohash
bounding
box
calculate
coverage of
bounding box
range: [9p]
calculate scan
ranges from
coverage
range: [9r]
range:
[d0,d1,d2,d3]
…
3z
6b
6c
6f
6q
9p
9r
9x
9z
d0
d1
d2
d3
d4
d5
d6
…
dg
accumulo
iteratorsaccumulo
iterators
accumulo
iterators
result
Row CF CQ
geohash RawTweetKey lat/lon](https://image.slidesharecdn.com/matthiaskrickemartingrimmermichaelschmeier-buildingarealtimetweetmapwithflinkinsixweeks-160919123928/85/Matthias-Kricke_Martin-Grimmer_Michael-Schmeisser-Building-a-real-time-Tweet-map-with-Flink-in-six-weeks-17-320.jpg)
![Martin Grimmer grimmer[at]informatik.uni-leipzig.de
Matthias Kricke kricke[at]informatik.uni-leipzig.de
www.mgm-tp.comwww.scads.de
Thank you
Michael Schmeißer michael.schmeisser[at]mgm-tp.com](https://image.slidesharecdn.com/matthiaskrickemartingrimmermichaelschmeier-buildingarealtimetweetmapwithflinkinsixweeks-160919123928/85/Matthias-Kricke_Martin-Grimmer_Michael-Schmeisser-Building-a-real-time-Tweet-map-with-Flink-in-six-weeks-22-320.jpg)
Matthias Kricke_Martin Grimmer_Michael Schmeißer - Building a real time Tweet map with Flink in six weeks
- 1. Building a real time Tweet map with Flink in six weeks OSTMap Fast poc development with flink
- 2. Proof of concept - an important tool in the industry • PoC often necessary to show feasibility to customers • touch several topics: • Scalability • Stream processing • Batch processing • Storage and querying of data • OSTMap as example PoC
- 3. Goals for OSTMap • Increase trust into big data technologies on customer side • It is easy to build an application with current technologies • With almost no experience • Teach students big data technologies • Recruiting • Bring big data to the university • Build a real time application to view recent geotagged tweets on a map • Search for terms and users, show these tweets on a map • Analytics: • First data science jobs • …
- 4. Industry in practice: IT-Ringvorlesung 2016 • A course at the University of Leipzig. • work on projects of local companies • six students • over a period of 6 weeks - no full time invest • Weekly meetings • Github project: github.com/IIDP/OSTMap Nico Graebling Vincent Märkl Hans Dieter Pogrzeba Christopher SchottChristopher Rost Kevin Shrestha Michael Schmeißer Martin Grimmer Matthias Kricke OSTMap
- 5. mgm technology partners We bring applications into production! • Innovative software solution provider with application responsibility • Specialist for highly scalable, transactional online applications • Central lines of business: Insurance, E-Commerce, E-Government • Founded in 1994 • 347 employees, 9 offices (2014) • Revenue: 43,7 Mio € (2014) • Part of Allgeier SE
- 6. ScaDS Competence center for scalable data services and solutions Dresden/Leipzig • bundled Big Data research expertise of the TU Dresden and Leipzig University • Drive Big Data innovations • Bring industry and science together • Knowledge exchange and transfer
- 7. Walking skeleton “A Walking Skeleton is a tiny implementation of the system that performs a small end-to- end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.” - Alistair Cockburn gif from http://blog.codeclimate.com/blog/2014/03/20/kickstart-your-next-project-with-a- walking-skeleton
- 8. Milestone 1 read stream, store data as json file, show tweets, read data from json files
- 9. Milestone 2 write to and read from accumulo, show tweets on map, full table scans, slow visualization
- 10. Milestone 3 Term index, geotemporal index, ui improvements, clustering, …
- 11. OSTMap – stream, batch, storage and querying geotagged tweets webservice a) stream processing b) batch processing c) querying data
- 12. Stream processing of incoming data – first version GeoTweetSourc e KeyGeneration RawTweetSinkDateExtraction This enabled us to build a slow term search and a slow map search via full table scans. time index data for
- 13. Stream processing of incoming data – final version TermIndexSink GeoTweetSourc e KeyGeneration RawTweetSinkDateExtraction Now we were able to build a faster term and map search and language frequency visualization. time index TermExtraction (tokenizing) UserExtraction LanguageFrequ encySink Language Extraction term index language statistics GeoTemporalInd exCreation GeoTemporalInd exSink geotemporal index data for 1 minute window sum by language
- 14. Batch processing • Initial creation of the term index and geotemporal index for already processed tweets • Data export • Other statistics like: • Area/ tweet distance a user covers with his tweets
- 15. Storage Table Row Column Family Column Qualifier Value RawTweetData (TimeIndex) timestamp, hash 8b + 4b - - raw tweet json TermIndex term field (user,text) RawTweetData key 12b - LanguageFrequency time bucket YYYYMMDDhhmm language-tag - tweet count 4b Accumulo table design
- 16. Geotemporal Index for OSTMap Geo index geo data geohashes used as row keys in accumulo … 3z 6b 6c 6f 6q 9p 9r 9x 9z d0 d1 d2 d3 d4 d5 d6 … dg 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g partitioned by geohash (z curve) function from 2d coordinate space to 1d key space Row CF CQ geohash RawTweetKey -
- 17. Geotemporal Index for OSTMap Geo index – querying? 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g partitioned by geohash bounding box calculate coverage of bounding box range: [9p] calculate scan ranges from coverage range: [9r] range: [d0,d1,d2,d3] … 3z 6b 6c 6f 6q 9p 9r 9x 9z d0 d1 d2 d3 d4 d5 d6 … dg accumulo iteratorsaccumulo iterators accumulo iterators result Row CF CQ geohash RawTweetKey lat/lon
- 18. 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g Geotemporal Index for OSTMap Add some time! 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g partitioned by geohash, with timebuckets … 13z 16b 16c 16f 16q 19p 19r 19x 19z 1d0 1d1 1d2 1d3 1d4 1d5 1d6 … 1dg day lon lat … 23z 26b 26c 26f 26q 29p 29r 29x 29z 2d0 2d1 2d2 2d3 2d4 2d5 2d6 … 2dg … Row CF CQ day, geohash RawTweetKey lat/lon day 1 day 2 day i …
- 19. 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g Geotemporal Index for OSTMap What about Hotspots? 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g partitioned by geohash, with timebuckets … 13z 16b 16c 16f 16q 19p 19r 19x 19z 1d0 1d1 1d2 1d3 1d4 1d5 1d6 … 1dg day lon lat … 23z 26b 26c 26f 26q 29p 29r 29x 29z 2d0 2d1 2d2 2d3 2d4 2d5 2d6 … 2dg … Row CF CQ day, geohash RawTweetKey lat/lon
- 20. 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g Geotemporal Index for OSTMap What about Hotspots? 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g partitioned by geohash, with timebuckets day lon lat … 12d2 12d3 12d4 … … Row CF CQ sb, day, geohash RawTweetKey lat/lon … 11d2 11d3 11d4 … … 02d2 02d3 02d4 … … … 01d2 01d3 01d4 … … 22d2 22d3 22d4 … … … 21d2 21d3 21d4 … … spreading byte node 0 node 1 node 2 node n • spreading byte = hash(tweet) % 255 • reproducable • pre table splits in accumulo
- 21. demo
- 22. Martin Grimmer grimmer[at]informatik.uni-leipzig.de Matthias Kricke kricke[at]informatik.uni-leipzig.de www.mgm-tp.comwww.scads.de Thank you Michael Schmeißer michael.schmeisser[at]mgm-tp.com
Editor's Notes
- #9: 8