Where in the world is Franz Kafka? | Will LaForest, Confluent
0 likes406 views
The document discusses geospatial capabilities in stream processing using ksqlDB, focusing on functions for handling geographical data such as creating streams, calculating geohashes, and performing intersection and containment operations. It includes examples of SQL commands for processing geospatial data, creating alerts, and generating heat maps. Additionally, it outlines future enhancements for spatial functions and integrations with other geospatial libraries.
![Shape Encodings
10
GeoJSON Geometries
{
"type": "Polygon",
"coordinates": [
[[49.4340992,7.7697603],
[49.4391228,7.7707383],
[49.4395496,7.7695902],
[49.4351295,7.7679744],
[49.4340992,7.7697603]]
]
}
Spatial4j does not support FeatureCollection
WKT
POLYGON((-77.25595325282619
38.90049514948347,-77.25389331630275
38.90503717840573,-77.289598882709
38.91171610485367,-77.2760376339297
38.889539649049034,-77.25595325282619
38.90049514948347))](https://image.slidesharecdn.com/willlaforest-210927175229/85/Where-in-the-world-is-Franz-Kafka-Will-LaForest-Confluent-10-320.jpg)
Where in the world is Franz Kafka? | Will LaForest, Confluent
- 1. Geospatial in the streams Where in the World is Franz Kafka? Will LaForest Public Sector CTO will@confluent.io
- 3. 3 Geospatial based conditions ETL Steaming geo analytics
- 4. 4 ksqlDB: Streaming SQL Engine for Apache Kafka ksql > CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = ‘Platinum’ EMIT CHANGES; Develop real-time stream processing apps writing only SQL! No Java, Python, or other boilerplate to wrap around it
- 5. 5 geo_distance (lat1, lon1, lat2, lon2, units)
- 7. Geospatial Functions 7 KSQLGeo functions - https://github.com/wlaforest/KSQLGeo geo_contained(lat, lon, shape), geo_contained(shape1, shape2), Determine whether the point provided in contained in the shape. geo_intersected(shape1, shape2) Determine if two shapes intersect anywhere. Spherical and non-spherical geo_area(shape) Calculate the area of a shape in squared degrees geo_hash(lat,lon), geo_hash(lat,lon, precision) Calculate the geohash of a point. Default precisions is 12 geo_covering_geohashes(shape),geo_covering_geohashes(shape,precision) Table function which returns a row for each geohash that a the shape falls into
- 8. Representing Geometry in ksqlDB - Points CREATE STREAM BUS_RAW (VehicleID VARCHAR, Lat DOUBLE, Lon DOUBLE, TripID VARCHAR, RouteID VARCHAR, DirectionNum BIGINT, TripHeadSign VARCHAR, TripEndTime VARCHA) WITH (KAFKA_TOPIC='bus_raw', VALUE_FORMAT='JSON') 8
- 9. Representing Geometry in ksqlDB - Shapes CREATE STREAM FENCE_RAW (ROWKEY VARCHAR KEY, type VARCHAR, "properties" MAP<VARCHAR, VARCHAR>, geometry MAP<VARCHAR, VARCHAR>, geoshape VARCHAR) WITH (kafka_topic='fence_raw', value_format='JSON', PARTITIONS=1); 9
- 10. Shape Encodings 10 GeoJSON Geometries { "type": "Polygon", "coordinates": [ [[49.4340992,7.7697603], [49.4391228,7.7707383], [49.4395496,7.7695902], [49.4351295,7.7679744], [49.4340992,7.7697603]] ] } Spatial4j does not support FeatureCollection WKT POLYGON((-77.25595325282619 38.90049514948347,-77.25389331630275 38.90503717840573,-77.289598882709 38.91171610485367,-77.2760376339297 38.889539649049034,-77.25595325282619 38.90049514948347))
- 11. 11 Functions
- 12. Geohash • Base32 encoded • A hash represents a bin in a grid that covers the earth • The number and size of the bin depends on the precision 12
- 13. Geohash for Keying & Partitioning geo_hash(lat,lon) geo_hash(lat,lon, precision) 13 CREATE STREAM BUS AS SELECT DTIME, as_value(geo_geohash(Lat,Lon,5)) GEOHASH, * FROM BUS_RAW b PARTITION BY geo_geohash(Lat,Lon,5) EMIT CHANGES;
- 14. Geohash for Binned Aggregates geo_hash(lat,lon) geo_hash(lat,lon, precision) 14 CREATE TABLE geo_heat_map AS SELECT windowstart ws, windowend we, GEOHASH COUNT(*) total FROM bus WINDOW HOPPING (SIZE 30 SECONDS, ADVANCE BY 10 SECONDS) GROUP BY GEOHASH EMIT CHANGES;
- 15. Intersection and Containment geo_contained(lat, lon, shape) geo_contained(shape1, shape2) geo_intersected(shape1, shape2) 15 CREATE STREAM ALERT AS SELECT b.VEHICLEID, b.Lat, b.Lon, f.GEOHASH FROM BUS b INNER JOIN FENCE f WITHIN 7 days ON b.GEOHASH = f.GEOHASH WHERE GEO_CONTAINED(b.Lat, b.Lon, f.geoshape) EMIT CHANGES; https://docs.ksqldb.io/en/latest/developer-guide/joins/partition-data/
- 16. Shape Partitioning geo_covering_geohashes(shape), geo_covering_geohashes(shape, precision) 16 CREATE STREAM FENCE_PRE_PARTITION AS SELECT GEO_COVERING_GEOHASHES(_raw_data, 5) GEOHASH, * FROM FENCE_RAW; CREATE STREAM FENCE AS SELECT * FROM FENCE_PRE_PARTITION PARTITION BY GEOHASH;
- 17. Shape Area geo_area(shape) 17 CREATE TABLE WEATHER_SYSTEM_AREA AS SELECT windowstart ws, windowend we, AVG(geo_area(_RAW_DATA)) AVG_AREA FROM WEATHER_SYSTEM WINDOW HOPPING (SIZE 1 HOUR, ADVANCE BY 10 MINUTES) GROUP BY weather_type EMIT CHANGES;
- 18. Configuration - ksql.extension.dir - ksql.functions._global_.spatial4j.geo - ksql.functions._global_.spatial4j.normWrapLongitude 18
- 20. Demonstration
- 21. Currently Planned Enhancements and Work 21 • Shape buffering • Compute and return a shape with a buffer around it • Particularly useful with lines and paths • Shape center • Find the center point for a shape. • Testing and Optimization • Additional shape and point encodings • PostGIS for backlog