Real-time location tracking with FME Server Automations
0 likes311 views
FME Server can be used to automate data integration workflows by processing continuous data streams and triggering actions in response to events. A demo showed capturing real-time location data from mobile devices using MQTT, map matching the data to calculate bike-to-work incentives, and sending push notifications. FME Server Automations allow defining triggers that start workspaces when new data arrives through protocols like JMS or S3, and actions like running other workspaces. This enables event-driven processing of thousands of messages per second.
Real-time location tracking with FME Server Automations
- 1. Real-time Location Tracking with FME Server Automations Stijn Goedertier, GIM #FMEWT Leuven, 23 May 2019
- 2. FME Server Run FME workspace on a server to automate data integration: ● Set up schedules and notifications ● Deliver self-service data access ● Set up data collection apps ● Receive messages and trigger actions
- 3. Concepts in this talk ● FME Server stream processing: process a stream of messages when they become available ● FME Server Automations: respond to events and trigger actions
- 4. Demo: bike-to-work incentives In Flanders: a tax-free benefit of max. 0,23€ per kilometer ● Capture real-time location data from mobile devices ● Detect arrivals at home / in the office ● Map match track and calculate benefits ● Send encouraging push message
- 5. Demo: bike-to-work incentives MQTT (Message Queuing Telemetry Transport): IoT messaging protocol JMS (Java Message Service: Java messaging standard ) Use MQTT to exchange messages with Internet of Things devices! Mobile App MQTT MQTT JMS WebSockets
- 6. Challenge “We want to process a continuous stream of thousands of messages per second.”
- 7. Continually send and receive messages •TCPIPSender •TCPIPReceiver •JMSSender •JMSReceiver •SQSSender •SQSReceiver •WebSocketReceiver •WebSocketSender •PythonCreator •PythonCaller •…
- 8. Demo: receiving MQQT message streams
- 9. Demo: calculate distance between adjacent features
- 10. Demo: detect event and send JMS message
- 11. Demo: what we have so far ● Sparse points with timestamps ● Arrival events ● BUT: traveled distance is not accurate enough
- 12. Demo: what we need: map matching ● Map matching: derive the most plausible route ● Using: Graphhopper and OpenStreet- Map
- 13. Challenge “We want to watch for data and run an FME task whenever new data arrives.”
- 14. AutomationsA simple concept consisting of Triggers and Actions
- 15. Watching for Data Trigger when data arrives. Amazon S3 Watch Triggers when data arrives in location in S3 Bucket Directory Watch Triggers when data arrives in File Directory. FTP Watch Triggers when data arrives at location on ftp site Dropbox Watch Triggers when data arrives in Dropbox Directory. Azure Event Grid Can Trigger when data arrives in Azure Storage
- 17. When Trigger Occurs then Perform Action Every Automation has the same pattern:
- 18. Demo: on arrival (JMS message), start workspace
- 19. Demo: Map matching and benefit calculation
- 20. Demo: Map matching and benefit calculation
- 21. Demo: Map matching and benefit calculation
- 22. Demo: Map matching and benefit calculation
- 23. Demo: Map matching and benefit calculation
- 24. Demo: Map matching and benefit calculation
- 25. Demo: Query FME Server over WebSockets
- 26. Demo: Query FME Server over WebSockets
- 27. Conclusion: message/event-driven data integration ● FME Server stream processing: process a stream of messages when they become available. Thousands of messages per second. A workspace keeps running. ● FME Server Automations: respond to events and trigger actions. Orchestrate workspaces to run in parallel or sequentially.