scalable Smart aNalytic APplication builder for sentient Cities Overview -- September 2018

DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
Snap4City Overview, September 2018 (C)
https://www.disit.org/
Paolo Nesi, paolo.nesi@unifi.it
https://www.Km4City.orghttps://www.snap4City.org
scalable Smart aNalytic APplication builder for sentient Cities
Overview ‐‐ September 2018
1
Powered by

http://www.disit.org
• Snap4City has been created in response to Select4Cities PCP
(http://www.select4cities.eu/) call as an open, standardized, data‐driven,
service‐oriented, user‐centric platform enabling large‐scale co‐creation IOT/IOE
applications and services for Helsinki, Copenhagen and Antwerp.
• Snap4City is a fully open source, robust, scalable, easy to use solution, provides
tools for co‐creation of mixt data driven, stream and batch processing, extending
the powerful semantic reasoner of Km4City https://www.km4city.org, with
IOT/IOE, GDPR, and city dashboards.
– Km4City has been validated in multiple devices (PC, Android, Raspberry, ..), and
domains: mobility and transport, tourism, health, welfare, social and cities such as
Florence, Pisa, Arezzo, and large area of millions on inhabitants as Tuscany and million
of data per day.
• The innovation is mainly related to semantic reasoning, IOT interoperability,
microservices, automated dashboard production, .. thus
• setting up smart city solutions in a
Snap4City Overview, September 2018 (C) 2

Agenda
• Comparison of Snap4City with state of the art
• Services for the city users, stakeholders
• From Level 1 to developers
• Snap4City Living Lab

Comparison with
State of the Art

Confronto: lo stato dell’arte

unique about our platform vs competitors?
• flexible and IOT smartcity‐oriented dashboards, visual editor and wizard for
creating dashboards, large collection of widgets, (addressing: sensors and
actuators, selectors, maps, etc.);
• Visual tool for IOT Applications (not rule based, largely avoiding coding), large
number of MicroServices, easy to expand;
• management of any kind of data and resources (High Level Types) by using
Km4City Expert system and semantic indexing;
• providing tools for managing the Living Lab community;
• Coworking for: IOT applications, dashboards, back office processes,
microservices, external services, etc.;
• end‐2‐end encryption and security/privacy with GDPR: several data types:
My Personal Data, My Annotations, etc.;
• integrated management of referral data, historical data, traffic analyzer;

Smart City Cloud Infrastructure
Km4City Smart City API
Knowledge
Base
ETL Processes, Data Analytic, R; IOT App; etc.
Data Processing Tools
Development and Management Tools
ETL Processes
Resource
Manager
DataGate/
CKAN
Km4City Ontology
Phoenix, Hbase
+ indexing
Big Data Storage Knowledge
IoT/IoE Applications
AMMA
Linked
Open Graph
ServiceMap Data Flow Analysis
DevDash
Elastic Management of Containers
Mobile and Web Apps
Final Users’ Tools
Dashboards
Social Media
IoT/IoE
Open Data
Personal Data
Industry 4.0
GIS + Map Data
IOT / IOE Apps
IOT Directory
Management
Authentication, Authorization, GDPR, Security AssessmentPowered by

Requirements and
Objectives

Requirements and Objectives
• Serve as a City Dashboard, App User Interface, etc.
– Real time and historical data, any device, sensors and actuators
– Sensors, KPI, maps, data trends, real time data, charts, etc.
• Real Time communication:
– in/out data driven from/to the field into: applications, notifications, etc.
• Referral data:
– shadow, access (API, storage, any protocol), production of OD, export
• Data Driven processing: IOT Applications, IOT edge, multiple operating
systems, embedded systems, MicroServices
• Data Analytics: Machine Learning, statistics, reasoning, …
• Serve as Living Lab: open innovation, coworking; collaborative work;
sharing: data, processes, dashboard, experiences, solutions, ….
• Experimented on large scale cases

Non functional requirements
• Open Source and based on Open Source Tools and OS
• Open Standard for communication and API for In/Out
• Scalable, Robust, Distributed and Decoupled, modular, Service
Oriented, open to external services and data sets
• Data driven, for reading and data analytic
• Heterogeneous: any device, private and public, custom and..
• Interoperability: protocols, internal API, Smart City API, capable to
integrate with legacy conditions in place, modular, reusable,…
• Communication with things: any protocol, any format, …
• Security by Design: HTTPS, TLS, …
• User Centric Design: privacy by Design (and GDPR), personalized,
personal data management, ...

Services for City users
and Stakeholders

For the user: different levels of engagement
• Observer:
– Level 0: observers of data with private access
• Manager: Final Users
– Level 1: create Dashboards and IOT Applications
– Level 2: create Dashboards that get and produce data, act on city
– Level 3: add your own IOT Device, create Dashboards with them and city data
– Level 4: create IOT Applications to make smarter your Dashboards, services,
notifications, exploiting MicroServices
• Area Manager: Developers, Researchers, Operators
– Level 5: Developer of complex services: R, ETL, External Srv, MicroApp, web and
mobile application exploiting Smart City API…

Level 0 users

Level 0 user: Observer (public or invited)
• Access to the Portal, view Dashboards with public data
IOT and City data World City Dashboard
14

https://dashboard.km4city.org/view/index.php?iddasboard=MjY1
https://www.disit.org/dashboardSmartCity/view/index.php?iddasboard=MjI5
https://www.disit.org/dashboardSmartCity/view/index.php?iddasboard=MjQ2 https://www.disit.org/dashboardSmartCity/view/index.php?iddasboard=MjUw

https://dashboard.km4city.org/view/index.php?iddasboar
d=MTY2
https://dashboard.km4city.org/dashboardSmartCity/view/inde
x.php?iddasboard=MTE5
https://dashboard.km4city.org/view/index.php?iddasboard=MjY3 https://dashboard.km4city.org/view/index.php?iddasboard=MjQ3

https://dashboard.km4city.org/view/index.php?iddasboard=MTU1 https://dashboard.snap4city.org/dashboardSmartCity/view/index.php?iddas
board=MjY2
• Grid on the frame to simplify the sizing of dash
and widgets
• Spaces among widgets to simplify the reading
• Full set of colour, which can be used more
moderately than us 

http://www.disit.org Coverage: examples
• Search all services in the area Snap4City Overview, September 2018 (C)
Antwerp
Bologna Toscana
Sardegna
Pisa
19

Km4City in Tuscany Area
What is enabling and providing smart services
• Smart Parking, in Tuscany
• Smart First Aid in Tuscany
• Smart search for POI and public transport srv.
• Public Transportation in Tuscany
• Routing in Tuscany, simple and multimodal
• Social Media Monitoring and acting
• Traffic events and Resilience in Florence
• Bike Sharing in Pisa and Siena
• Recharge stations for e‐vehicles
• Entertainment Events in Florence
• Traffic Sensors in Tuscany
• IOT/IOE sensors and actuators
• Weather forecast/condition in Tuscany
• Pollution and Pollination in Tuscany
• People Monitoring, in Tuscany via App
• ..People Monitoring Assessment in the City, in
Florence via Wi‐Fi
All Point of Interests, cultural activities, IOT, …
Over than 1.2 Million of complex events per day!https://servicemap.km4city.org 20

Level 2 users

Level 2 users: Create their own Dashboards
• With smart city data, and
• Sensor/actuator: info/IOT/IOE, if any provided by infrastruct. (e.g., Meter)
•  create events/notifications from them
City DashboardIOT and City data World

From Data to Dashboards by Wizard
Examples:
• Official environmental data compared with temperature of the house
• Weather actual/forecast vs the consumption of energy in the company
• Number invoices and revenues vs energy
23

From Templates to Wizard and Dashboards
• From the scratch to
create a new
Dashboad
• From a Dashboard to
Add widgets and
groups of them

• Select the area of your interest: panning and zooming
• Select the
• graphic aspect of your interest, or
• High Level Type of your interest, or
• Make a search if you a have a precise idea or
• Act on filters: nature, subnature, type, name, value, date, healthi, owner, …
• Combine them as you like
• Select the lines of your interest
• Then click on Next and get the Dashboard by wizard

Filtering data to reach selection
• High Level Types: sensors, special, KPI, POI, etc.
• Nature: accommodation, mobility, financial, wine, health, etc.
– Dashboard  IOT APP, IOT Device to system
• SubNature: a subclassification of the nature
• ValueType: temperature, humidity, memory, pressure, glucose, etc…
• ValueName: name of variable of the: IOT sensor, data, etc.
• DataType: integer, text, Boolean, float, webpage
• LastDate: last date and time of data acquisition
• LastValue: the last value, an example; in some case a parameter
• Healthiness: green or red according to the data quality criteria defined
• LastCheck: date and time of the last valid data
• Ownership: Public or Private; it may be: private, private (Delegated), private
(MyOwn) Snap4City Overview, September 2018 (C) 27

Data Kinds vs Widgets: Complex events
Traffic Events
• Complex Event
• External Service
• MicroApplication
• Special Widget
• POI, Point of Interest
• KPI, Key Performance
Indicator
• Sensor
• Sensor‐Actuator
• My Personal Data
• Dashboard‐IOT App
28

Data Kinds vs Widgets: external services
• Complex Event
• Special Widget
Indicator
• Sensor
29

External Services
• Twitter Vigilance:
– daily and real time
– Volume and sentiment
analysis
• Traffic Flow
• Services on Maps
• Tracking vehicles
• Real time sensors on 3D
• Web HTML5 Applications
• User behaviour
monitoring

Data Kinds vs Widgets: MicroApplications
• Complex Event
• Special Widget
Indicator
• Sensor
31

Data Kinds vs Widgets: Special Widgets
• Complex Event
• Special Widget
Indicator
• Sensor

Data Kinds vs Widgets: POI
• Complex Event
• Special Widget
Indicator
• Sensor

Data Kinds vs Widgets: KPI
• Complex Event
• Special Widget
Indicator
• Sensor
Real Time
Event Driven
Historical Data
34

Data Kinds vs Widgets:
Sensors• Complex Event
• Special Widget
Indicator
• Sensor
Real Time
Event Driven
Historical Data
35

Data Kinds vs Widgets: KPI
• Complex Event
• Special Widget
Indicator
• Sensor

http://www.disit.org edit mode
CONTEXTUAL
MENU to
edit features

Level 3 users

Level 3 user: add personal devices and create Dash
• With Smart city data and information +
• Personal IOT/IOE, which can be registered and created
My DashboardIOT and City data World
Registration of My IOT DevicesSnap4City Overview, September 2018 (C) 40

Add IOT/IOE Devices
• From Models
• From SigFOX standard
• A range of protocols,
formats, approaches
• Create your own devices:
– Arduino,
– Raspberry,
– Android,
– LoraWAN + Arduino,
– etc.
Secure Communication:
HTTPS, TLS (K1, K2)
41

Custom IOT Edge, and IOT Devices
SigFOX
Arduino
LoraWAN + Arduino + I2C
IOT Edge NodeRED:
RaspberrySensors/
Actuators

Dash with city data and your data/actuators
• Sensors
– Values
– status
• Actuators:
– Buttons
– Dimers
– Etc.

Dashboard vs IOT Devices
Snap4City
IOT Brokers
Snap4city Platform
Towards any IOT Device
and/or DashboardFrom any IOT Device
and/or Dashboard
Managing Public and
Private IOT/IOE DevicesSensors/
Actuators

Data Kinds vs Widgets:
MyPersonalData• Complex Event
• Special Widget
Indicator
• Sensor
Real Time
Event Driven
Historical Data
45

Level 4 users

Level 4 users: dashboard with intelligence App
• Dashboards with IOT Applications for enforcing smart and
intelligence into them.
DashboardsIOT and City data World IOT Applications
My IOT Devices
Dashboard‐IOT App
Applications

City Dashboard + IOT App
Control Room Operator
Would like to:
‐ Monitor traffic flow,
Environment, Car parking,
Cycling, First aid, temp., ..
‐ Act and monitor Dynamic
Plates
‐ Act and monitor red lights
Driver, Policeman
Would like to:
‐ Monitor traffic,
Parking, env., speed
limit, …
‐ Act and monitor red
lights

IOT Application with City Dash
simpler development

TC: Edge Computing
City user
Would like to:
‐ Monitor and exploit
temperature and humidity
‐ Manage sensors
‐ Perform edge computing
‐ Using these data for
multiple applications
Steps:
1. Registering the device and sensors
2. Create flow on edge device using NodeRed with Snap4City,
sending data to Broker
3. Use data from Broker on Snap4City IOT App
IOT Broker
Click
here

Reporting Critical Events
Control Room Operator
Would like to:
‐ Monitor events vs services in the
city and receive critical event
notifications from on the road
operators.
‐ Assess contextual condition,
services status
On the road operator
Would like to:
‐ Monitor traffic, Parking,
env., speed limit, services,
‐ Send critical event
notifications via coded
description

IOT Application with City Dash
simpler development

Node.js Blocks on NodeRed SotA

Snap4City MicroServices

Managing MyPersonalData in secure manner
Examples:
• 1) Social IOT: A group of friends share some data with other according to GDPR: GPS
position, Medical parameters as Glucose, etc.
• 2) saving and retrieve personal sensitive information.
The users manage their Personal data via personal mobile Dash and IOT App, and
configuration on the portal and/or Mobile App

Managing MyPersonalData in secure manner
Example:
• Piero shares some data with
selected friends according to
GDPR: GPS position
• He managed the data via
personal mobile Dashboard
and IOT Application
Encrypted
Data Storage
Smart City
Services and
IOT/IOE

• Manage Profile and MyPersonalData
• For each Data Type:
– Start as private  making them public
(anonymous) and revoke
– The Owner is the only one that can: (1) modify
values; (2) change the ownership
– Define/revoke Delegation to Access
– Delete/forget per Data Type and “me all!”.
– Auditing
GDPR Compliant

Level 5 users and above

AreaManager: Level 5, the developers
• Creation of External Services
• Management of IOT Devices, IOT edges
• Development tools, for example:
– Creation of Data Analytic, Machine Learning in: R, Tensor Flow
– Creation of massive ingestion, data transformation processes in
ETL Penthao kettle
– Creation of MicroServices for IOT Applications, from REST Call or
from Data Analytics
– Management of resources and their sharing
– Open Source with many examples
– Smart City API with Swagger & Postman documentation/test
• Many tools for monitoring and control

IOT Directory
Back Office Processes
IOT Broker
IOT Broker
IOT Broker
IOT Broker
ETL Process
Data Analytics
ETL Process
ETL Process
ETL Process
Data Analytics
Data Analytics
Data Analytics
Knowledge Base,
Km4City
Smart City API from Knowledge Base and other tools
Ontology SPARQL, FLINT LOG.disit.org
ServiceMap ServiceMap3D
Swagger MicroServices
IOT ApplicationsWeb and Mobile AppsDISCES and back office management tools
MicroApplications
Resource Manager

Developer in R Studio + Tensor Flow

Developer in ETL: Penthao Kettle

Living Lab

Connect
IOT/IOE
Upload context
Open Data
Connect external
Services
Advanced Smart
City API
Run
Applications &
Dashboard
Monitor
City Platform
Life Cycle
experiments
workshops
tutorials
networking
agreements
events
Start‐ups
Research
groups
City
Users
City Operators
Large
Industries
collaborations
Licensing,
Gold services
personal
services
Case
Studies
Inhouse
companies
Resource Operators
Tech
providers
partnerships
documentation
Help desk
Category
Associations
Corporations
Advertisers
Community
Building
subscription to
applications
Produce City
Applications &
Dashboard
Promote
Applications &
Dashboards
Produce
Applications for City
Users
Set Up: ETL & Data
Analytic algorithms
Collaborative
Platform
Early Adopters
Snap4City

IOT Sensors and Actuators
Services
Big Data
Storage
Services
Computation
Management Services
Data Analytic Services
ETL Process Services
External Services
Social Media
Twitter Vigilance
Advanced Smart City
API Services
Dashboard Services
Knowledge
Services
Notification
Services
Data Mapping
Services
EventLog Services
App
Services
DashboardsDashboard Builder
Dashboard
Widgets
Application Builder
Data Processing
Development
Environment
Access
Services
Authentication and
Authorization
Support
ServiceMap
ServiceMap 3D
LOG & FLINT
IOT Brokers
IOT Discovery
IOT Directory
IOT Edge
Stream &
Batch
Indexing and Search
Service
Spark
Spark/ML
R, Python,
Java, etc.
Penthao Kettle, Karma
MapReduce
HDFS, HBase
Phoenix
DISCES
Scheduler
VM, Cluster, Cloud
Event Logger
Notificator
Data Mapper
App
Develop‐
ment Kit
Application & MS
Monitor and Analyser
RDF, Graph DB
LDAP
OAuth
NodeRed
Engagement Service
City Users Directory
MySQL
IOT devices
Personal
Device
Control Room
PC Mobile
VM,
Cluster,
Cloud
App Deployer & Res.
Manager
VM,
Cluster,
Cloud
R, Python,
Java, etc.
Data Mart
City Users Directory
Panel for
Messages
Process
Uploader
MicroService Directory
IOT Edge
Application Development
Knowledge BaseBig Data
Storage
Data Save and Select
Services
DataGate
CKAN
Kerberos
Auditing
Node.JS
Node.JS
Node.JS
Twitter
Snap4City: from tools to MicroServices
Data Annotator
Cloud Elastic
Management
IOT Authentication and
Authorization
Applications
Snap4City
Mongo
Snap4City

Data & Model
http://www.km4city.org/?controlRoom
http://www.km4city.org/?devTools
http://www.km4city.org/?app
http://www.km4city.org/?infoDocs
67

Km4City: Knowledge Base
– Street‐Guide
– Mobility and
transport
– Points of interest
– Sensors, IOT, ..
– Energy
– Administration
– Citations from
strings
– ..
– Multiple DOMAINS
– Geospatial reasoning
– Temporal reasoning
– Metadata
– Statistics
– Risk and Resilience
– Licensing
– Open and Private Data
– Static and Real time
Big Data Tools
LOD and
reasoners
Ontology Documentation:
http://www.disit.org/6506
http://www.disit.org/6461 68

Km4City  in Tuscany Area
Road Graph (Tuscany region)
132,923 Roads
389,711 Road Elements
318,160 Road Nodes
1,508,207 Street Numbers
Info on: points, paths, areas, etc.
Services (20 cat, 512 cat.)
16 Public Transport Operators
21.280 Bus stops & 1081 bus lines
Dynamic/real‐time in Tuscany Region
• Real time bus lines: 144 updates X day X line
• 1081 Transport Pub Lines:  1‐2 up per day, time‐path
• >210 parking lots status:  76 updates X day X sensor
• >796 traffic Sensors: 288 updates X day X sensor
• 285 weather area: 2 updates X day X area
• >12 hospital Triage status: 96 updates X day X FA
• 1600 Fuel stations: 1 update X day X station
• 22 Environmental data: 20 updates X day X sensor
• 39 Bike Sharing data: Pisa and Siena
• 12 Pollination data
• 140 recharging stations
• Smart benches, waste mng, irrigators, lighting,…
• Florence ent.events: about 60 new events X day
• Different kinds of Florence traffic events,
• Wi‐Fi: > 400.000 measures X day
• App mobiles: >50.000 measures X day
• more than 40.000 distinct users X day
• From 600.000 to 4.5 M Tweets X day
• many IOT ……http://servicemap.km4city.org Snap4City Overview, September 2018 (C) 69

Other Sensors and Actuators, IOT
• Restricted Traffic Zone Gates
– Passages, payment, alerts,
– Wi‐Fi control, RFI control, etc.
• Road Direction manager: panel,
red‐light, etc.
– Status and action
• Environmental Sensors:
– Air quality, pollution, rain,
allergens, temperature,
– humidity,…
• Public Light Pillar
– Traffic flows, environment,
– Wi‐Fi, Tv‐Camera, BT servers,
on/off, percentage of light, ..
• Waste Manager
– Level, kind, status, on/off
• Recharge station, column
– Free slots, consumption, next
time slot, …
• Environmental
Sensors:
– Air, temperature,
humidity,
– water level in rivers
– Status of underpass
and bridges
• Risk assessment
– Value of the
buildings,
– hydrogeological
risk map,
– earthquake risk
map, …
– people distribution
and location
– Position of recover
places,
• Traffic Zone Gates
– Passages, alerts,
– etc.

Cycling Paths

Along a Line
Into an Area

• Areas, Bus lines, bike lanes, tram, RTZ, etc.
Km4City in Firenze

Pisa
TPL &
sensors
75

Km4City in Sardegna

Smart City Control Room
http://www.km4city.org/?controlRoom

Transport
systems,
Mobility, Parking
Transport
systems,
Mobility, Parking
Sensors, IOT
Cameras, ..
Sensors, IOT
Cameras, ..
Environment,
Water, energy
Environment,
Water, energy
Shops,
services,
operators
Shops,
services,
operators
Social Media,
WiFi, Network
Social Media,
WiFi, Network
Public services,
Govern, Events
Public services,
Govern, EventsDashboards 79

Smart City Dashboard

(1)
(3a)
(2)
(3b)
(4)
(5)
(6)
(7)
(8)

(1)
(2)
(3)
(4a) (4b) (4c)
(7)
(5) (6 gas)
(6 calore)
(12)
(11)
(10)
(9)
(8)
(15)
(16)
(14)
(17)

(1)
(2a) (2b)
(3) (4)
(7) (10) (8)
(9)
(5)
(6)
(11)
(12)

Acknowledgement
• Thanks to the European Commission for founding. All slides reporting logo of
REPLICATE H2020 are representing tools and research founded by European
Commission for the REPLICATE project. REPLICATE has received funding from the
European Research Council (ERC) under the European Union's Horizon 2020
research and innovation programme (grant agreement n° 691735).
• Thanks to the MIUR for co‐fouding and to the University of Florence and
companies involved. All slides reporting logo of Sii‐Mobility are representing tools
and research founded by MIUR for the Sii‐Mobility SCN MIUR project.
• Thanks to the European Commission for founding. All slides reporting logo of
Snap4City of Select4City H2020 are representing tools and research founded by
European Commission for the Select4City project. Select4City has received
funding from the European Research Council (ERC) under the European Union's
Horizon 2020 research and innovation programme (grant agreement n° 688196)
• Km4City is an open technology exploited by those projects and line of research of
DISIT Lab. Some of the innovative solutions and research issues developed into the
above mentioned projects are also compliant and contributing to the Km4City
approach and thus are contributing to the open Km4City model of DISIT lab.

scalable Smart aNalytic APplication builder for sentient Cities Overview -- September 2018

More Related Content

What's hot (20)

Similar to scalable Smart aNalytic APplication builder for sentient Cities Overview -- September 2018 (20)

Recently uploaded (20)

scalable Smart aNalytic APplication builder for sentient Cities Overview -- September 2018