Digital Twin: convergence of Multimedia Technologies

Digital Twin: the Convergence of
Multimedia Technologies
Abdulmotaleb El Saddik
FIEEE, FCAE, FEIC
Abdulmotaleb El Saddik
FIEEE, FCAE, FEIC
Distinguished Professor & University Research Chair
Multimedia Communications Research Lab
elsaddik@uottawa.ca
© elsaddik@uottawa.ca 2018

Digital Twin
A digital twin is a digital replica of
a living or non-living physical entity1.
By bridging the physical and the virtual world, data is transmitted seamlessly
allowing the virtual entity to exist simultaneously with the physical entity.
1 El Saddik, A. (2018). Digital Twins: The Convergence of Multimedia Technologies. IEEE MultiMedia, 25(2), 87-92.

Why is digital twin important
now?• According to Gartner,
– Digital Twin is the 4th of the top 10 technological trends for
2019
– More than 50% IoT companies teams have digital twin in
their annual plan as a strategic mandate
• According to Market Research Future:
– it is expected that the digital twin market will reach $15B by
2023
• Smart Cities becoming the new political
mandate

Facts
Things and Being are interconnected
Demographic data
from various parts
of city
Data from various
soft sensors
Data from
various
hard-sensors
Data from
surveillance
cameras
Data from the
crowd
Data from health
monitors of
citizens
Source of data: People,
Process, Product
BigMM & AI
Multimodal Interactions Cybersecurity & Biometrics
Massive
throughput
Massive low
latency
Massive
sensing
Massive
heterogeneity
Fast feedback
Massive
privacy
Security/Trust
5G & Tactile Internet

© 2016, Prof. A. El Saddik, elsaddik@uottawa.ca, reproduce with permission
Convergence of MM Tech
Cybersecurity
(including biometric,
privacy)
Data
IoT
Social
Networks
QoE-powered
Communications
(BT, Wi-Fi, 4G,
5G, etc.)
AI MMI
AR/VR
Holograms Haptics/
H-Robots
Machine
Learning
Deep
Learning
Ontology
Audio/
Video
Rules
Cognition
FinanceMedical

Digital Twin
Topological
Structure
Plug & Play
Transducer
Kazi Masudul Alam and Abdulmotaleb El Saddik, “C2PS: A Digital Twin Architecture Reference Model for the Cloud‐
Based Cyber‐Physical Systems”, IEEE Access, vol. 5, pp. 2050–2062, 2017

Our journey with
Digital Twins

9

By placing multiple 3D sensors around the measurement area and tracking simultaneously, we can obtain
a more accurate result by applying trilateration method.
3D Motion Capture

Depth image accuracy evaluation
Accuracy Distribution – Result : elliptical cone
Lin Yang, Longyu Zhang, Haiwei Dong,
Abdulhameed Alelaiwi and Abdulmotaleb El
Saddik, "Evaluating and improving the depth
accuracy of Kinect for Windows v2," IEEE
Sensors Journal, vol. 15, no. 8, pp. 4275-4285,
2015.

Experiment Results
Self calibrating Motion Capture
Bowen Yang, Haiwei Dong, and Abdulmotaleb El Saddik, “Development of a Self‐Calibrated Motion Capture System based
on Nonlinear Trilateration of Multiple Kinect v2”, IEEE Sensors Journal , pp: 2481 – 2491, Vol. 17(8), 2017

Our sensing device is a consumer-
grade composite sensor, including
Kinect v2 sensor, gyroscope, digital
compass, and accelerometer. The Kinect
v2 sensor is used to capture depth data,
while other sensors are mainly used to
reduce accumulative errors of our
system.
The composite sensor is
fixed on a tripod, and mounted on a
robot , which automatically rotates
around the human subject with
approximate 1-meter radius, to capture
the full-view information.
Scanning Device
Proposed sensing device carried with a robot
Development of an Automatic 3D Human Head Scanning-
Printing System

Nadia Figueroa, Haiwei Dong and Abdulmotaleb El Saddik, “A combined approach towards consistent reconstructions of indoor
spaces based on 6D RGB-D odometry and KinectFusion,” ACM Transactions on Intelligent Systems and Technology, vol.
6, no. 2, pp. 14:1-10, 2015.
Development of an Automatic 3D Human Head
Scanning-Printing System

Accuracy Comparison 1: Standard Human Head Model
To validate the capabilities of our
proposed scanning system, a
visualized 3D CAD model of a
standard human head model is
prototyped, 3D printed and
scanned separately by our
proposed scanning system and by
a commercial handheld 3D laser
scanner FastSCAN. Furthermore,
we computed the geometric
differences (represented by
Hausdorff distances) between the
two scanned 3D models and the
ground-truth model separately.
Longyu Zhang, Bote Han, Haiwei Dong and Abdulmotaleb El Saddik, “Development of an Automatic 3D Human Head
Scanning‐Printing System”, Springer Multimedia Tools and Applications (2016). doi:10.1007/s11042‐016‐3949‐2

Hausdorff distance is the maximum value from Model_1 → Model_2 and Model_2 → Model_1 in the
Euclidean space:
Visualization of Hausdorff distance between scanned models and
the ground-truth model separately. Top Row: Proposed system
result. Bottom Row: FastSCAN scanner result.
Accuracy Comparison 1: Standard Human Head Model

To further validate our
experimental results, we
compared scanned real human
models from our proposed
system with the ones from
commercial Cyberware laser
scanning system on real-human
subjects.
Accuracy Comparison 2: Real Human
Experimental setup for the evaluation of real human
scanning

Male and female subjects' real image, scanned results from our proposed system and Cyberware
separately, and the Hausdorff distance visualization results.
Development of an Automatic 3D
Human Head Scanning-Printing System
Accuracy Comparison 2: Real Human
Experimental results of real human scanning

1 min scanner
Longyu Zhang, Bote Han, Haiwei Dong and Abdulmotaleb El Saddik, “Development of an Automatic 3D Human Head
Scanning‐Printing System”, Springer Multimedia Tools and Applications (2016). doi:10.1007/s11042‐016‐3949‐2

Stiffness Rendering
Original Models
Registered Models
Skull Models
Haptics Rendering
2) Head Stiffness Rendering

21
Facial Landmark Detection and Model Registration
The head mesh and skull meshes are registered through ICP procedure by minimizing the
correspondences between the facial features detected from both meshes.
Features on skull mesh are generated with aesthetic proportions.
1.

22
Deformed skull mesh of a male subject’s head mesh:
a) is the original skull template mesh and b), c) and d) are the fully deformed skull mesh of
isometric, frontal and profiles view respectively.
Fully processed approximation results
2. Hierarchical Region of Interest
Process

Minggao Wei, Yang Liu, Haiwei Dong, and Abdulmotaleb El Saddik “Human Head Stiffness Rendering”, IEEE Transactions
on Instrumentation & Measurement, Vol 66(8), pp: 2083‐2096, DOI: : 10.1109/TIM.2017.2676258

24
As a haptic rendering method applying magnetic
repulsive force to generate tactile sensation in
mid-air
Haptic display :
electromagnet array
driven by DC current
Glove attached with
magnet discs
Visual display
3) Magnetic Rendering
Introducing a new way of haptic volumetric shape rendering

25
Design of Electromagnet
Design of a powerful electromagnet model
Magnetic Field Concentrate Electromagnet
-3 -2 -1 0 1 2 3
-0.5
0
0.5
1
1.5
2
2.5
3
3.5
X (cm)
MagneticRepellingForce(N)
Force along X among three models for one element at 1cm
designed
concentrated
traditional
3cm
210%
FEM simulation result using COMSOL Multiphysics
Traditional Model
Multi-turn coil
Soft iron core
Magnetic field concentrator
Mu-metal shielding
Concentrated Model
Designed Model
Electromagnet array

26
Video
Qi Zhang, Haiwei Dong, and Abdulmotaleb El Saddik, “Magnetic Field Control for Haptic Display: System Design and
Simulation” IEEE Access , Vol 4, 2016, pp 299‐311, DOI: 10.1109/ACCESS.2016.2514978

Design Requirements
• Frequency of the UPEMF wave (30Hz)
• Shape of the UPEMF wave (Rectangular)
• Intensity of the UPEMF wave (0.12T)
Define the desired waveform

System Design
• Proposed UPEMF system

Test
Yuxiang Jiang, Haiwei Dong, Abdulmotaleb El Saddik, “A Unipolar Pulse Electromagnetic Field Apparatus for Magnetic
Therapy” IEEE Instrumentation & Measurement Magazine (to appear)

Why Haptics – Biometrics?
Haptic Systems
Biometrics Systems
The problem with Biometrics
without Haptics (movie)

Case Study: Identifying Human
Pattern with Haptics
The Experiment
Methodology
Dynamic Time Warping :
+ Nelder-Mead non-linear minimization
Spectral analysis: Fast Fourier Transform
Unsupervised Method: K-Means
   

3
1c
N
1i
2p2
c,i
1
c,i tldMS
Graphic Representation

Verifying such feasibility
Virtual Check
Virtual Mobile
Phone
Performance of Classifier
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
FAR er r or r at e
Virtual Check Maze Virtual Phone
Ambient Intelligent Engine
Performance
El Saddik et al., “A Novel Biometric System for Identification and
Verification of Haptic Users”, IEEE Transactions on
Instrumentation and Measurement, Vol.56, No. 3 (2007), pp: 895
– 906.

Experimental Results

Virtual Check
FAR = ~7%
FRR = ~12%
Nizar Sakr, Fawaz Alsulaiman, Julio J. Valdes and Abdulmotaleb El Saddik, “Identity Verification based
on Haptic Handwritten Signatures using Genetic Programming” ACM Transactions on Multimedia
Computing Communications and Applications Vol 9(2), 2013

Multibiometrics
ECG + Fingerprint

ECG Biometrics
 We obtained 84.93% for TAR and 1.29% for FAR, with the advantage that the required time to perform
the authentication with our proposed algorithm is 4 seconds only.
• US Patent: US9699182B2 “Electrocardiogram (ECG) biometric authentication” by A. El Saddik, J, Arteaga Falconi & H. Al Osman
• ECG Authentication for Mobile Devices, Juan Sebastian Arteaga-Falconi ; Hussein Al Osman ; Abdulmotaleb El Saddik
• IEEE Transactions on Instrumentation and MeasurementYear: 2016, Volume: 65, Issue: 3, Pages: 591 - 600

Emotion
What is behind your facial features ?
Picture from: http://edwardpun.blogspot.ca/2006_12_01_archive.html
Is the Medical Field the only place for
HRV analysis ?
What about Affective Computing ?

Proposed System
One input frame from the video Amplitude mapping Phase mapping
Compared with the red and blue channel, the green channel contains the
strongest color variation due to blood circulation.

HRV parameters
Parameter Description
Time
Domain
HR The heart rate (bpm)
The mean of RR intervals
SDNN The standard deviation of RR intervals
RMSSD the root mean square of successive RR intervals difference
NN50 Number of RR intervals pairs differing more than 50 ms
pNN50 The division of NN50 over total number of RR intervals (%)
Frequency
Domain
LF The low frequency extracted from RR intervals after applying PSD
HF The high frequency extracted from RR intervals after applying PSD
LF/HF The LF frequency value divided by the HF value

Capturing Emotions
• Image pre-processing
• CNN training process
– Two-stage fine-tuning
– Center Loss

Real-time Experiments

iAware
iAware visual feedback: (a) for natural, (b) happy, (c) sad, (d) love, and (e) fear emotions}
Emoji-based version of the emotion wheel
The iAware emotion monitoring system

Real-time Emotion (DL) and FeedbackStimuli: Argentina scored the first goalStimuli: Conversations with a friendStimuli: Argentina's player injured
Amani Albraikan, Diana Patricia Tobón Vallejo, and Abdulmotaleb El Saddik, “Toward User‐Independent Emotion
Recognition using Physiological Signals”, IEEE Sensors (accepted)

Direct Feedback

CTV News
Emotional Cities
Neural Network
Pre-processing /
feature extraction
Training and testing
Emotion detection
Geo-tagged tweets
with emotion (PADU)
Aggregation and
visualization
Google Maps API
#hashtagged
geo-location
Social networks
Surveillance
cameras
Smartphones
features /usages
Affective
Data Sources
....
Affective
Data Type
Images
Voice
Video
Physiological
signals
Text
Biofeedback
Sensors
....

Single City Analysis
Zhongli Li, Shiai Zhu, Huiwen Hong, Yuanyuan Li, and Abdulmotaleb El Saddik “City Digital Pulse: A Cloud Based
Heterogeneous Data Analysis Platform”, Springer Multimedia Tools and Applications 76 (8), 10893‐10916, 2017

Hashtag Img and Sentiment Change

Multiple City Comparison

Champions League Analysis
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
Jun-16 Jul-16 Aug-16 Sep-16 Oct-16 Nov-16 Dec-16 Jan-17 Feb-17 Mar-17 Apr-17 May-17 Jun-17
TweetsNumber
Months
Negative
Neutral
Positive
Round of 16
“ The long bomb was sick “
Negative words
“ Luis. F***. Suarez. Again ”

Champions League Analysis
• The Final Game
-5
-4
-3
-2
-1
0
1
2
3
4
5
2 hours before during 2 hours after
Tweetssentiment
Juve
RM
0
200
400
600
800
1000
1200
1400
0'-15' 16'-30' 31'-45' 46'-60' 61'-75 76'-90 91'-120'
TweetsNumber
Minutes during game
RM
Juve
Samah Aloufi; Fatimah Alzamzami; Mohamad Hoda; Abdulmotaleb EI Saddik, “Soccer Fans Sentiment Through
The Eye of Big Data: The UEFA Champions League as a Case Study”, in Proceedings of the 1st IEEE International
Conference on Multimedia Information Processing and Retrieval. Venue Pullman Airport Hotel, Miami, FL, USA,
April 10‐12, 2018

Human Needs in Affect-Aware City
• Understand the affective states of the citizen
– Provide a form of implicit feedback.
• The interpretation of the analyzed affective states can guide
authorities in improving situational awareness and quality of life.
Actions and Behaviors
Positive , Negative , Neutral
Happy , Sad , Love , Joy , Anger , Fear, …
?
What people are doing?
How they feel ?
Why they are behaving in
certain way?
-Sentiment Analysis
-Opinion Mining
-Automatic Emotion
Identification
-Modeling Crowd Activity
Basic Needs
-Analyzing Basic Needs
Rajwa Alharthi, Benjamin Guthier, Camille Guertin, Abdulmotaleb El Saddik, “A dataset for psychological human needs
detection from social networks”, IEEE Access, 10.1109/ACCESS.2017.2706084, 2017

Case Study: Florida Shooting
1st 2nd 3rd 4th
0
10
20
30
40
50
60
70
80
90
100
Wednesday Thursday Friday Saturday
PERCENTAGEOFTWEETS
NEED SATISFACTION LEVELCHANGES DURING FLORIDA SHOOTING
Satisfied Relatednees Need Dissatisfied Relatednees Need Satisfied Autonomy Need Dissatisfied Autonomy Need
1.#TalkAboutItNow
2.#GunControl
3.#Broward
4.#GunControlNow
5.#LoveisLouder
6.#Prayers
7.#MassShooting
8.#RIP
9.#ThoughtsandPrayer
10.#Condolences
1.#TalkAboutItNow
2.#GunControl
3.#Broward
4.#GunControlNow
5.#LoveisLouder
6.#Prayers
7.#MassShooting
8.#RIP
9.#ThoughtsandPrayer
10.#Condolences

Florida Shooting
0
10
20
30
40
50
60
70
80
90
100
PERCENTAGEOFTWEETS
1st 2nd 3rd 4th
1.#GunControl
2.#GunControlNow
3.#Islam
4.#Poor
5.#NRA
6.#PrayForDouglas
7.#America
8.#PolicyandChange
9.#GunReformNow
10.#NRABloodMoney

Florida Shooting
0
10
20
30
40
50
60
70
80
90
100
PERCENTAGEOFTWEETS
1st 2nd 3rd 4th
1.#GunControl
2.#FBI
3.#GunReformNow
4.#GunControlNow
5.#MAGA
6.#NikolasCruz
7.#Trump
8.#SecondAmendment
9.#StandYourGround
10. #NRA

Florida Shooting
0
10
20
30
40
50
60
70
80
90
100
PERCENTAGEOFTWEETS
1st 2nd 3rd 4th
1.#ThrowThemOut
2.#GunControl
3.#GunReformNow
4.#GunControlNow
5.#NRA
6.#NikolasCruz
7.#GunReform
8.#FBI
9.#StudentsDemandAction
10.#EndGunViolence

Digital Twin
4
Health and Well-being

user
Wrapper
X73 Communication
Module
X73 Compliant Digital Twin for
Healthcare Cloud Based System
Visualization and Data Analysis
Module
Non-standardized
health devices
X73 compliant
health devices
X73
Standardized
Communicatio
n
Proprietary
Communication
Health Device Transmitted
Data
Data Storage Module
Health Data and
Profile
Standardized
Communication
following X73
Communicatio
n
Mobile Device
IEEE X73 Platform 4 DT
H. Badawi, F. Lamarti, F. Arafsh, and A. El Saddik, “Standardizing a Shoe Insole as a Personal Health Device (PHD) Based on ISO/IEEE
11073 (X73)” In Proceeding of the 2019 International Conference on Information Technology & Systems - ICITS'19 (Feb. 6-8, 2019) Ecuador

Physical Activity Advisory System
Mobile cloud-based physical activity advisory system using biofeedback sensorsHF Badawi, H Dong,
A El Saddik, Future Generation Computer Systems 66, 59-70

Collect data about Mobile usage
5 student-athletes
from the University
of Ottawa install app
on their personal
smartphone
Smartphone usage is
tracked automatically,
remotely, and in real-time
via the mobile app
Participants complete self-report
surveys pertaining to:
a) key psychosocial variables
b) perceived smartphone usage
c) perceived sport performance
Usage data is subjected to
algorithms to extract
detailed statistical
information. All data is
used to create preliminary
participant profiles
15-days
All surveys are collected via
the mobile app
Poppy DesClouds, Fedwa Laamarti, Natalie Durand‐Bush, Abdulmotaleb El Saddik, “Developing and testing an
application to assess the impact of smartphone usage on well‐being and performance outcomes of student‐athletes”, in
Proceedings of the 2018 International Conference on Information Technology & Systems

Results
0
5
10
15
20
25
1 2 3 4 5
SmartphoneUsage(hours)
Participant
Usage Throughout the Day
Morning Usage Afternoon Usage Evening Usage Overnight Usage
Fig. 1. Prevalence of smartphone usage throughout four periods of the day, morning (6am-12pm), afternoon
(12pm-6pm), evening (6pm-12am), and overnight (12am-6am).
• Participants total smartphone usage ranged from 20.5 hours to 119.4 hours
• Average usage of 4.5 hours per day (31.7 hours per week)

Reconfigurable Transducer Network
• Multiple types of sensors
are developed, such as
pressure sensor, light
sensor, temperature
sensor, accelerometer,
CO-gas sensor, etc.
• The dynamic
condition of a multiple
rooms during a period
of 24 hours is
monitored.
Recommendations
are given by fuzzy
logic.
Basim Hafidh, Hussein Al Osman, Haiwei Dong and Abdulmotaleb El Saddik ,“A framework of reconfigurable transducer
networks and its XML-based communication,” IEEE Embedded Systems Letters, vol. 7, no. 3, pp. 81-84, 2015.

Backward
Walking
Forward Walking
Smart Insole

Results

Results
Faisal Arafsha , Christina Hanna, Ahmed Aboualmagd, Sarah Fraser and Abdulmotaleb El Saddik, “Instrumented
Wireless SmartInsole System for Mobile Gait Analysis: A Validation Pilot Study with Tekscan Strideway”, J. Sens.
Actuator Netw. 2018, 7(3), 36; doi:10.3390/jsan7030036

Post Stroke rehabilitation
- Ali Karime, Hussein Al-Osman, Jihad Mohamad Alja'am, Wail Gueaieb, and Abdulmotaleb El Saddik, “Tele-Wobble: A Tele-Rehabilitation
Wobble Board for Lower Extremity Therapy”, IEEE Transactions on Instrumentation and Measurements, 61 (7), 1816-1824 , 2012
- Atif Alamri, Jongeun Cha, and Abdulmotaleb El Saddik, “AR-REHAB: an Augmented Reality Framework for Post-Stroke Patients
Rehabilitation”, IEEE Transactions on Instrumentation and Measurements, Vol. 59(10), pp: 2554 – 2563, 2010

Elderly

Smart gloves for diabetics

ALS – Eye Gaze, Sensors
A Novel Eye-Gaze-Controlled Wheelchair System for Navigating Unknown Environments: Case Study With a
Person With ALS.MA Eid, N Giakoumidis, A El-Saddik, IEEE Access 4, 558-573

Final Thoughts
• Digital twin will fit in very well the AI strategy
which every country has.
• Digital twin is democracy of data in its truest
sense.
– It’s going to shake the norms and laws of privacy and we will as a society
have to revisit what value do we perceive.
• We need to balance convenience versus privacy
– technology ethics will be the newest branch of ethics that’s going to be
out there
• Is it ethical to smart surveillance people and know
their emotions or heart rates

Final Thoughts
• Digital twin can act as a tool to remove the
gender biases we see today in some
domains
– It gives hope for an inclusive society
• It will definitely challenge the existing
power structures .
• Like we have the feminist movement,
– we will soon have anti‐technology movements on a grand
and serious scale

Acknowledgment

73Thank you for your attention!

Digital Twin: convergence of Multimedia Technologies

More Related Content

What's hot (20)

Similar to Digital Twin: convergence of Multimedia Technologies (20)

Recently uploaded (20)

Digital Twin: convergence of Multimedia Technologies