Nui e biometrics in windows 10

NUI e Biometrics in Windows 10

Andrea Sassetti
andrea.sassetti@live.it
@andreasassetti
andreasassetti.wordpres.com
Software Engineer, Siemens

Marco Dal Pino
SW Eng - Consultant
@MarcoDalPino
mobileprog.com (ENG)
mobileprog.net (ITA)
Microsoft MVP
Intel Black Belt Software Developer
Intel Realsense Innovator Realsense & IoT
about.me/marcodalpino
m.dalpino@dpcons.com

User interaction evolution
CLI
GUI
NUI

Natural User Interface
• Use movements / gestures inherent in human nature or in the actions
of every day
• Rapid functionality learning
• Can help people with disabilities to use the software

Microsoft Kinect v2
Kinect for Windows gives computers
eyes, ears. With Kinect for Windows,
businesses and developers are creating
applications that allow their customers
to interact naturally with computers by
simply gesturing and speaking.

Kinect v2 - Sensor Components
Power Light
RGB Camera
IR Emitters
IR Emitters
Depth Sensor
Depth Sensor
Microphone Array

Kinect v2 - Hardware requirements
• Physical dual-core 3.1 GHz (2 logical cores per physical) or faster
• USB 3.0 port dedicated to the Kinect for Windows v2 sensor
• 2 GB of RAM
• Graphics card that supports DirectX 11
• Windows or Windows Embedded 8 or 8.1, 10
Kinect Configuration Verifier tool
http://go.microsoft.com/fwlink/?LinkID=513889

Kinect v2 - High level architecture
Kinect Drivers
Kinect Runtime
.NET
API
.NET
Apps
Native API
Native Apps
WinRT API
Windows Store Apps
C#, VB, JS, C++/CX
Physical Kinect Sensor

Kinect
Service
Source
Kinect
Reader
Kinect
Reader
Kinect
Reader
Application
Application
Application
Multiple Kinect-enabled applications can run simultaneously

PC
PC
PC
PC
HUB SERVER
Multiple connectionOne connection

Kinect v2 – Frame source
Color Depth Infrared
Body index Body Audio

Kinect v2 – Color frame source
• 1920 x 1080 array of pixel
• Different sensor than depth
• 30 fps
• Will maintain brightness and quality
by dropping to 15 fps in low-light

Kinect v2 – Depth frame source
• 30 fps
• Distance range 0.5 – 4.5 meters

Kinect v2 – Infrared frame source
• 30 fps
• Ambient light removed
• From the same physical sensor as
depth

Kinect v2 – Body index frame source
• Up to 6 simultaneous bodies
• 30 fps
• Same resolution as depth

Kinect v2 – Body frame source
• Frame data is a collection of Body
object each with 25 joints (each joint
has position in 3D space and
orientation)
• Up to 6 simultaneous bodies
• 30 fps
• Hand state (Open, Closed, Lasso)
• Lean

Kinect v2 – Body frame source

Kinect v2 – Audio source
• Data is audio samples captured over
a specific interval of time
• Audio is associated with an “audio
beam”

Kinect v2 – Multi frame source
• Allows to get a matched set of frames
form multiple source on a single time.
• Delivers frames at the lowest fps of
the selected sources.

Kinect v2 – Coordinate Mapper
It provides conversion between each
systems. It is possible perform single
or multiple conversion.
Name Applies to Dimension Units Range Origin
ColorSpacepoint Color 2 pixel 1920x1080 Top left
DepthSpacePoint Depth, Infrared, Body
index
2 pixel 512x424 Top left
CameraSpacePoint Body 3 meter NA Infrared / Depth camera

Kinect v2 – Application Lifecycle
KinectSensor kinectSensor = KinectSensor.GetDefault();
if (!kinectSensor.IsAvailable)
return;
if(!kinectSensor.IsOpen)
kinectSensor.Open();
// CODE HERE
kinectSensor.Close();
First step:
Add reference to Microsoft.Kinect assembly
Microsoft Kinect NuGet Packages available

Kinect v2 – Basic flow of programming
KinectSensor.GetDefault().DepthFrameSource.OpenReader().AcquireLatestFrame().CopyFrameDataToArray(frameData);

Kinect v2 – Basic flow of programming
Readers allow to access to the Kinect frame.
It is possible to read the frame via pooling:
var colorFrameReader = kinectSensor.ColorFrameSource.OpenReader();
using (var colorFrame = colorFrameReader.AcquireLatestFrame())
{
if (colorFrame == null)
return;
// CODE HERE
}
var colorFrameReader = kinectSensor.ColorFrameSource.OpenReader();
colorFrameReader.FrameArrived += OnColorFrameReader;
---
private void OnColorFrameReader(object sender, ColorFrameArrivedEventArgs e)
{
using (var colorFrame = e.FrameReference.AcquireFrame())
{
if (colorFrame == null)
return;
// CODE HERE
}
}
or by event:

Kinect v2 – Face Detection and
Alinement
Through Kinect Face it is possibile detects body faces point and expression.
Face points:
• The user's left eye point
• The user's right eye point
• The user's nose point
• The user's left corner of the mouth point
• The user's right corner of the mouth point
Each point can be visualized in color or Infrared space.

Kinect v2 – Face Expression
Face expressions:
• The user's happy facial expression
• The user's right eye is closed
• The user's left eye is closed
• The user's mouth is open
• The user's mouth has moved since the previous frame
• The user is looking at the sensor
• The user is wearing glasses

Kinect v2 – Face Detection and
Alinement
• Add reference to Microsoft.Kinect.Face assembly
• Use FaceFrameSource and FaceFrameReader classes
• Use FaceFrame class in order to get face information

Kinect v2 – High Definition Face
Through Kinect High Definition Face it is possibile
detects over 1000 facial points in the 3D space.
• 1347 facial points
• 2340 triangles
• Hair color
• Body color
• Face 3D Model
Not sufficiently documented
Many API are available only in C++

Kinect v2 – High Definition Face
• Add reference to Microsoft.Kinect.Face assembly
• Use HighDefinitionFaceFrameSource and HighDefinitionFaceFrameReader classes
• Use HighDefinitionFaceFrame class in order to get high definition face information

Kinect v2 – Hand Pointer Gestures
Engagement
Press Panning / ZoomTargeting

Kinect v2 – Hand Pointer Gestures -
Press
Recommended Minimum Size:
• 208 x 208 (in 1080p resolution)
• Press attraction towards center
• Larger buttons will just attract
away from the edge
Adapting to smaller visual sizes:
• Make large-size hit testable
• Set KinectRegion.KinectPressInset (Thickness)
to non-visible part of button
• Attraction region is smaller

Kinect v2 – Hand Pointer Gestures -
WPF
• Add Reference to Microsoft.Kinect.Wpf.Controls
• Add KinectRegion as container for your Windows Control
• Run it!

Kinect v2 – Hand Pointer Gestures –
Store App
• Add Reference to Microsoft.Kinect.Xaml.Controls
• Enable Microphone + Camera capabilities in App Manifest
• Add KinectRegion as container for rootFrame in App.xaml.cs
• Run it!

Demo
Hand Pointer Gestures
and Frame source

Kinect v2 – Recording and Playback
Kinect Studio
• Record sample clip of data from the Kinect v2 device.
• Playback a recorded sample clip of data
• Play data from a live stream directly from a connected Kinect v2 Device

Kinect v2 – Visual Studio Visualizer
Visualizers are components of the Visual
Studio debugger user interface. A visualizer
creates a dialog box or another interface to
display a variable or object in a manner
that is appropriate to its data type.
It is possible to create custom visualizer just
implementing some .NET interfaces.

Kinect v2 – Fusion
Microsoft KinectFusion provides 3D
object scanning and model creation
using a Kinect for Windows sensor. The
user can paint a scene with the Kinect
camera and simultaneously see, and
interact with, a detailed 3D model of the
scene.

Kinect v2 – Custom gesture / pose
Inside Kinect v2 SDK it is available a tool that allow you to create
custom gesture.
Visual Gesture Builder
It is possible define discrete or continuous gesture using Machine
Learning technology
• Adaptive Boosting (AdaBoost) Trigger: Determine if player is
performing a gesture
• Random Forest Regression (RFR) Progress: Determine the
progress of the gesture performed by the player

Kinect v2 – Custom gesture / pose
Visual Gesture Builder
• Organize data using Project/Solution
• Give meaning to data by tagging gesture
• Build gesture using Machine Learning technology
• Analyze & Test the result of the gesture detection
• Live preview of result

Kinect v2 – Visual Gesture Builder
Basic steps for using Visual Gesture Builder:
1. Create a solution.
2. Create one or more projects.
3. Add a set of clip files to each project.
4. Tag the frames in the clip files associated
with each gesture.
5. Build a training gesture database.
6. Use the built gesture database into your
application.

Kinect v2 – Custom gesture/pose
Heuristic
• Gesture is a coding problem
• Quick to do simple
gestures/poses (hand over head)
• ML can also be useful to find
good signals for Heuristic
approach
ML with Visual Gesture Builder
• Gesture is a data problem
• Signals which may not be easily
human understandable (progress
in a baseball swing)
• Large investment for production
• Danger of over-fitting, causes you
to be too specific – eliminating
recognition of generic cases

Kinect v2 – Other Framework
Computer Vision APIs
Face APIs

Kinect v2 – Official Resources
• General Info & Blog
http://kinectforwindows.com
• Developer Forums
http://aka.ms/k4wv2forum
• Download Kinect v2 SDK
http://aka.ms/k4wv2sdk
• Twitter
@KinectWindows

Nui e biometrics in windows 10

Intel® RealSense™ 3D Camera F200

F200 Camera Specs
Spec Color Depth (IR)
Resolution 1080p VGA
Aspect Ratio 16:9 4:3
Frame Rate 30/60/120 FPS
FOV (D X V X H) 77º x 43º x 70º 90º x 59º x 73º
Other
Effective Range 0.2 – 1.2 m
Environment Indoor/Outdoor

Intel Information Technology
51
Where to find the Intel® RealSense™ technology?

Intel® RealSense™ 3D Cameras
Intel® RealSense™ 3D
Camera (Front F200)
Intel® RealSense™
Snapshot
Intel® RealSense™ 3D
Camera (Rear R200)

• Free tools and APIs to develop
apps with NUI in a simple way
• Focus where is important: the
content
• Acessible for beginners and
extensible for experts

Hands -
Tracking and Joints
• 22 Joints
• Detects Body Side
• Tracks X, Y and Z positions from
detected hands

Face – Tracking
• Tracking ( 4 faces)
• Landmarks Tracking (1 Face)
• X, Y, Width and Height
• Average Depth

Face – Landmarks
• 78 landmark points:
• Nose
• Mouth
• Jaw
• Eyes
• Eyebrows

Face – Emotions (Experimental)

Speech – Recognition
Switch
Weapon!

Speech – Synthesis
Your Score is: Five
Hundred and Sixty
Five!

Microsoft Passport e Windows Hello

Going beyond passwords
Problems:
• Passwords are hard to remember
• Passwords are re-used -> server breach attacks
Microsoft passport solution:
• User has to remember only one PIN or can use
Windows Hello
• No secret is stored on servers -> Two factor
authentication with asymmetric keys

Windows 10 and Microsoft Passport Simple for
Developer
• Reduces the cost associated with password
compromise and reset
• Native API support for strong authentication via
Universal Windows Platform
• JavaScript API support for browsers coming later this
year.

Windows Hello: Sensor Support
• Fingerprint
• All current fingerprint capable devices are supported
• Face
• All current (f200) and future Intel® RealSense™ are supported for Windows Hello face
authentication
• All devices that include an IR sensor that meets Microsoft sensor spec
• Iris
• A selection of devices will be arriving to market within the next 12 months
• More details on sensor requirements and support coming soon

Windows Hello
• Windows 10 support three type of biometric Auth:
• Fingerprint
• Face
• Iris
• All of three provide gesture and action recommended on the
hardware supported
• Integrated with the Windows Biometric Framework
• use the same scenary and experience
• Share the same language for use and definition

Microsoft Passport: Provisioning
Two Step
Verification
Create Microsoft
Passport
Register Public
key with Server
• KeyCredentialRetrievalResult kcResult = await
• KeyCredentialManager.RequestCreateAsync(accountID,
KeyCredentialCreationOption.FailIfExists);

Microsoft Passport: Usage
Open Microsoft Passport
Sign challenge from Server
with Microsoft Passport
Send digital signature to
server
• KeyCredentialOperationResult kcOpResult = await
kcResult.Credential.RequestSignAsync(serverChallenge);

Microsoft Passport: Deletion
Open Microsoft
Passport
Delete Passport
Delete Public
Key on Server
• KeyCredentialRetrievalResult kcResult = await
KeyCredentialManager.OpenAsync(accountID);

Reference
• Build Session
• Microsoft Passport and Windows Hello: Moving Beyond Passwords and Credential Theft
• https://channel9.msdn.com/events/Build/2015/2-639
• API
• Windows.Security.Credentials namespace
• Https://msdn.microsoft.com/en-us/library/windows/apps/windows.security.credentials.aspx

Nui e biometrics in windows 10

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to Nui e biometrics in windows 10 (20)

Recently uploaded (20)

Nui e biometrics in windows 10

Editor's Notes