Holo basics develop your mixed reality hololens app with unity and visual studio

DEVELOP YOUR MIXED REALITY APP FOR HOLOLENS
WITH UNITY AND VISUAL STUDIO
HOLOBASICS

“HoloLens Evangelist”
At ETTU I am responsible architecture related business for our project services
department at ETTU. I’m involved as architect at different companies like Planon,
PGGM, Gemeente Amersfoort and others
Alexander Meijers
ABOUT ME
 Solutions Architect for ETTU
 Founder of the Mixed Reality User Group
 MixUG: http://www.mixug.nl
 Email: a.meijers@ettu.nl or alexander@appzinside.com
 Twitter: @ameijers
 Blog: http://www.appzinside.com

3INTRO ETTU
TODAY’S TALK
AGENDA
 Mixed Reality
 Microsoft HoloLens
 Tools
 Our first hologram
 Sound
 Gestures
 Spatial Mapping
 Occlusion
Slider
Content Ads

HOLOBASICS 5
VIRTUAL REALITY
 Generation of realistic images, sounds and other
sensations that replicate a real environment or create an
imaginary setting
 An immersive experience created entirely from
computer-generated Content. Also similar to 360 degree
video

HOLOBASICS 6
AUGMENTED REALITY
 Direct or indirect view of a physical, real-world
environment whose elements are augmented by
computer-generated sensory input such as sound, video,
graphics or GPS data
 An overlay of content on the real world where that
content is not anchored to or part of it

HOLOBASICS 7
MIXED REALITY
 Merging of real and virtual worlds to produce new
environments and visualizations where physical and
digital objects co-exist and interact in real time.
 Overlay of synthetic content on the real world that is
anchored to and interacts with the real world

HOLOBASICS 9
HISTORY OF HOLOLENS
 Codenamed Project HoloLens
 Chief inventor Alex Kipman
 Official title is technical fellow
 He dreamed up Kinect in at the end of 2007
and set a vision which incorporated HoloLens
 It is the start of a transforming world
 In the new reality, sensors will be anywhere
 A visual computing platform controlled by speech and gesture
Satya Nadella - “The emergence of
the next computing interface”
Alex Kipman

HOLOBASICS 10
MICROSOFT HOLOLENS
SPECIFICATIONS
 Windows 10 device based on 32 bit architecture
 64GB flash
 2GB memory
 1GB Holographic Processor Unity (HPU)
 First of its kind
 Device is more powerfull than a laptop
 No overheating due to warm air flows to the sides
 2-3 hours active and 2 weeks standby
 Weight 579g

HOLOBASICS 11
SENSORS, OPTICS AND
SPEAKERS
Environment
camera
Depth
camera
Video
camera
Spatial Sound
With
speakers
High definition
lenses

HOLOBASICS 12
MICROSOFT HOLOLENS
 First Holographic computer running on Windows 10
 Contains a CPU, GPU and HPU
 Human understanding
 Gesture and voice input
 Gaze tracking
 Spatial sound
 Uses wi-fi to connect and has no wires attached
 Design for comfort

HOLOBASICS 13
MICROSOFT HOLOLENS
SPECIFICATIONS
 Contains depth camera
 Field of vision that spans 120 by 120 degrees
 18 sensors flooding the device with terabytes of data
every second
 Tricks your brain into perceiving holographic images
 Light engine in which light particles are bouncing million times
 Photons enter the two lenses
 Ricochet some between layers of blue, green and red glass
 Hitting back of the eye

HOLOBASICS 14
HOLOLENS EXAMPLES

HOLOBASICS 15
HOLOLENS EXAMPLES

HOLOBASICS 16
HOLOLENS EXAMPLES

HOLOBASICS 17
HOLOLENS EXAMPLES

HOLOBASICS 18
HOLOLENS EXAMPLES

“DEMONSTRATION OF HOLOLENS
APPLICATIONS”
“HoloBasics by Alexander Meijers”

HOLOBASICS 21
DEVELOPMENT TOOLS
 Visual Studio 2017
 UWP workload
 Game development with Unity workload
 Windows 10 SDK (version 1511 or later)
 There is no separate SDK for HoloLens
 Unity 5.6 or Unity 2017.1
 Be aware! - Unity 2017.2 beta for immersive headsets
 HoloLens emulator
 HoloLens Emulator (build 10.0.14393.1358)
 Hyper-V
 Contains DirectX project templates for Visual Studio
 HoloLens device

HOLOBASICS 22
DEVELOPMENT TOOLS
GITHUB
 Microsoft/MixedRealityToolkit-Unity
 Ameijers/HoloLens-Unity-Utilities

HOLOBASICS 23
BUILD LIFECYCLE OF A
HOLOLENS PROJECT
 Create Unity Project
 Configure HoloLens settings
 Create scene
Visual Studio
Unity HoloLens
 Configure build settings
 Build and generate Visual Studio
project
 Open project with Visual Studio
 Pair with HoloLens
 Build & deploy Visual Studio project
 Start Application
 Test & debug
 Monitor
Unity

HOLOBASICS 24
DEVELOPMENT FOR
HOLOLENS
 HoloLens Device
 Developer mode needs to be turned on
 Pair the device with your development machine
 Retrieve IP address of HoloLens Device
 Ask Cortana
 Settings  Network & Internet  Advanced options

HOLOBASICS 25
WINDOWS DEVICE PORTAL
 3D View
 Mixed Reality Capture
 Performance
 Performance tracing
 System performance
 Processes
 Apps
 Maintenance
 Crash dumps
 Additional tools
 Logging
 File Explorer
 Virtual Input

HOLOBASICS 26
UNITY
EDITOR
Main
Camera
Directional
Light
Game
Objects
Assets
Components
Script
Scene

HOLOBASICS 27
UNITY
PREFABS
• Predefined Game Objects
• Contains components with predefined values
• Prefabs used in Scene inherit the components and settings

HOLOBASICS 28
UNITY
GAMEOBJECT SCRIPT
• Derived from MonoBehaviour
• Contains methods which are called by Unity through the a
broadcast messaging system
• void Awake() – Called when class is instantiated
• void Start() – Called when object is enabled once before
the first Update()
• void Update() – Called once per frame
• More methods are available
• FixedUpdate, LateUpdate, OnGUI, OnDisable, OnEnabled

HOLOBASICS 29
UNITY
MESSAGE SYSTEM
public void BroadcastMessage(string methodName, object parameter = null, SendMessageOptions options =
SendMessageOptions.RequireReceiver);
• Used for calling methods on GameObjects
• Calls the methods also on its Children
• Examples
• this.BroadcastMessage(“OnSelect”);
• focusedObject.BroadcastMessage(“OnSelect”);

HOLOBASICS 30
UNITY
COMPONENTS
• Transform
• Position of object
• Rotation of object
• Scale (size) of object
• Mesh Renderer & Mesh Filter
• Renders the object
• Mostly done using a mesh
• Mesh Collider
• Determines the collision for the object
• Script
• Perform actions on the object
• Handle messages on an object

HOLOBASICS 31
TOOLS
3D MODELING
Blender
Studio 3DS
Max
Many others like
Maya, Rhino3D
Paint 3D

HOLOBASICS 33
COORDINATE SYSTEM
HOLOLENS AND UNITY
forward (x, y, z)Camera (0, 0, 0)
Rotation (x, y, z)
Camera (1, -1, 4)
Rotation (x, y, z)
Hologram
Object Pinned
HoloLens
moved to
new location
View direction
Gaze

HOLOBASICS 34
HOLOGRAMS
AND PINNING
 Device can project a hologram into a
room and keep it locked in position
 Called pinnnig
 Objects are not moving relative to you
but you can move around the object
 Allows you to view the object from any angle

“DEMONSTRATION OF PINNED HOLOGRAM”

HOLOBASICS 36
PINNED HOLOGRAM
SUMMARY
 Basics of Unity and 3D objects
 Building an app using prefabs, components and script
 Build lifecycle of a HoloLens application using Unity and Visual Studio
 Pinned hologram robot in the room continue facing the HoloLens wearer

HOLOBASICS 38
SPATIAL SOUND
OVERALL
• Simulates 3D sound using direction,
distance and environmental
simulations
• Above, below, behind, to the side, etc.
• Attach sound to holographic objects
• Works also when object is not in line of
sight
• Used to draw attention
• Audio engine in HoloLens
• CPU and memory considerations
• 10-12 spatial sound voices

HOLOBASICS 39
SPATIAL SOUND
TYPES
• Gaze mixing
• Highlighting objects
• Play a sound on the object to get the users attention
• Audio Haptics
• Reactive audio for touchless interactions
• Play a sound when user selects an object or when his hands appear inside
the gesture frame
• Immersion
• Ambient sounds surrounding the user
• Support your scene with background sounds or music

HOLOBASICS 40
SPEECH
CORTANA
• Start with “Hey Cortana”
• Some thing you can say
• Move <app name> here
• Take a picture
• Start or stop recording
• Call <contact>
• How much battery do i have left?
• Restart or Shutdown
• Cortana can be turned on/off

“DEMONSTRATION OF SPATIAL
SOUND ADDED TO HOLOGRAM”

HOLOBASICS 42
PINNED HOLOGRAM
SPATIAL SOUND
 Unity and audio sources
 Creating a spatial sound with an audio source
 Explained how the speakers of HoloLens adjust based on the location of the sound

HOLOBASICS 44
GESTURES
GAZE
• The direction of the HoloLens pointing is called the
gaze
• Hits an object or part of the environment
• Uses a Vector3 based value to define its so called
“forward” direction

HOLOBASICS 45
GESTURES
GESTURE RECOGNIZER
• Recognize input by tracking the position of either or both hands
• Gesture frame (Frustum)
• Recognized input from hands
• Bloom
• Press, hold and release
• Following your hand
• Gestures
• Hold
• Manipulation
• Navigation

HOLOBASICS 46
GESTURES
GESTURE RECOGNIZER
• Connect commands to gestures
• GestureRecognizer gestures = new GestureRecognizer();
• gestures.TappedEvent += (source, tapCount, ray) => { … };
• gestures.StartCapturingGestures();
• Events
• Tapped
• Select press and release
• Hold (Started, Completed, Canceled)
• Select press beyond system hold threshold
• Navigation (Started, Updated, Completed, Canceled)
• Select press with relative movement
• Velocity based continuous scrolling or zooming
• Virtual Joystick
• Manipulation (Started, Updated, Completed, Canceled)
• Select press with absolute movement
• Move, resize or rotate hologram

HOLOBASICS 47
GESTURES
SPEECH RECOGNIZER
• Use words or sentences to control your environment
• English language only at the moment
• Connect commands to spoken text
• Dictionary<string, System.Action> keywords = new Dictionary<string,
System.Action>();
• keywords.Add(“some text”, () => { … };
• KeywordRecognizer keywords = new
KeywordRecognizer(keywords.Keys.ToArray());
• keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;
• keywordRecognizer.Start();
• OnPhraseRecognized( PhaseRecognizedEventArgs args);
• Invoke the action
• keywords[args.text].Invoke();

“DEMONSTRATION OF GESTURES BASED ON
HAND AND SPEECH TO INTERACT WITH
HOLOGRAM”

HOLOBASICS 49
PINNED HOLOGRAM
GESTURES WITH HAND AND SPEECH
 Implemented code for handling the gaze and cursor
 Implemented code for adding gestures
 Implemented code for adding speech
 Control the robot by shooting at it with a click gesture
 Control the scene by using words and sentences like “reset”

HOLOBASICS 51
SPATIAL PERCEPTION
HOW DOES IT WORK?
• Spatial perception and mapping
• Rooms
• Different ways of detecting and working with spatial
perception
• Device scans your environment and builds a digital model in
real time
• It allows HoloLens to see different surfaces, walls and ceiling
• Meshes & planes
• Possible to simulate a physical space
• E.g. Projecting a terrain over your floor
Spatial perception is the
ability to be aware of
your relationships with
the environment around
you and with yourself

HOLOBASICS 52
SPATIAL MAPPING AND
UNDERSTANDING
 Device scans your environment and builds a
digital model in real time
 It allows HoloLens to see different surfaces, walls
and ceiling
 Possible to simulate a physical space
 E.g. Projecting a terrain over your floor

“DEMONSTRATION OF LIVE SPATIAL MAPPING
AND UNDERSTANDING BY HOLOLENS”

HOLOBASICS 54
PINNED HOLOGRAM
SPATIAL MAPPING UNDERSTANDING
 Windows Device Portal connected to the HoloLens device
 Mapping the room by walking around
 Show mapping in the Windows Device Portal

HOLOBASICS 55
SPATIAL MAPPING AND
UNDERSTANDING
 Spatial Mapping components in Unity
 Spatial Mapping Renderer script
 Spatial Mapping Collider script
 Use an empty GameObject in your scene
 Spatial Mapping render state
 None
 Visualization
 Occlusion
 Cursor on spatial mapping
 SpatialMappingCollider.layer = 31;
 Cursor uses Raycast with layer mask 1<< 31
 Rendering material
 Occlusion material
 Visual material

“DEMONSTRATION OF SPATIAL MAPPING
USED WITH HOLOGRAM”

HOLOBASICS 57
PINNED HOLOGRAM
SPATIAL MAPPING
 Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity
 Show spatial mapping by using the HoloDeck material/shader

HOLOBASICS 59
OCCLUSION
CULLING
 Process of determining which surfaces and parts are not visible from a certain viewpoint
 Create a more naturally feeling with mixed reality
 Holograms can be occluded by
 Other holograms
 Real-life objects in your environment which are spatially mapped
 Culling enforces the HoloLens not to (partially) render the object
 To speed up the rendering process
 Objects which are too far away are left out the view (frustum)
 Use Spatial Mapping Renderer
 Set Occlusion material
 Set SpatialMappingRenderer.renderState = RenderState.Occlusion;

“DEMONSTRATION OF SPATIAL MAPPING AND
OCCLUSION WITH HOLOGRAM FOR TRUE
MIXED REALITY”

HOLOBASICS 61
PINNED HOLOGRAM
SPATIAL MAPPING AND OCCLUSION
 Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity
 Change the way spatial mapping is done for None, Visualization and Occlusion
 Placed another robot behind a real-life object and explain occlusion

HOLOBASICS 62
WRAP-UP
TAKAWAYS
 Unity and Visual Studio are the tools when it comes to developing mixed reality apps for HoloLens
 Know your tools like the Windows Device Portal
 Understand physics when creating an app for HoloLens
 Lot’s of stuff to make your life easy for gestures, speech, spatial mapping and other
 Understand the basics and how it works
 Lot of stuff available on GitHub like the MixedReality Toolkit to make your life easier

THANKS FOR WATCHING!
HoloBasics code on gitHub - https://github.com/ameijers/HoloBasics
Kampenringweg 45b, Gouda
0182-686 000
a.meijers@ettu.nl / alexander@appzinside.com
Contact me:
facebook.com/alexandermeijers.5
@ameijers
Follow me on:

Holo basics develop your mixed reality hololens app with unity and visual studio

More Related Content

What's hot (20)

Similar to Holo basics develop your mixed reality hololens app with unity and visual studio (20)

More from Alexander Meijers (20)

Recently uploaded (20)

Holo basics develop your mixed reality hololens app with unity and visual studio

Editor's Notes