Wearable Computing and Human Computer Interfaces

Jeffrey Funk
Division of Engineering and Technology
Management
National University of Singapore
Human-Computer Interfaces and
Wearable Computing
Thanks to Karthik Nandakumar for the first drafts of these slides in Spring 2012
For information on other technologies, see http://www.slideshare.net/Funk98/presentations

What do the FollowingTechnologicalTrends
Mean for Human-Computer Interfaces?
 Better ICs, MEMS, bio-electronic Ics, wireless transceivers
 Moore’s Law
 More than Moore
 Better cameras
 Higher resolution and sensitivity
 Better displays
 Cheaper, flexible and more durable
 More sensitive touch, new forms of touch
 Better neural interfaces
 Better combinations of these interfaces in smart phones,
augmented reality, and virtual reality

A New Generation of Input Interfaces
Neural
Speech
Touch
Gesture
Augmented Reality
Wearable
Virtual Reality

For which applications will these
Interfaces Be Used?
 Do you remember the videos on displays in the last session
by Corning Glass?
 Many applications
 Touch screen displays in cars, offices, classrooms, retail, bus stops
 in tables, walls, automobile windows, closet doors, mirrors, MRT
maps
 Transparent displays for communication between doctors,
designers, and other professionals
 3D displays in middle of air and in forest - with augmented reality
 Smart phones, tablet computers, and e-book readers are just
the first step!And the market for them is booming!

Many of
these New
Interfaces
Target
Mobile
Devices
Number of
Mobile
Users is
Growing,
as is Time
Spent by
Users

Mobile Ad Growth is Also Rising and
Apps have Become Main Method of Access

Category Total
Software 41
E-Commerce 28
Consumer
Internet
37
Financial 15
Hardware 10
Healthcare 8
Energy 2
Space 1
Retail 1
Total 143
The Billion Dollar Startup Club Includes Many Startups
that Emphasize Mobile Devices and Their Applications
Most of them are Internet Related
(122 of 143)
Some software are mobile related
Most e-commerce and consumer
Internet are mobile apps
Some financial are mobile apps
Hardware is all mobile devices

But Not Just Consumer Applications
 Assemblers can see drawings
 Construction workers can see through walls,
wires, and pipes
 Prospectors can see through the ground
 Architects can see entire 3D image of
building
 Students can see 3D representation of
anatomy, materials, universe
 Similar systems could be useful for tourists,
shopping, soldiers, and artists

And not Just Hardware
 Software including Big Data are likely to be the big
winners
 Mobile phones continue to generate new data at a rapid
rate
 AR,VR, and wearable computing will generate vast
amounts of data for many new applications
 Who will analyze and benefit from this data?
 Which firms, which startups?
 Who will offer software and services that will protect
us and our data?
 Which firms, which startups?

Session Technology
1 Objectives and overview of course
2 How/when do new technologies become economically feasible?
3 Two types of improvements: 1) Creating materials that better
exploit physical phenomena; 2) Geometrical scaling
4 Semiconductors, ICs, electronic systems
5 Sensors, MEMS and the Internet of Things
6 Bio-electronics, Wearable Computing, Health Care, DNA
Sequencers
7 Lighting, Lasers, and Displays
8 Human-Computer Interfaces, Wearable Computing
9 Information Technology and Land Transportation
10 Nano-technology and Superconductivity
This is Eighth Session of MT5009

Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality
• Virtual Reality
• Wearable Computing
• Neural Interfaces

HCI
Human-Computer Interface (HCI)
• HCI is the technology that connects man and machine
• Robust HCIs are needed to enable ubiquitous computing
We primarily focus on input interfaces
Human Computer
Thoughts
Action
Input
Interface
Action
Recognition
Task
Execution
Understanding
Output
Interface
Sensory
Perception
Rendering

Traditional Input Interfaces
are Disappearing..
Command Line Interfaces (CLI), i.e., keyboard
Batch Interfaces
Graphical User
Interfaces
(GUI)

Performance of Input Interfaces
• Accuracy: Precision in recognizing the action
• Throughput: Information that can be processed per unit
time
• Affordability: Inversely proportional to the cost
• Ease of use: Ease in learning to use the interface
• Sociability: Multi-person interactivity
• Mobility: Size and mass of device, power consumption
• User experience: Subjective perceptions of utility &
efficiency

Comparison of Input Interfaces
Voice,AR,
Gesture,
Wearable,
VR

Key Components of Input Interfaces
Human-Computer
Input Interfaces
Neural
Interfaces
Natural UI
Speech
Micro
phone
Neural
electrodes/
sensors
GestureTouch
3D
Camera
Touch
sensor
Tracking &
Recognition
Software
Materials/
Nanotechnology
Signal
Processing
Hardware
(Semicon-
ductors)
AR,VR,
Smart Phone,
Wearable

Outline
• Overview
• Speech Interfaces, sometimes called Virtual
Assistants
• Virtual Reality
• Wearable Computing

Speech Interfaces
Key Components
Microphone
Automated Speech Recognition
(ASR) and Natural Language
Understanding (NLU) Software
Possible methods of improvement are
• Increase Signal to Noise Ratio (SNR) from microphone
• Achieve human-level performance on ASR/NLU tasks
Key dimension that needs improvement is Accuracy
Oct 2011 Video on Siri
(iPhone 4S)
http://www.youtube.com/watch?gl
=SG&hl=en-GB&v=L4D4kRbEdJw
But it didn’t succeed,
Is it ready now?

Evolution of Microphone Technology
Electret Condenser
Microphone (ECM)
Signal to Noise
Ratio (SNR): 55-58 dB
MEMS Digital Microphone
SNR: 61 dB
Flatter frequency response
Smaller size (CMOS fabrication)
Can be reflow soldered
Analog Devices, MEMS Microphone Technology, October 2010

MEMS Microphone Technology
• Wavelength of audible sound waves (> 17 mm) is
much larger than size of MEMS microphone (few
mm)
• Smaller isn’t better!! Further reductions in size can
lead to worse signal to noise ratio
• But arrays of microphones do improve signal to noise
ratio
Air gap spacing
Sound wave
Elko et al., Capacitive MEMS Microphones, Bell Labs Technical Journal, 2005
S. Beus, “MEMS mic enables thinner phone designs”, 2005

Accuracy based on Microphone Array
1. LOUD project from MIT Computer Science and Artificial Intelligence Laboratory, 2005
2. Microphone Array project in MSR: Approach and Results, Microsoft Research, June 2004
Noise suppression algorithms can increase SNR by 18dB with
just 4 microphones in an array1,2
Improvement in SNR Corresponding decrease in Word Error Rate

Multiple Microphones are Now Used in
all Devices
 Multiple microphones improve the word error rate
 Advances in bio-acoustic engineering also help; they measure
 Frequency
 Wavelength
 Sound intensity
 Other properties of the voice
 Result is that microphones may finally be good enough for
speech recognition

Automated Speech Recognition (ASR)
“Increase in vocabulary sizes needs exponential increase in
computing power due to potential combinatorial explosions”
L. Rabiner, “Challenges in Speech Recognition”, NSF Symp. on Next Gen. ASR, 2003

Automated Speech Recognition (2)
“Increase in vocabulary sizes needs exponential increase in
computing power due to potential combinatorial explosions”
L. Rabiner, “Challenges in Speech Recognition”, NSF Symp. on Next Gen. ASR, 2003

Only Acceptable in Some Niches
Worderrorrate

ASR Accuracy in Text Dictation
• Stand-alone speech interfaces may be useful for tasks like dictation
• Speech as an important modality in multimodal user interfaces
(e.g., Microsoft Kinect) may be the future
* http://blogs.msdn.com/b/sprague/archive/2004/10/22/246506.aspx
*
?

 Word error rate (WER) for SR technology in automotive has
been reduced to below <1%
Time
Source: http://whatsnext.nuance.com/in-the-labs/deep-learning-in-connected-cars/
More Recent Data on Accuracy of SR
Word Error Rate in Cars

Many Startups Have Tried: Is it Time for a Big Success?
Source: Matt M., Joshua S., and David H. 2014. Dynamic Commercialization Strategies for Disruptive Technologies: Evidence from the Speech
Recognition Industry
Total
Number
Of
Firms

Are Voice Recognition Systems (and
ASR) Now Good Enough?
 ASR, OnlineTranslation
 Google’s service translates documents with inference models that
were developed from analysis of a trillion words or 95 billion
English sentences (Big Data)
 By 2010, its dataset covered more than 60 languages and could
accept voice input in 14 languages and English is sometimes used as
a bridge between two different languages when direct translations
don’t exist
 What about voice recognition?
 Will Siri,Android, Echo or others change the way we do things? in
our cars, homes, and offices?

Siri, Android, Amazon and Others
 Car
 Plug phone into USB port
and screen displays OS
interface
 Maps, music, messaging
 Home
 Use voice to controlTV,
watch movies, play games
 Use voice to controlApple
Home Kit (e.g., lights) or
speak with Echo
http://www.wsj.com/articles/apple-carplay-review-siris-finally-on-the-right-road-1439920604
http://www.wsj.com/articles/first-look-apple-tv-heads-in-a-new-direction-1441847861
http://nyti.ms/1QWkJOk
Apple TV, Siri Remote,
Amazon Echo

Amazon’s Echo
 Voice Recognition in home
 Bluetooth (alsoWiFi) connection
 Low price
 $130Tap
 $90 Dot
 Alex is artificial intelligence based system,
that benefits from Moore’s Law
 Open to application developers
 300 apps have been developed
 Can help us control lights, music, air con
other things in “smart” home
 Compatibility with Nest and Honeywell
thermostats
Echo speaker (rear), Dot (left) and Tap (right)
devices, all powered by Alexa voice-activated
virtual assistant
http://blogs.wsj.com/digits/2016/03/03/amazon-expands-alexa-ai-with-new-tap-and-dot-voice-controlled-speakers/

Wearable Computing and Human Computer Interfaces

Performance of Siri
 Apple claims that iOS 9,
Siri will be up to 40
percent faster and 40
percent more accurate
What holds it back?
1.There is learning
curve.
2.It’s far from perfect
3.The use cases are
limited
4.Lack of integration of
third-party apps

 HIS Automotive:About 25% U.S. motorists use speech
recognition in their cars daily and 53% use it at least once
a week; by 2020, 68 million vehicles worldwide will have
voice controls, increased by 84% from 37 million in
2014.
Voice/Speech Recognition in Automotive

Voice/Speech Recognition in Home
Source: http://publications.lib.chalmers.se/records/fulltext/203117/203117.pdf
Most common
used features
Other features
that users would
like

Test of Virtual Assistants
 Tests of Google, Siri (Apple), Cortana (Microsoft),Alexa
(Amazon) and M
 Productivity tests like travel, commuting, appointments, mail
 Music related tasks – play specific songs
 Food – restaurants, deliver food
 Scores
 Google (Google) - 3.1; Siri (Apple) - 2.9
 Cortana (Microsoft) - 2.3; Alexa (Amazon) - 1.7
 But are these the tasks people want to do in the home?
 What is funny about this video? (from 0:30)
http://www.nytimes.com/2016/01/28/technology/personaltech/siri-alexa-and-other-virtual-assistants-put-to-the-test.html?ribbon-ad-idx=5&rref=
technology&module=Ribbon&version=origin&region=Header&action=click&contentCollection=Technology&pgtype=article

What is the Future of Voice Recognition?
 Will we talk to our computers in our office?
 Probably not
 Then what will we do?
 In the short term, we will probably use voice recognition
when are hands aren’t free
 Busy parents with small children in the house
 Workers using hands in factories
 In the long term, new ways of working will probably emerge
 Challenge for firms, is to find applications for which voice
recognition is important

Touch Screens
 Many kinds
 But most are variations
of either
 Resistive
 Capacitive (iPhone)
 Depend on new
materials that are
deposited on top of an
LCD display
 Processors interpret
the data http://www.youtube.com/watch?v=FyCE2h_yjxI&src_vid=5fOI-
EQCOOQ&feature=iv&annotation_id=annotation_558874

What’s Next
 What are the limitations with these touch
screens?
 Which technologies might contribute
towards overcoming these limitations?

Source for next 15 slides: Group
presentation in Fall 2014

Challenges – Sensitivity
y = 0.1383x-1.082
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
140.00%
160.00%
180.00%
200.00%
0 1 2 3 4 5 6
Percentage
Change in Raw
Count
Overlay Thickness (mm)
Sensitivity Level vs OverlayThickness
Ratio
trendline
*Note: Microprocessor doesn’t
recognize capacitance domain
but rather it register the change
in raw count
Source: FYP Industrial Collaboration (Fischer-Tech and NUS) Project Report 2013 – An Empirical Approach towards Capacitive Touch-Sensing in Functional Plastics
APPLE IPHONE
Model Type Overlay Thickness
3GS, 4, 4S Gorilla Glass 1 1.0mm
5, 5S Gorilla Glass 2 0.8mm
SAMSUNG GALAXY
Model Type Overlay Thickness
S1, S2 Gorilla Glass 1 1.0mm
S3 Gorilla Glass 2 0.8mm
S4, S5, Note 3 Gorilla Glass 3 0.4mm
1.0mm
0.8mm
0.4mm

Thinner Glass Increases Sensitivity and
Use of Smart Phones in Cold Countries
Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com and http://www.corning.com/WorkArea/showcontent.aspx?id=63819

 Low Damage Resistance
 Low Bending Strength
 Low Critical Load Bearing

New Generations of Gorilla Glass: Higher Loads
Through IOX: ion-exchanged glass and new process
https://www.youtube.com/watch?v=q4ZU7zUxdM8
Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com
*Note: Critical Load is the min. amount of load before radial cracks start to form and propagate
4500
15000
7500

New Generations of Gorilla Glass Have
Tighter Strength Distributions
(low probabilities have been eliminated)
Gorilla Glass 1 Gorilla Glass 2 Gorilla Glass 3
4 MPa 6.25 MPa 6.75 MPa

Greater Strength at all Thicknesses
65
75
85
Gorilla
Glass 1
Gorilla Glass 2
Gorilla
Glass 3

Thinner Glass Leads to Greater
Flexibility
Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957
Willow Glass

Thinner Glass Leads to Lower Bend Stress
and Failure Probability
Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957 and http://www.corning.com/WorkArea/showcontent.aspx?id=63819

But Other Layers Also Must be Flexible
 OLEDs are more flexible than are LCDs
 But other parts of the display are not flexible
 Transparent conductor
 Existing Glass
 Indium-tin oxide is inflexible and must be replaced with
new material
 Silver Nano wires?
 Carbon nanotubes, graphene?
 Glass
 need thinner glass, which is becoming feasible

Carbon Nano Tubes are More Flexible than
Indium-Tin Oxide
Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article

Games
Surgical Discussions through Glass
Needed for Google Glass, AR

LG announced
a transparent
television (30%)
Do we need
one?
Or is that an
oxymoron?
http://www.extremetech.com/computing/18
6241-lgs-flexible-and-transparent-oled-
displays-are-the-beginning-of-the-e-paper-
revolution

New Forms of Transparent Conductors are Needed
for Higher Transparency
Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article , http://www.beilstein-journals.org/bjnano/single/articleFullText.htm?publicId=2190-4286-4-12
ITO is expensive
and inflexible
Can we use:
Silver Nanowires?
CNTs?
Graphene?

New Alternatives to indium oxide?
 Thin conductive wires used for capacitive displays
 Gold and silver are very conductive but block
transmission
 Nanowalls with 3D printing
 high walls that are perpendicular to surface
 thus very conductive but don’t block ligh (80 to 500 nm)
Make very small droplets so that the cost of metals is small
 Graphene
 Infrared sensors mean no touch is needed
Economist, Jan 30, 2016, the moving finger moves on, pp. 67-68

OTHER CHALLENGES
 Existing touch screens require one to look carefully at
screen while touching a specific place
 Fingers can easily touch wrong places
 Tactus offers an overlay to existing
touch screens that facilitates proper
location of finger
 “Bubbles” rise out of the display when fingers touch the display thus
helping fingers find the “right spot”
 These bubbles are formed using MEMS (micro-electronic
mechanical systems)
 Studies have found that faster and more accurate typing are achieved
with theTactus overlay

How the Tactus System Works
Micro-channels are filled with fluid whose refractive index
matches that of top polymer layer. Thus, transparency is
even across surface.
http://www.youtube.com/watch?v=wrSKbTzc4BI 0:40-1:20

Texture Touch Displays
 Sensation of texture can provide more information for users
 This can be done using changes in vibration with
 small motors or
 transparent electrodes (Senseg) that provide information about
texture, etc. www.youtube.com/watch?v=FiCqlYKRlAA (from 0:30-2:00
minute mark)
 From IBM: http://www.youtube.com/watch?v=Gg3tmZrwbDs from 0:30 to end
 Another one from Disney: www.washingtonpost.com/
blogs/the-switch/wp/2013/10/08/disney-invents-touchscreen-that-lets-you-feel-
textures/ 0:30 to 2:00
 Early applications? 3D modeling or remote surgery can
benefit from data on texture of materials or organs

Detecting Degree of Pressure
 Apple iPhone 6s Plus (Fall 2015) detects how hard
user is pressing on screen
 Can distinguish between light tap and deep press
 New applications?
 Piano-playing app?
 New types of games?
 What types of apps might succeed?

Other Applications for Touch Screens
 Remember that there are many other applications
for displays
 Advertising displays at bus stops or MRT stations
 Mall information displays
 Self checkout in stores
 Information counter in stores
 For example, Sony’sAtracTable is
being developed for these
applications

Components of Gesture Interfaces
Key Components
2D/3D Camera
(image sensor)
Tracking,
Recognition &
Gesture
Understanding
Software
Key dimensions that need improvement are
Accuracy, Throughput and Affordability

Working of a 2D Image Sensor
http://www.cameratechnica.com/2011/11/30/five-reasons-you-may-soon-be-shooting-at-iso-50000/

Image Sensor Characteristics
• Spatial resolution: Number of pixels
• Temporal resolution: Frames per second
• Image sensor area: Size of the image sensor – area is proportional
to no. of pixels and pixel size
• Photometric exposure: light gathering ability of the sensor –
depends on the properties of the lens
• “Light available per pixel”: No. of photons incident on a pixel –
proportional to photometric exposure and pixel size
• Pixel sensitivity: is proportional to “light available per pixel”,
quantum efficiency of photodiodes, and optical efficiency
• Dark Limit & Dynamic range: Ability to detect dim details & bright
details in one image – depends on pixel sensitivity and capacity

Improvements in Image Sensors
Accuracy
• Higher spatial resolution (no. of pixels)
• Robustness to lighting changes (high pixel sensitivity,
low dark limit, and high dynamic range)
• More accurate depth sensing (lower depth error)
Throughput
• Higher frame rate
Affordability
• Smaller pixel size reduces price per pixel

Improvements in Spatial Resolution
T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010
Number of pixels (resolution) has increased, but image sensor size
has not increased because of reduction in pixel size
Year

Pixel Size vs. Sensitivity Tradeoff
CMOS-based image sensors are also expected to follow Moore’s Law
in size and cost scaling
As pixel size decreases,
“light available per
pixel” will become less,
so need higher
sensitivity pixels
Back illuminated CMOS
technology provides
better trade-off between
pixel size and sensitivity
than traditional charge
coupled device (CCD)-
based image sensors
Lightavailableperpixel(relative)

Camera Technology Improvements
Reducing pixel-size (green square) and improving sensitivity
(Yellow circle ) miniaturized cameras without reducing quality

http://www.future-fab.com/documents.asp?d_ID=4926
Improvements in Spatial Resolution, Wafer Size, and
Process Improvements Have Led to Lower Costs

But How Good are Image Sensors: Image
Sensors vs. Human Eye
Number of frames per second
Spatialresolution(cyclesperdegree)
Human Eye
Better than
Human Eye
Modern cameras are close to human eye in terms of resolution
Skorka & Joseph,
“Toward a digital
camera to rival
the human eye”, J
of Electronic
Imaging, 2011

Image Sensors vs. Human Eye
Dynamic Range (dB)
DarkLimit(cd/sq.m)
Human Eye
Better than
Human Eye
But improvement can be achieved in terms of sensitivity
Pixel sensitivity determines
the dark limit and dynamic
range (range of luminance)
of an image sensor

3D Depth Sensing Technologies
* R. Lange, “3D Time-of-flight distance measurement with custom solid-state image sensors in
CMOS/CCD-technology”, PhD Thesis, 2000

Comparison of 3D Sensing Technologies
Application Range (m)
DepthResolution(m)
Usable Range
for Gesture
Interfaces
Microsoft Kinect
• Cost-effective 3D image sensors are now becoming available
(e.g., Microsoft Kinect ~ 150 USD)
• Such cameras will further improve the accuracy of gesture UIs

3D Depth Sensing: Interferometry
• Most accurate depth sensing technology (accuracy depends only
the wavelength of light)
• Low miniaturization potential and very limited range
• Similar technique used with holography

3D Depth Sensing:
Time of Flight (used in GPS)
• Time of flight (ToF) cameras requires processors with high clock
speed (3 GHz speed can provide only 4.5 cm depth resolution)
• High miniaturization potential and large range
• Improvements in CMOS technology are likely to very beneficial

3D Depth Sensing: Triangulation (estimate
location with mobile phone base stations)
Passive Triangulation
Active Triangulation
• Limited range
• Low miniaturization potential
• Depth accuracy decreases
with square of the distance

Leap and Kinect had Some Success
http://www.youtube.com/watch?v=_d6KuiuteIA
Leap ($70) uses multiple camera sensors to recognize gestures.
Workspace is about 3 cubic meters. Better sensors will enable
larger work spaces. MIT’s Technology Review calls Leap, “The
most important new technology since smart phone…”
How about Microsoft’s Kinect?
http://www.youtube.com/watch?v=o4U1pzVf9hY
Or wearable ring (each position represents different number)?
http://www.youtube.com/watch?v=Gx3zWHS8amA
We probably won’t use gesture interfaces on our desktop
computers

Replace Cameras with MEMS-based wrist
band from Thalmic Labs, called MYO
Gestures are
recognized
before
movement
Muscle activity
is monitored with
9-axis inertial
measurement
unit (MEMS)
For this and other startups: https://angel.co/human-computer-interaction

But Big Changes May be Finally Occurring
 3D sensors are much cheaper than they used to be
 Intel’s Real Sense is in more than 25 models of laptops and will be in
Android phones by 2017
 RealSense gives 3D vision via a four-millimeter-thick strip that
includes two cameras and one processor
 By comparison, Kinect required a foot-long box that relied on Xbox’s
processors
 And the improvements in cameras and processors have not stopped!
 The new 3D sensors facilitates gesture interfaces, augmented reality,
robotics, and 3D scanning
http://www.wsj.com/articles/more-devices-gain-3-d-vision-1444859629

3D Cameras and Virtual Reality Can
Improve Video Conferencing
 Many believe that Intel’s RealSense technology will expand
video conferring
 It gives cameras in laptops the ability to see and understand depth
 Notebooks this year; tablets and phones next year?
 Might this reduce need for business travel?
 Huge time and energy expended on business travel
 Eliminates the background, enables better communication with
hands
 After RealSense, virtual reality will make the video
conferencing even better
 Are there better applications?
http://www.wsj.com/articles/virtual-reality-isnt-just-about-games-1438558372

Other Applications for 3D Cameras
 Use cameras to track eye
movements
 Monitor drivers or other
operators of machines
 Help paralyzed people use
computers
 As cost of cameras fall
 Eye tracking might become
user interface for non-
paralyzed
 Eye tracking can also be used
with Google Glasses (see
below)
Source: http://www.economist.com/news/technology-quarterly/21567195-computer-interfaces-ability-determine-location-persons-gaze

Types of Augmented Reality
Glasses
Phones

How Different from Virtual Reality?
 Augmented Reality Supports our understanding of
the real world while virtual reality immerses us in a
new type of world
 Many propose combination of phone and real world,
using mobile phone’s camera
 Other cameras can also be used, like with Google
Glass (see below)
 These slides focus on mobile phones
 Many startups are pursuingAR
https://angel.co/augmented-reality

What do you see
through the camera lens
or google glasses?

Much of this information can be obtained from Google Earth

Handheld devices may be
sufficient, particularly if
the images are easily
integrated with your
surroundings

What about a or a virtual one
Supermarket? at a subway station?
http://www.youtube.com/watch?v=yKNSOwLcrkE

What about superimposing the images on a
car’s windshield?

Google’s Project Glass
 Image and information are displayed on the glasses
 Users choose which information to display on the glasses
 Choice controlled by voice, remote control, or maybe thoughts in future
 Help you manage your day? http://www.youtube.com/watch?v=9c6W4CCU9M4
 Help you talk to people? https://www.youtube.com/watch?v=t-m6YL64lkU
 Improvements in ICs, displays, other components are leading to
better performance and cost of glasses
 But maybe they won’t help men find a girlfriend
http://www.youtube.com/watch?v=8UjcqCx1Bvg
 Unless you can access data without talking (e.g., tilting one’s head,
touching the device, or blinking your eyes)
 Best ofAll – Cost is Less than $100 – set high price of $1500

Translation
 Because of advances in bioacoustic engineering measuring the
frequency, wavelength, sound intensity and other properties
of the voice, the software in the cloud connected to the
earpiece in your ear will re-create the voice of the speaker,
but speaking your native language.
 While I was traveling recently in Ukraine, a group of
engineering students showed me a shiny black-and-blue robot
glove called EnableTalk that uses flex sensors in the fingers to
recognize sign language and to translate it to text on a
smartphone via Bluetooth.The text is then converted to
speech, allowing the deaf and mute person to “speak” and be
heard in real time.
http://www.wsj.com/articles/the-language-barrier-is-about-to-fall-1454077968

Google Glass in Factories (1)
 Replace stacks of wiring instructions
with Google Glass display
 Test programs are being conducted at
Boeing, Daimler, United Parcel, and
others
 Boeing workers use them to assemble
wire-harnesses
 When assembler reads out loud coding
on wire, correct hole on electronic
version lights up and flashes, providing
easy to follow guide
 Error rates and assembly time have
fallen
http://www.wsj.com/articles/smart-glasses-get-new-
look-on-factory-floor-1433301177

Google Glass in Factories (2)
 Deutsche Post is working with DHL and Ricoh
 Warehouse workers process orders for parts and
equipment by scanning bar codes on cartons with their
glasses
 Eliminates need for hand-held bar-code scanners and paper
invoices
 Instructions are relayed through glasses
 Daimler uses them on assembly lines
 Glasses provide checklists so workers don’t have to memorize or hold
paper lists
 Discovered defects are immediately reported through voice-recorded
report with photos
 Safety and security are issues
http://www.wsj.com/articles/smart-glasses-get-new-look-on-factory-floor-1433301177; http://www.wsj.com/articles/dhl-unit-plans-google-glass-experiment-in-us-warehouses-1439568950

Needed Improvements
 Faster processors
 Position of superimposed information must be changed each time
user moves her head
 Faster processors needed to do this
 Desktop computers have sufficient speeds, but not current smart
phones
 Transparent Displays
 Current displays block too much view
 Transparent displays are need so less view is blocked
 Better programming tools
 Will reduce application development costs
 But many of these improvements are already available in the latest
products
Source: conclusions from presentation by MT5009 students; Augmented Reality: Applications in Manufacturing and Maintenance

Engineering Applications for Magic Leap

https://www.youtube.com/watch?v=kw0-JRa9n94

Virtual Reality is becoming economically feasible because of improvements in:
Motion tracking with lasers and LEDs, 3D capture, hi-resolution displays,
fast graphic chips, deep library of 3D software
http://www.wareable.com/oculus-rift/how-oculus-rift-works. http://www.wsj.com/articles/virtual-reality-isnt-just-about-games-1438558372

Combines
Oculus Rift’s
Software with
Samsung Galaxy
Note 4
(about 300USD)
http://www.samsung.com/global/
microsite/gearvr/gearvr_specs.html
Oculus Rift
succeeded by
using standard
components.
Efforts by previous
companies had
focused on special
components

Others have Entered the Market
 DodoOne, ZeissVR One are low-end devices
http://www.wsj.com/articles/a-beginners-guide-to-virtual-reality-1426883929
 Google plans to introduce a low-end devices –
that look like cardboard box, 40USD? http://nyti.ms/1SEGYdH
 Sony has introduced MorpheusVR headset and
games http://www.wsj.com/articles/a-beginners-guide-to-virtual-reality-1426883929
 Microsoft introduced HoloLens, which uses
holography (see display slides, or is this AR?) 3000
USD? http://www.vrworld.com/2015/07/17/introducing-sonys-project-morpheus/
 Can be used to play games
 Or to interact with the world
https://www.youtube.com/watch?v=C3rNIxMlKmI
 From 3:00 to 5:00
 For your projects, can you explain the economics
ofVR headsets or new services?
 Will they become cheap enough so that most are high-
end?
 What services will be useful?

Other Possible Applications
 Enables users to take virtual tours of anything with Hi-End
VR devices
 Software companies like Matterport and others* are making
it possible to
 rapidly and cheaply digitize the interior of building and then walk
through it
 this includes hazardous environments such as nuclear power plants,
tourist locations such as the GreatWall, or the Louvre
 Change all 2D pictures and video into 3D – immersive captures of
entire spaces and events
 Imagine a version of Google Maps that doesn’t end at the
front door
 VR services will probably be bigger market thanVR hardware
*https://angel.co/3d-technology

Reduce the need for plywood mock ups and allow architects
to change plans and allow clients an immersive view
Also Enables Better Architectural Design

Examples of 3D CAD for Buildings
Left are real buildings, Right is 3D CAD

Iris VR Architecture Demo Walkthrough
https://www.youtube.com/watch?v=BjEcA2p_mMY
Iris VR CAD can do even better:
Allowing Users to See the Inside of the Building

VR for Movies
 Many projects proposed at
film festivals such as
Sundance andTribeca
 10 minute movies
 New features that go
beyond 3D
 User controls viewpoint,
can look at movie from any
angle
http://www.wsj.com/articles/virtual-reality-movies-get-ready-for-the-vr-revolution-1457030357?mod=WSJ_TechWSJD_topRight

Outline
• Overview
• Virtual Reality
• Wearable Computing Main Source: A Rajaraman, B Madhumita, Mayank Tewari, D
Nelson, S K Rao, Wearable Technology Design, Spring 2015, many startups: https://angel.co/personal-health

Where will the Devices be Attached?

Where will the Devices be Attached?
 Different data can be collected from different parts of the
body
 Data can be viewed better on some parts of the body
 What does this mean for the products and services that will
succeed?
 Services and software will be an important part of wearable
computing
 Not just hardware!
 Lets look at current products and where they are attached
 Future projects should probe deeper, providing a better idea about
the best places to attach devices

Existing Products, by Body Part
Arm
Body
Leg
Head

Jawbone
valued at
$3.3 Billion

What Should we Measure?
 Some people focus on the number of
steps
 10,000 steps a day is common
 Others focus on heart rate
 Raise your heart rate above certain
level, say 150 for 30 year-old person
 Some devices try to integrate both
measures into a new measure
 http://www.wsj.com/articles/stop-counting-10-000-steps-check-your-personal-
activity-intelligence-1453313834?mod=LS1

Can this Analysis be taken Further?
 What are the costs/prices?
 How much will they come done and how fast?
 What are the benefits and where are they the largest?
 Can we quantify the potential benefits from specific
wearables?
 Do specific sensors (Session 6) work with specific
wearables?
 Which sensors are getting cheaper and better?
 Can we identify where the largest benefits might be?
 Can we use this information to design better wearable
devices

Perhaps More Importantly
 New forms of software are needed for wearable computing
 Operating systems
 Cloud computing
 Open source
 What startups will do this and be valued at more than $1 Billion
 New forms of software services are needed
 Big data
 Consumer Internet
 What startups will do this and be valued at more than $1 Billion

One example of New Software
 Will new forms of passwords be needed for wearable
equipment
 Fingerprints and iris scans require special equipment, thus
increasing the weight of wearable equipment
 Ballisocardography is study of body’s movements in response
to the activity of the heart
 Body shifts slightly as heart beats
 Unique to individuals
 Shifting Identity, Economist, June 20, 2015, p. 76

Outline
• Overview
• Virtual Reality
• Wearable Computing Main Source: A Rajaraman, B Madhumita, Mayank Tewari, D
Nelson, S K Rao, Wearable Technology Design, Spring 2015

Neural Interface: Can we use our
thoughts to control machines?
Key Component
Brain scanning device
Key dimensions that need
improvement are Accuracy,
Throughput and Affordability
Required improvements in brain scanning technology
Accuracy & Throughput – Higher spatial and temporal
resolution
Affordability – Size and better materials

Current State of Art
 Best systems enable a person to control a robot or cursor or type
one letter a minute with their “mind”
 For paralyzed, very expensive invasive systems that require training
 http://www.youtube.com/watch?v=C7H_M8-dBHc (0-1 minute)
 http://www.youtube.com/watch?v=cDiWFcA0gaw&playnext=1&l
ist=PL7FD931F8953A0F87&feature=results_main (0:15-2:45
minute)
 Accessory for your iPhone
 $99 device measures your
brain waves.App displays
data on phone
• Basically a toy
http://singularityhub.com/2011/01/07/iphone-accessory-from-
xwave-channels-your-brain-waves-to-the-iphone/

SPECTEEG
1936 1950 1972 19751968
CT Scan
1983
MEG
1991
fMRINIRS
1973
MRI PET
US$2.9M
US$1M-1.5M
US$250K US$2.4M US$0.5M-3M
US$180K- 250K>US$30K
Non-Invasive Brain Scanning
Electro
Encephalo
Graphy
Magneto
Encephalo
Graphy
Near-
Infra Red
Spectros
copy
functional
Magnetic
Resonance
Imaging
EEG & MEG directly measure neuronal activity, NIRS & fMRI measure blood activity

Activities can be Linked to Brain Locations
EEGMEG
By knowing which
parts of the brain
are used for which
activities, can
interpret the
electrical and
magnetic fields
But need high
resolution

Where we Are for Resolution (1)
 Ideally, a non-invasive technology with high spatial resolution
and high temporal resolution is required
 Additionally, the technology must be affordable and portable in
order to be useful in HCI applications
Gerven, M. v., et al., “The Brain-Computer Interface Cycle”, J. Neural Eng, 2009
Non-invasive
Neuron can fire ~0.1mm (spatial) & ~10 ms (temporal)
Invasive

Where we are, and where we want to be (2)
(in 60 years?)
http://singularityhub.com/2011/01/07/iphone-accessory-from-xwave-channels-your-brain-waves-to-the-iphone/
Invasive
techniques

Spatial Resolution Improvement
While spatial resolution is important for accuracy, high temporal
resolution is also critical for user interfaces
R. Kurzweil, “The Singularity is Near”, 2005
fMRI

EEG Challenges
 Key limitation: Poor spatial
resolution
Increasing number of EEG
electrodes may provide
limited improvement in spatial
resolution and higher SNR
J. Malmivuo, “Comparison of the Properties of EEG and MEG”, Intl J of Bioelectromagnetism , 6 (1), 2004

MEG Challenges: lots of magnetic noise
Baranga, A. B.-A. (2010). "Brain's Magnetic Field: a Narrow Window to Brain's Activity".
Electromagnetic field and the human body workshop, (pp. 12).
fT

Invasive Techniques
Chips are implanted into
a person’s brain using
electrodes, basically MEMS
(remember Session 6 and
eye retinal implants)

Source: Stevensen I, Kording K 2011, How Advances in Neural Recording Affect Data Analysis,
Nature Neuroscience 14 (2): 139-142
Improvements in Invasive
Techniques

Source: Stevensen I, Kording K 2011, How Advances in Neural Recording Affect Data Analysis,
Nature Neuroscience 14 (2): 139-142

But the improvements are probably not
occurring fast enough
 At current rates of improvement, in 220 years we
will be able to simultaneously record all 100 billion
neurons
 On the other hand,
 maybe we don’t need to record all of them simultaneously
because we will find ways to interpret the data
 Or maybe we will find better non-invasive techniques

Conclusions
•Rapid improvements are occurring in HCI
•Most of these improvements are being
driven by improvements in ICs, MEMS,
lasers, and displays (reductions in scale)
•Speech recognition (microphones,
processors)
•Gesture interfaces (cameras)
•Neural interfaces (electrodes)
•Other improvements are being driven by
creation of new materials
•Touch screens

Conclusions (2)
• Is voice recognition ready to take off?
• Apple Siri, Amazon Echo
• How about gesture interfaces?
• More optimistic about
• AR – revolution for workers?
• VR – games, education
• Wearable computing – large numbers of
startups

Conclusions (3)
• All of these devices will generate data
• Who will own this data?
• How will this data be generated
• Wearable computing may provide the most
valuable data
• This was discussed in session 6 on health care
• Who will provide the software and data
analysis techniques

Session 8 Topics for Write-ups
 Identify all the entrepreneurial opportunities for one
of the following technologies
 Google glass
 Gesture interface
 Health data recorded with wrist device
 Augmented reality with cameras and phones

Wearable Computing and Human Computer Interfaces

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Wearable Computing and Human Computer Interfaces (20)

More from Jeffrey Funk (20)

Recently uploaded (20)

Wearable Computing and Human Computer Interfaces