![confidential
[j, d]j=2pp
jmeas
Vacf,amplitude
Voffset
convolutedsignal
S0 S1 S2 S3
2
20
2
13, SSSSV amplacf
[V]
Information derived in SW from raw data of each pixel
Distance information
Amplitude information
Intensity information
Voffset
2D b/w image based
on modulated light
2D b/w image based on ambient light
distance @ j>0, d ~ j
S0…3 measurement points of signal amplitude
20
13
arctan
SS
SS
meas
=j
13
Depth map
On chip On host system/ in AP
© 2017 pmdtechnologies ag Infineon Technologies AG](https://image.slidesharecdn.com/awemunich2017pmdv0118-171102184017/85/Tobias-Rothermel-pmd-technologies-pmd-ToF-the-Swiss-Army-Knife-of-3D-Sensing-13-320.jpg)
![confidential
© 2017 pmdtechnologies ag Infineon Technologies AG19
Royale - Predefined use cases for the CamBoard pico flexx
Use case
Range
[m]
Frame
rate
Exposure
time (us)
Modulation
Frequencies
Indoor room reconstruction 1.0 – 4.0 5 fps 2000 80 & 60 MHz
Indoor navigation 1.0 – 3.0 10 fps 1000 80 & 60 MHz
3D object reconstruction 0.5 - 1.5 15 fps 700 80 & 60 MHz
Face reconstruction 0.3 – 1.0 25 fps 450 80 & 60 MHz
Small object recognition 0.3 – 2.0 35 fps 600 60 MHz
Hand tracking 0.1 – 1.0 45 fps 500 60 MHz
4 m 3 m 2 m 1 m 0 m
TANGOIndoor room reconstruction (5 frames/s, 2 ms int. time)
Room scanning, indoor navigation (10 frames/s, 1 ms int. time)
3D object reconstruction (15 frames/s, 0.7 ms int.)
Medium size object recognition, face reconstruction (25 frames/s, 0.45 ms int. t)
Remote collaboration, table-top gaming (35 frames/s, 0.6 ms int. time)
Hand tracking (45 frames/s, 0.5 ms int.)](https://image.slidesharecdn.com/awemunich2017pmdv0118-171102184017/85/Tobias-Rothermel-pmd-technologies-pmd-ToF-the-Swiss-Army-Knife-of-3D-Sensing-19-320.jpg)
Tobias Rothermel (pmd technologies): pmd ToF – the Swiss Army Knife of 3D Sensing
- 1. c o n f i d e n t i a l© 2017 pmdtechnologies ag Infineon Technologies AG pmdtechnologies – 2017-10-04 – V0118 pmd ToF the Swiss army knife of 3D sensing AWE Munich 2017
- 2. confidential © 2017 pmdtechnologies ag Infineon Technologies AG2 Why is the guy with the beard on the stage?
- 3. confidential To show that with pmd’s ToF cameras (e.g. CamBoard pico flexx) you can do this: © 2017 pmdtechnologies ag Infineon Technologies AG3
- 4. confidential © 2017 pmdtechnologies ag Infineon Technologies AG4 IR greyscale, Depthmap and point cloud
- 5. confidential © 2017 pmdtechnologies ag 5 … and this… Color scheme: normal map color dependent on surface direction Infineon Technologies AG
- 6. confidential 6 … and much more Secure face recognition 3D face scanning Inside-out- tracking for 6DoF self localization Hand and finger tracking gestures AR/VR/MR Surveillance & People counting Robotics Computational photography Drones Automation © 2017 pmdtechnologies ag Infineon Technologies AG
- 7. confidential © 2017 pmdtechnologies ag Infineon Technologies AG7 Pico family https://www.youtube.com/watch?v=BHunyOy4A6c
- 8. confidential 8 Google Tango Asus Zenfone AR „Giving mobile devices a human scale understanding of space and motion“ Combination of sensors Depth sensing 2D camera Fisheye camera IMU Integrated pmd depth sensor Up to 4m range 1% of distance = max. depth error 5 fps <300mW power consumption 38k pixels ToF module size 1cm x 2cm © 2017 pmdtechnologies ag Infineon Technologies AG
- 9. confidential 9 Tango use cases Room Scanning Augmented Reality Gaming © Google © Google Indoor Mapping and Navigation Utility © 2017 pmdtechnologies ag Infineon Technologies AG
- 10. confidential 10 Infotainment Gesture control Driver Monitoring In cabin sensing Passenger detection Pedestrian Detection Autonomous parking Gesture & obstacle detection Automotive © Google With friendly permission of © 2017 pmdtechnologies ag Infineon Technologies AG
- 11. confidential • CamBoard pico flexx = ToF based depth sensing USB camera from pmd with an Infineon 3D image sensor inside • ToF = Time of Flight • pmd = photonic mixer device = the pixel • pmd = pmdtechnologies ag • + = best friends ;) © 2017 pmdtechnologies ag Infineon Technologies AG11 First things first
- 12. confidential Emission of electrically modulated light Electrical modulation signal Detection of the reflected modulated light pmd Time-of-Flight (“indirect” ToF) Convolution of reflected modulated and the emitted light Autocorrelation of the emitted and the reflected light Goal: Determine where the “peak” is it is the phase delay, which corresponds to the distance © 2017 pmdtechnologies ag 12 Infineon Technologies AG
- 13. confidential [j, d]j=2pp jmeas Vacf,amplitude Voffset convolutedsignal S0 S1 S2 S3 2 20 2 13, SSSSV amplacf [V] Information derived in SW from raw data of each pixel Distance information Amplitude information Intensity information Voffset 2D b/w image based on modulated light 2D b/w image based on ambient light distance @ j>0, d ~ j S0…3 measurement points of signal amplitude 20 13 arctan SS SS meas =j 13 Depth map On chip On host system/ in AP © 2017 pmdtechnologies ag Infineon Technologies AG
- 14. confidential 14 What is inside a ToF camera? VCSEL cover housing VCSEL USB Chip ToF Chip, lens holder, lens thermal pad housing © 2017 pmdtechnologies ag Infineon Technologies AG
- 15. confidential 15 CamBoard pico flexx - What you will get © 2017 pmdtechnologies ag • 3D resolution: 38k • eye safety class 1 • VCSEL (laser) illumination • measurement range: 0.1 to 4 m • optimized opening angle: 62°x 45° Hardware • one SW for all platforms: Win, Linux, Android, MacOS • easy approach: predefined use cases for kick start eval. • systematically tested on product level Software „Royale“ • Interface: drivers for USB available • GUI: QT based for 2D and 3D presentation • improved API: threading handled by Royale consistently Interfaces Infineon Technologies AG
- 16. confidential Approaches to the Royale software packages 16 CamBoard pico flexx: Royale library royale viewer royale API Purpose: getting started Purpose: reference source code for device integration © 2017 pmdtechnologies ag Infineon Technologies AG
- 17. confidential Structure of the API 17 Royale - API User of camera data (camera HAL) royale Camera Driver deliver raw data request sensor data get depth data configure camera device CameraManager ICameraDevice CameraManager Main component which can queue commands for connected cameras ICameraDevice Interface which represents the camera IDepthDataListener Interface to be implemented in the HAL to retrieve the depth data DepthData Contains the processed depth data © 2017 pmdtechnologies ag Infineon Technologies AG
- 18. confidential Light-weight framework for time-of-flight cameras Joint development by pmd and Infineon C++ interface C++11 based Platform independent Windows Linux Mac OS X Android ARM Linux ARM Matlab interface C# interface Examples to connect to OpenCV OpenNI © 2017 pmdtechnologies ag Infineon Technologies AG18 Royale - Purpose
- 19. confidential © 2017 pmdtechnologies ag Infineon Technologies AG19 Royale - Predefined use cases for the CamBoard pico flexx Use case Range [m] Frame rate Exposure time (us) Modulation Frequencies Indoor room reconstruction 1.0 – 4.0 5 fps 2000 80 & 60 MHz Indoor navigation 1.0 – 3.0 10 fps 1000 80 & 60 MHz 3D object reconstruction 0.5 - 1.5 15 fps 700 80 & 60 MHz Face reconstruction 0.3 – 1.0 25 fps 450 80 & 60 MHz Small object recognition 0.3 – 2.0 35 fps 600 60 MHz Hand tracking 0.1 – 1.0 45 fps 500 60 MHz 4 m 3 m 2 m 1 m 0 m TANGOIndoor room reconstruction (5 frames/s, 2 ms int. time) Room scanning, indoor navigation (10 frames/s, 1 ms int. time) 3D object reconstruction (15 frames/s, 0.7 ms int.) Medium size object recognition, face reconstruction (25 frames/s, 0.45 ms int. t) Remote collaboration, table-top gaming (35 frames/s, 0.6 ms int. time) Hand tracking (45 frames/s, 0.5 ms int.)
- 20. confidential © 2017 pmdtechnologies ag Infineon Technologies AG20 Thank you!
- 21. confidential ToF, pmdtechnologies and Infineon Dresden 2002 • pmdtechnologies is founded • HQ Siegen 2005 • efector pmd • first mass product based on pmd ToF technology • Industrial sensor developed with ifm electronic 2011 • New pmd branch in Dresden, Germany 2012 • More than 500,000 pmd sensors sold to date • Ramp up of first commercially available ToF imager: pmdPhotonICs 19k-S3 2013 • ifm electronic becomes 100% shareholder • Infineon and pmd announce jointly developed 3D image sensor family 2016 • New pmd branch in San Jose, USA 2016/17 • ToF Tango phones on the market • ToF HMDs on the market © 2017 pmdtechnologies ag 21 Infineon Technologies AG
- 22. confidential © 2017 pmdtechnologies ag 22 How do pmd and Infineon cooperate? Infineon Technologies AG
- 23. confidential Consumer Automotive Industrial © 2017 pmdtechnologies ag 23 What is pmdtechnologies? target • pmd: fabless IC company Time-of-Flight pixel design + ToF camera design +13 years expertise/ IP/ key patents on ToF + calibration know-how • Collaboration: Infineon (production & distribution) jointly developed ToF image sensor IC • Mother: ifm electronics successfully sells pmd-based industrial sensors since more than 10 years. Infineon Technologies AG
- 24. confidential Chip Design System Support • Lens • Illumination • Calibration • Production bring-up 3D ToF Camera Reference Designs © 2017 pmdtechnologies ag 24 What is pmd doing? • High integration SoC • Low power • Outdoor robust Infineon Technologies AG