Designing Apps for Intel RealSense Technology
4 likes4,686 views
The document provides user experience guidelines for designing applications utilizing Intel® RealSense™ technology, focusing on both the F200 short-range and R200 long-range cameras. Key topics include effective usability testing, input technology selection, user feedback mechanisms, and API utilization for capturing volumes and tracking. It emphasizes the importance of understanding user needs and iterative testing to enhance application performance in various environments.
![26
APIs to Detect Speed and Precision
• Joint Speed – Enabling joint speed allows developers to obtain the
absolute/average speed of motion of each joint
- PXCMHandConfiguration.EnableJointSpeed
- Disabled by default to preserve CPU and memory resources. Only enable as necessary.
• Hand jitters can hamper tracking. Use PXC[M]Smoother to reduce jitter
F200](https://image.slidesharecdn.com/sf15sfts018slideshare-150822213117-lva1-app6892/85/Designing-Apps-for-Intel-RealSense-Technology-26-320.jpg)
![27
APIs to Detect Speed and Precision
• Use PXCMHandData.TrackingStatusType to obtain tracking status for speed
- TRACKING_STATUS_HIGH_SPEED
• If using hand gestures to control the OS using Touchless Controller, use
the PXC[M]TouchlessController member functions to set pointer and scroll
sensitivity:
- SetPointerSensitivity
- SetScrollSensitivity
• For face detection, fast movements may lead to lost tracking. Use
PXCMFaceData.AlertData.AlertType – ALERT_FACE_LOST to determine if
tracking is lost
F200](https://image.slidesharecdn.com/sf15sfts018slideshare-150822213117-lva1-app6892/85/Designing-Apps-for-Intel-RealSense-Technology-27-320.jpg)
![46
Checking Scene Quality
• Use PXC[M]ScenePerception:CheckSceneQuality to check whether the
stream quality is suitable to start scene perception
• Returns a value between 0 and 1. Higher the return value, better the
scene
• Code:
R200](https://image.slidesharecdn.com/sf15sfts018slideshare-150822213117-lva1-app6892/85/Designing-Apps-for-Intel-RealSense-Technology-46-320.jpg)
![48
Setting Voxel Resolution in Your Application
• Use PXC[M]ScenePerception:SetVoxelResolution/QueryVoxelResolution
• The PXC[M]VoxelResolution enumerator itemizes the supported voxel
resolutions
Name Description
LOW_RESOLUTION The low voxel resolution. Use this resolution in
a room-sized scenario (4/256m).
MED_RESOLUTION The median voxel resolution. Use this resolution
in a table-top-sized scenario (2/256m).
HIGH_RESOLUTION The high voxel resolution. Use this resolution in
a object-sized scenario (1/256m).
R200](https://image.slidesharecdn.com/sf15sfts018slideshare-150822213117-lva1-app6892/85/Designing-Apps-for-Intel-RealSense-Technology-48-320.jpg)
Designing Apps for Intel RealSense Technology
- 2. 2 Designing Apps for Intel® RealSense™ Technology: User Experience Guidelines with Examples for Windows® Kevin Arthur – Senior User Experience Researcher, Intel Corporation Meghana Rao – Developer Evangelist, Intel Corporation SFTS018
- 3. 3 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 4. 4 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 5. 5 F200 Short Range Camera • Integrated into Laptop and All-In-One Devices • Front, user-facing configuration F200
- 6. 6 F200 Features • Features available with the Intel® RealSense™ SDK - Hand tracking and gestures - Face recognition and animation - Background replacement - 3D scanning - Speech F200
- 7. 7 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 8. 8 UX Guidelines Learned from F200 Apps • Dozens of rounds of 1-on-1 usability testing with prototypes • Co-design with select software vendors Selected apps from the Intel® RealSense™ App Showcase https://appshowcase.intel.com/realsense/ F200
- 9. 9 What the Camera Sees: Know the Effective Range and Capture Volume Effective Range for Gestures 0.2 - 0.6 m Effective Range for Face Tracking 0.35 - 1.2 m RGB Resolution Up to 1080p at 30 fps Depth Resolution Up to 640x480 at 60 fps F200
- 10. 10 Provide Visual Feedback About Range Distance Prompts World Diagrams F200
- 11. 11 Provide Continuous Feedback When Needed User Viewport Move It! by Umajin F200
- 12. 12 Provide Continuous Feedback When Needed User Overlay Kagura by Shikumi Design F200
- 13. 13 User Input: Choose the Right Technology Hand Gestures • Fun and engaging for short experiences • Natural and unencumbered Mouse • Precise, fast, and efficient • Comfortable for long-term use Understand the strengths of input technologies. Use gestures to enhance, not replace. Touch • Simple and direct F200
- 14. 14 Follow Gesture Best Practices to Minimize Fatigue • Design for left-right or arc gesture motion • Avoid actions that require your users to lift their hand above the height of their shoulder • Design for breaks, and break up activities into small, short actions • Do not require many repeating gestures • Do not require precise input, like entering text or selecting small items • Continuous and low-risk actions are best for gesture F200
- 15. 15 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 16. 16 APIs To Detect Capture Volume • Developers should use device neutral functions to obtain Field of View (FOV) frustrum information to help guide users through visual feedback • Device interface – QueryColorFieldofView and QueryDepthFieldofView functions allow you to obtain the Depth and Color capture volumes Code: F200
- 17. 17 APIs To Detect Capture Volume Other Device interface functions of interest: • PXCMCapture.Device. QueryDepthSensorRange – returns depth sensor range in mm • PXCMCapture.Device. QueryColorFocalLengthMM – returns color sensor focal length in mm • PXCMCapture.Device. QueryDepthFocalLengthMM – returns depth sensor focal length in mm F200
- 18. 18 Alerts to Indicate Face Out of the Field of View • PXCMFaceData.AlertData.AlertType • AlertType enumerator itemizes alert events Alert Type Description ALERT_NEW_FACE_DETECTED A new face is detected ALERT_FACE_NOT_DETECTED There is no face in the scene. ALERT_FACE_OUT_OF_FOV The face is out of camera field of view. ALERT_FACE_BACK_TO_FOV The face is back to field of view. ALERT_FACE_LOST Face tracking is lost F200
- 19. 19 Coding Face Alerts in Your App F200
- 21. 21 Alerts to Indicate Hands Out of FOV Use PXCMHandData.TrackingStatusType Alert Name Description ALERT_HAND_OUT_OF_BORDERS A tracked hand is outside of a 2D bounding box or 3D bounding cube defined by the user ALERT_HAND_INSIDE_BORDERS A tracked hand has moved back inside the 2D bounding box or 3D bounding cube defined by the user. ALERT_HAND_TOO_FAR A tracked hand is too far to the camera ALERT_HAND_TOO_CLOSE A tracked Hand is too close to the camera. ALERT_HAND_DETECTED A tracked hand is identified and its mask is available ALERT_HAND_NOT_DETECTED A previously detected hand is lost, either because it left the field of view or because it is occluded. And more… Refer to the documentation F200
- 22. 22 Coding the Hand Alerts in Your App F200
- 24. 24 World Space to Screen Space Mapping – Projection Interface • The Projection interface provides functions to map or project among color, depth and world coordinates - MapColorToDepth - MapDepthToColor - ProjectCameraToColor - ProjectCameraToDepth - ProjectColorToCamera - ProjectDepthToCamera F200
- 25. 25 Adjusting UI to Different Screen Resolutions – Unity To make the App go Full-screen in windowed/unwindowed mode, whatever the resolution of the display: • Pre-compilation, in player’s settings, make sure the "Use Direct3D 11" box is checked • Set it to use ONLY specific aspect ratios (16:9 and 16:10 and matching children) Result - when the ‘resolution selection screen’ appears: • If the “Windowed” check box is UNchecked, it will always start at Full Screen • If it’s checked, it will start windowed but can be resized/maximized F200
- 26. 26 APIs to Detect Speed and Precision • Joint Speed – Enabling joint speed allows developers to obtain the absolute/average speed of motion of each joint - PXCMHandConfiguration.EnableJointSpeed - Disabled by default to preserve CPU and memory resources. Only enable as necessary. • Hand jitters can hamper tracking. Use PXC[M]Smoother to reduce jitter F200
- 27. 27 APIs to Detect Speed and Precision • Use PXCMHandData.TrackingStatusType to obtain tracking status for speed - TRACKING_STATUS_HIGH_SPEED • If using hand gestures to control the OS using Touchless Controller, use the PXC[M]TouchlessController member functions to set pointer and scroll sensitivity: - SetPointerSensitivity - SetScrollSensitivity • For face detection, fast movements may lead to lost tracking. Use PXCMFaceData.AlertData.AlertType – ALERT_FACE_LOST to determine if tracking is lost F200
- 28. 28 Bounding Boxes • Provide clear visual cues to the user on the source and destination areas for object of interaction • Bounding boxes provide a rough boundary for user action thus alleviating the need for high precision • Hand bounding boxes: - PXCMHandData.IHand.QueryBoundingBoxImage - Use PXCMHandData.AlertType – ALERT_HAND_OUT_OF_BORDERS to detect if hand is out of the bounding box • Face bounding boxes: - PXCMFaceData.DetectionData.QueryBoundingRect F200
- 29. 29 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 30. 30 R200 Long Range Camera • Integrated into Tablets and Two-In-One Detachable Devices • Rear, world-facing configuration R200
- 31. 31 R200 Features • Features available with the Intel® RealSense™ SDK - 3D scanning - Depth-enabled photo and video - Measurement - Gaming and play with mixed and augmented reality Scene perception module Camera tracking and localization Mesh reconstruction - Face detection and tracking - Speech (Windows® SDK only) R200
- 32. 32 Video – ToyZ Game Prototype • Shows real-time scene perception for collision and occlusion (no pre- scan) • By Shachar Oz, Omek Studio at Intel R200
- 33. 33 Video – Procedural Island Prototype • Shows scan as part of “capture and play” • Illustrates procedural shaders and set dressing • By Eddy Ortega, Garrett Stevens, Perceptual Computing at Intel R200
- 34. 34 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 35. 35 R200 Range and Capture Volume R200 Effective Range 0.6 - 3.5 m indoors RGB Resolution Up to 1080p at 30 fps Depth Resolution Up to 640x480 at 60 fps
- 36. 36 36 3D Scanning: Plan for the Scene Small object capture: Instruct users to place object on flat surface and move around the object to capture it from side and top views Person capture: The subject must remain still while the user walks around them and captures side views Scene capture: User will scan the tablet around the room as if taking a panoramic photo Focus 3D scanning applications on one of these 3 scenarios, as the interaction flow will differ according to the size and surfaces being captured R200
- 37. 37 User Prompts • Give simple instructions to start • Preview the subject or area that will be captured R200
- 38. 38 User Prompts • Tell user if tracking is lost • Give distance cues R200
- 39. 39 Augmented Reality: Give People a Reason to Move, or They Won’t • Tablet as window into a virtual space vs. tablet as fixed screen • Address with motion hints – explicitly with instructions or implicitly with content Google Spotlight Stories “Windy Day” R200
- 40. 40 Minimize Fatigue from Holding the Device Active Camera Mode: Tiring Inactive Camera Mode: Less tiring Support both active and inactive camera modes R200
- 41. 41 Choose the Right Mode for the Activity Active Camera Mode Touch interaction less comfortable, less precise. Best for short capture interactions. Inactive Camera Mode Touch interaction more comfortable, more precise. Best for extended play interactions. Touch Zones Touch Zones R200
- 42. 42 Two Styles of Mixed-Reality Games R200
- 43. 43 Plan for the Scene and Camera Qualities • Understand the camera limitations. Depth data is less accurate on - Very bright areas - Clear glass - Black surfaces • Give relevant feedback when problems are detected • Fail gracefully and don’t prevent play R200
- 44. 44 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 45. 45 APIs to Obtain R200 Camera Device Information • The Device interface provides the following functions: API Description QueryDSLeftRightCropping SetDSLeftRightCropping Access to the left/right image cropping mode. QueryDSLeftRightExposure QueryDSLeftRightExposureInfo SetDSLeftRightAutoExposureSetDSLeftRightExposure SetDSLeftRightExposure Access to the left/right stream exposure value. QueryDSLeftRightGain QueryDSLeftRightGainInfo SetDSLeftRightGain Access to the left/right stream gain adjustment. QueryDSMinMaxZ SetDSMinMaxZ Access to the distance value range. R200
- 46. 46 Checking Scene Quality • Use PXC[M]ScenePerception:CheckSceneQuality to check whether the stream quality is suitable to start scene perception • Returns a value between 0 and 1. Higher the return value, better the scene • Code: R200
- 47. 47 Checking Tracking Accuracy • Use PXCScenePerception::TrackingAccuracy to obtain tracking accuracy data - The TrackingAccuracy enumerator itemizes the tracking accuracy definitions Name Description HIGH High tracking accuracy LOW Low tracking accuracy MED Median tracking accuracy FAILED Tracking failed R200
- 48. 48 Setting Voxel Resolution in Your Application • Use PXC[M]ScenePerception:SetVoxelResolution/QueryVoxelResolution • The PXC[M]VoxelResolution enumerator itemizes the supported voxel resolutions Name Description LOW_RESOLUTION The low voxel resolution. Use this resolution in a room-sized scenario (4/256m). MED_RESOLUTION The median voxel resolution. Use this resolution in a table-top-sized scenario (2/256m). HIGH_RESOLUTION The high voxel resolution. Use this resolution in a object-sized scenario (1/256m). R200
- 49. 49 3D Scanning Tracking Alerts • Use PXC3DScan::AlertEvent for 3D Scanning tracking alerts Name Description ALERT_IN_RANGE The scanning object is in the right range ALERT_TOO_CLOSE The scanning object is too close to the camera. Prompt the user to move the object away from the camera ALERT_TOO_FAR The scanning object is too far away from the camera. Prompt the user to move the object closer ALERT_TRACKING The scanning object is in good tracking ALERT_LOST_TRACKING Lost tracking on the scanning object R200
- 50. 50 Agenda • F200 Short Range Camera • F200 UX Guidelines for Laptop and All-In-One Experiences • F200 SDK tips • R200 Long Range Camera • R200 UX Guidelines for Tablet and Two-In-One Experiences • R200 SDK tips • Summary and Q&A F200 R200
- 51. 51 Summary and Next Steps • Understand what the camera sees and give the user effective feedback • Choose the right technology: Know when and when not to use gestures • Understand your users and do iterative usability testing! • Check out the UX guidelines and videos and send us feedback - software.intel.com/articles/realsense-ux-design-guidelines R200F200
- 52. 52 Day 1 Prize Completing an Online Session Evaluation by 10am Tomorrow Automatically Enters You in a Drawing to Win! Day 2 Prize Intel® Compute Stick (20) Copies of the complete sweepstakes rules are available at the Info Desk Winners will be notified by email You will receive an email with a link to the online evaluation prior to the end of this session. Day 3 Prize Microsoft* Surface* 3 (6) Dell Venue 10 7000 Series (4)
- 53. 53 Q&A
- 55. 55 Legal Notices and Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, RealSense, and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. © 2015 Intel Corporation.
- 56. 56 Risk FactorsThe above statements and any others in this document that refer to plans and expectations for the second quarter, the year and the future are forward- looking statements that involve a number of risks and uncertainties. Words such as "anticipates," "expects," "intends," "plans," "believes," "seeks," "estimates," "may," "will," "should" and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel's actual results, and variances from Intel's current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel presently considers the following to be important factors that could cause actual results to differ materially from the company's expectations. Demand for Intel's products is highly variable and could differ from expectations due to factors including changes in business and economic conditions; consumer confidence or income levels; the introduction, availability and market acceptance of Intel's products, products used together with Intel products and competitors' products; competitive and pricing pressures, including actions taken by competitors; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Intel's gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; and product manufacturing quality/yields. Variations in gross margin may also be caused by the timing of Intel product introductions and related expenses, including marketing expenses, and Intel's ability to respond quickly to technological developments and to introduce new products or incorporate new features into existing products, which may result in restructuring and asset impairment charges. Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Results may also be affected by the formal or informal imposition by countries of new or revised export and/or import and doing-business regulations, which could be changed without prior notice. Intel operates in highly competitive industries and its operations have high costs that are either fixed or difficult to reduce in the short term. The amount, timing and execution of Intel's stock repurchase program could be affected by changes in Intel's priorities for the use of cash, such as operational spending, capital spending, acquisitions, and as a result of changes to Intel's cash flows or changes in tax laws. Product defects or errata (deviations from published specifications) may adversely impact our expenses, revenues and reputation. Intel's results could be affected by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel's ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property. Intel's results may be affected by the timing of closing of acquisitions, divestitures and other significant transactions. A detailed discussion of these and other factors that could affect Intel's results is included in Intel's SEC filings, including the company's most recent reports on Form 10-Q, Form 10-K and earnings release. Rev. 4/14/15