Create Engaging Healthcare Experiences with Augmented Reality
2 likes762 views
The document discusses the integration of augmented reality (AR) and virtual reality (VR) in creating engaging healthcare experiences, highlighting various technologies and tools, such as Microsoft Hololens, Apple ARKit, and Amazon Sumerian. It covers topics including the definitions of AR, VR, and mixed reality, as well as practical applications in healthcare, like storytelling and patient education. The document also includes resources and tutorials for developing AR applications to support healthcare initiatives.
Create Engaging Healthcare Experiences with Augmented Reality
- 1. St. Pölten University of Applied SciencesSt. Pölten University of Applied Sciences Platzhalter für möglichen Bildeinsatz Create engaging Healthcare Experiences with Augmented Reality Andreas Jakl – www.andreasjakl.com build.well.being 2019 Digital Healthcare FH St. Pölten Platzhalter für möglichen Bildeinsatz
- 2. Andreas Jakl ▪ Focus areas ▪ AR / VR, mobile apps, sensors, interaction technology, software architecture, open source developer (NFC, Bluetooth Beacons) ▪ Microsoft MVP (Most Valuable Professional) & Amazon AWS Cloud Ambassador ▪ mobility.builders community: Mobile Developer After-Work Events ▪ Previous Experience ▪ Tieto, Start-up (Mopius), Nokia (Finland), Siemens Mobile (Munich), FH Hagenberg (Mobile Computing) Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten https://www.andreasjakl.com/ andreas.jakl@fhstp.ac.at @andijakl
- 3. WHAT IS AR / VR / MR? Overview & Intro Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 4. Virtual Reality? ▪ Dive into another reality ▪ Mainly visual + acoustic stimulation ▪ Immersion & presence ▪ Feeling to be at another place Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Screenshot: Ready Player One, 2018
- 5. Augmented und Mixed Reality ▪ Virtual Reality ▪ Only virtual, completely hides reality ▪ Augmented Reality ▪ Visible world gets digital overlays ▪ Mixed Reality ▪ Virtual content seamlessly blends into the real world Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Wired.com
- 6. Microsoft HoloLens 2 Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten https://www.hololens.com/
- 7. Apple ARKit Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten https://developer.apple.com/augmented-reality/
- 8. Google DayDream / ARCore Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Google DayDream https://vr.google.com/daydream/ Google ARCore https://developers.google.com/ar/
- 9. Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Source: CCS Insight VR and AR forecast (November 2018) https://www.ccsinsight.com/press/company-news/3726-virtual-reality-device-market-declines-in-2018-but-outlook-remains-positive Shipments of VR and AR devices, worldwide, 2015-2022
- 10. AR / VR Estimates: 2025 Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Source: Goldman Sachs Global Investment Research January 13, 2016
- 11. Hype & Reality Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Source: Gartner https://www.gartner.com/smarterwithgartner/5-trends-emerge-in-gartner-hype-cycle-for-emerging-technologies-2018/
- 12. Digital Healthcare Explained Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten App created by Digital Healthcare @ FH St. Pölten wins global Amazon AR/VR challenge in the category: Best AR App! https://devpost.com/software/digital-healthcare-explained-czflve https://www.fhstp.ac.at/en/newsroom/news/best-augmented-reality-app-in-amazon-hackathon
- 13. Immersive Media Lab ▪ Storytelling in AR / VR ▪ Use case: “Enlightening Patients with Augmented Reality” ▪ Explain corrective eye surgery ▪ Pre-operation, during operation, afterwards ▪ Study acceptance of different AR interaction methods Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 14. AMAZON SUMERIAN Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 15. Amazon Sumerian Overview ▪ Interactive 3D scenes ▪ Browser-based 3D editor + viewer (WebGL / WebXR) ▪ Supports VR & AR ▪ Special feature: ▪ Animated avatars (“hosts”) ▪ Speech + gestures ▪ Supports conversations ▪ https://aws.amazon.com/sumerian/ Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 16. AWS Account ▪ AWS = Amazon Web Services ▪ Sumerian ▪ Free tier for 12 months https://aws.amazon.com/free/ ▪ 5 GB scene traffic free ▪ Credit card required for sign-up, but not charged! ▪ Pricing after 1 year / exceeding free tier limits ▪ 3D object storage, scene traffic, speech services Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 17. Scene Setup ▪ Log in to Amazon Sumerian Console ▪ https://console.aws.amazon.com/sumerian/home 1. Choose world-wide region for your service 2. Create “Default Lighting” scene Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 18. 3D Editor Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Canvas / 3D view Click: select 3D object Shift + left mouse: move view Right mouse: rotate view Entities: everything in your scene Assets: currently available content Inspector: properties of selected part Top menu: new 3D models, publish scene
- 19. First 3D Object Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten 1) Import Assets 2) Select Table Cafe 3) Add
- 20. Add to Scene ▪ Drag & Drop Model from Assets to Entities ▪ Then wait a few seconds… ▪ Ensure object is visible. Some tips: ▪ Zoom in with mouse wheel ▪ Select object and press “f”-key to center Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 21. Publish! ▪ Link to your scene ▪ Runs in every Chrome + Firefox! Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 22. Custom 3D Models ▪ Free sources ▪ Remix 3D (Microsoft): https://www.remix3d.com ▪ Poly (Google): https://poly.google.com/ ▪ Sketchfab: https://sketchfab.com/features/download ▪ Own 3D Models ▪ Paint 3D & 3D Builder (Windows 10) ▪ Blender ▪ Autodesk Maya / 3DS Max Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 23. Import to Sumerian ▪ Human Heart ▪ From Remix3D (Microsoft)* ▪ Download adapted version: https://www.andreasjakl.com/ human-heart/ ▪ Import to Sumerian ▪ Drag from “Assets” to “Entities” ▪ Adapt scale & rotation ▪ Scale: 0.1, Rotate X: -90 Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten * https://www.remix3d.com/details/059f2766c027458787256ebb47a4094e Microsoft, Creative Commons CC-BY-SA
- 24. Adding Speech ▪ Conversational interfaces ▪ Amazon Lex: natural language understanding ▪ Amazon Polly: text-to-speech ▪ Different Amazon Services! ▪ Secure communication? ▪ Amazon Cognito credentials ▪ One-click setup: Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Amazon Alexa Built on these two services, adds “skills” concept, skill store, etc. https://docs.sumerian.amazonaws.com/tutorials/create/beginner/aws-setup/
- 25. Stack → CognitoIdentyPoolID Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten 3 Copy CognitoIdentyPoolID
- 26. Connect with Sumerian Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Paste
- 27. Create Host Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 28. Position Host ▪ Drag host from Assets to Entities ▪ Then wait a few seconds! ▪ Re-position host ▪ Press Play ▪ Host already moving naturally Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 29. Let the Host Explain the Heart ▪ Edit the Speech settings in the inspector of your host ▪ (1) Choose voice: “Salli” is US English ▪ (2) Click on “+” in “Speech Files” to create a new file ▪ (3) Enter text ▪ (4) Save (Default name: “Speech”) Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten 1 2 3 4
- 30. Create Logic: When to Speak? ▪ (1) Choose “+ Add Component” ▪ “State Machine” ▪ (2) Add a new behavior (“+”) ▪ (3) Opens editor in bottom part Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten 1 2 3
- 31. Add Actions ▪ Actions define what to do in a situation (“state”) or what to wait for ▪ “Add Action” to default “State 1” ▪ “Click/Tap on entity” Hint: use search in top right corner Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 32. Second State: Start Speaking! ▪ Add a second state ▪ What should happen after tapping the host? ▪ (1) + (2) “Add State” ▪ (3) Add action to “State 2”: “Start Speech” ▪ (4) Choose your “Speech” from drop-down Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten 1 2 3 4
- 33. Connect Everything ▪ Logic ▪ Clicking should trigger speech ▪ Speech end should go back to initial state ▪ Drag & drop from action box to new state ▪ Test! Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 34. More Life-Like Host: Gestures ▪ Hand-Movements of the host while speaking ▪ Select host in “Entities” ▪ (1) Inspector: “Speech” → “Gesture Map” → “+” ▪ Triggers gestures based on words in spoken text ▪ (2) Auto-assign gestures to “Speech” file Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten 1 2
- 35. Publish & Test! Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 36. Augmented Reality & More! ▪ Extended Tutorial ▪ Animation & Timelines ▪ Communicating within the scene ▪ Publish to Android & show as Augmented Reality Scene ▪ https://www.andreasjakl.com/amazon-sumerian- augmented-reality/ Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
- 37. THANK YOU! Andreas Jakl Lecturer / Digital Healthcare @ FHSTP https://www.andreasjakl.com/ Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten