Mixing reality with mobile AR, Дмитрий Щербина
Download as PPTX, PDF1 like194 views
ARKit allows developers to create augmented reality experiences for iOS devices. It uses advanced computer vision techniques to detect and understand the environment. Some key features of ARKit include motion tracking, plane detection, lighting estimation, and support for multiplayer shared AR experiences. ARKit works with technologies like SceneKit, Unity, and Core ML to enable placing virtual objects and detecting faces/objects. ARKit 2.0 added support for vertical plane detection, object scanning and recognition. Overall, ARKit makes it easy for developers to build AR apps that integrate virtual content with the real world.
Mixing reality with mobile AR, Дмитрий Щербина
- 2. Reality vs Virtuality ARKit DMITRY SHCHERBINA iOS, OS X Senior Developer, TeamLead DataArt Dnepr
- 3. Origins of AR http://augmentedreality.com.ua/zak hvatyvayushhaya-istoriya- dopolnennoy-r/ The Master Key: An Electrical Fairy Tale, Founded Upon the Mysteries of Electricity and the Optimism of Its Devotees, 1901 novel by L. Frank Baum 1962 the Sensorama created by Morton Heilig, along with five short films for it to display.
- 4. Origins of AR The Sword of Damocles, first head mounted display, created in 1968 by computer scientist Ivan Sutherland with the help of his student Bob Sproull. First AR game - ARQuake, 2000 year.
- 5. Milgram's Reality-Virtuality Continuum Paul Milgram and Fumio Kishino defined in 1994 MR as “Any space between the ends of the virtual continuum“. The main MR (mixed reality) characteristics are next: ● It allows to combine real and virtual environments; ● It is an interactive technology and real-time technology; ● Can be registered in three dimensions.
- 9. Mixed reality scale and existing projects
- 10. Username, be ready for VR/AR
- 11. Mobile SDKs
- 12. Vuforia Vuforia is one of the most popular platforms. Pricey - $499 mobile licence, or 99$/month cloud. Platforms: Android, iOS, UWP and Unity Editor. Main functionalities: - recognition of the different types of visual objects (a box, cylinder, plane); text and environments recognition; - VuMark (a combination of picture and QR-code); - Vuforia Object Scanner, helps to create object targets; - Unity support.
- 13. Vuforia
- 14. EasyAR Is a free and easy to use alternative to Vuforia. Supported platforms: Android, iOS, UWP, Windows, Mac and Unity Editor. Version 2.0 will include the following features: - 3D Object Recognition; - Environment perception; - Cloud Recognition; - Smart Glass Solution; - App Cloud Packaging.
- 15. EasyAR
- 16. Google’s ARCore Supported platforms: Android, Smart Glasses. ARCore uses three technologies to integrate virtual content with the real world: - Motion tracking, allows to understand and track device’s position relative to the world; - Environmental understanding, allows the phone to detect the size and location of flat horizontal surfaces like coffee table; - Light estimation allows the phone to estimate the environment's current lighting conditions.
- 17. AR for iOS users with ARKit!
- 18. Real Life
- 19. AR submodules
- 20. ARKit 1.0
- 21. ARKit
- 22. ARKit v1.0
- 24. How do you get 3D from a single lens? One camera captures overlapping images of a moving object. Based on this, the 3D shape of the object is determined. Two or more cameras acquire images from different points of view. Then, the disparity map of these images is calculated. Based on this, the 3D shape of the object is determined.
- 25. Depth API (iOS 10)
- 26. ARKit v 1.0 (iOS 11)
- 28. ARKit Quick Start Basic requirements: ● OS X or Hackintosh, v10.13 ● XCode 9 ● iOS 11 or 12 beta (recommended) ● Apple device with Apple A9, A10, and A11 processors (iPhone 6S, SE or newer) https://mobile-ar.reality.news/collection/apple-arkit/
- 30. Core ML Core ML lets you integrate a broad variety of machine learning model types into your app. You can run machine learning models on the device so data doesn't need to leave the device to be analyzed.
- 31. Apple Vision framework Supported features: ● face tracking ● face detection ● landmarks ● text detection ● rectangle detection ● barcode detection ● object tracking ● image registration
- 32. ML + Vision TODO add demo video Inception v3 model. Detects the dominant objects present in an image from a set of 1000 categories such as trees, animals, food, vehicles, people, and more.
- 33. ARKit + ML + Vision https://github.com/nahttoter/CoreML-in-ARKit
- 34. Vertical surface detection Your app can see and place virtual objects on vertical surfaces, and more accurately map irregularly shaped surfaces. Real world images, such as signs, posters, and artwork can be integrated into the AR experience, so your app can fill a museum with interactive content.
- 35. ● Create an app that runs an AR session and uses plane detection to place 3D content using SceneKit. Building Your First AR Experience ● Discover supporting concepts, features, and best practices for building great AR experiences. Understanding World Tracking in ARKit ● Scanning an objects to create AR Objects https://developer.apple.com/documentation/arkit/scanning_and_detecting_3d _objects ● Tips for making your own multiplayer games using ARKit, SceneKit, and Swift. Sources available! SwiftShot: Creating a Game for Augmented Reality From where to start. ARKit 2.0
- 36. ARKit v2.0 and SwiftShot iOS 12 +
- 37. ARKit v2.0 and SwiftShot =
- 38. SwiftShot Internals ● Establishing a shared coordinate space ● Networking and state sharing ● Physics ● Asset import and management ● Flag simulation ● Dynamic audio 605th session of WWDC 2018
- 39. Establishing a shared coordinate space ● Device scans area, captures features ● Asks ARSession for world map ● Serializes to disk Share over network ● Peer-to-peer network connection ● Encrypt data in flight ● UI guidance to aid relocalization
- 40. Establishing a shared connection flow ● Capture and Send the AR World Map ● Device starting the game creates a session, starts advertising ● Other devices see session listed in menu ● User selects game to join ● Device sends request ● Advertising device accepts or denies Once session set up, devices are peers in the network,.UDP for transport were used in this specific app
- 42. ARKit v2.0 and SwiftShot
- 43. ARKit v2.0 Resources AR File Format Backed by Apple, Pixar and Adobe
- 44. ARKit Scanning and Detecting 3D Objects In iOS 12, you can create such AR experiences by enabling object detection in ARKit https://developer.apple.com/documentation/arkit/scanning_and_detecting_3d_objects
- 45. Scanning tips ● Detailed, textured objects work better for detection than plain or reflective objects. ● Object scanning and detection is optimized for objects small to fit on table. ● An object to be detected must have the same shape as the scanned reference object. Rigid objects better than soft bodies or items that bend, twist, fold, or otherwise change shape. ● Lighting conditions same to original scanning. ● High-quality object scanning requires peak device performance. But may be evaluated at any supported
- 46. Awesome ARKit ● Projects ● AppStore ● Tutorials ● Resources
- 47. So what does it all mean?
- 48. 7 key drivers to deliver AR/VR to mass market ● Mobility: AR can be used anywhere because it doesn’t block out the real world. ● Vision: AR and VR are all about vision. ● Immersion: Truly immersive experiences are magical, lifting the spirit and compelling users to want more. ● Usability: Processing power, user comfort, reliable input. ● Flexibility: Easy switch for user between VR and AR. ● Wearability: Wearing VR helmet outside is weird. ● Affordability: People don’t like to buy extra devices.
- 49. Why ARKit is good Pros: ● Free to use ● Low entrance level, native iOS support ● Lots of features available from box (Visual Inertial Odometry, Scene understanding, Surface detection, Light estimation) ● Awesome performance, compatibility with Metal, Unity, Unreal, Scene Kit ● Multiplayer, shared AR experience Cons: ● Errors gathered over the time ● Battery consumption ● Small screen size ● Not so precise
- 52. HoloLens 2
- 53. Questions are welcomed FB: Dmitry Shcherbyna
- 54. What to do next? Useful Links ● SwiftShot https://developer.apple.com/documentation/arkit/swiftshot_creating_a_game_for_augmented_reality ● ARKit Sources/Tuts/Apps https://github.com/olucurious/Awesome-ARKit ● VR History https://www.youtube.com/watch?v=43mA_ypfwKg, http://augmentedreality.com.ua/zakhvatyvayushhaya-istoriya- dopolnennoy-r/ ● Introducing ARKit by Apple https://developer.apple.com/arkit/ ● Best AR SDK https://thinkmobiles.com/blog/best-ar-sdk-review/ ● Comparing current alternatives https://www.imore.com/apple-arkit-vs-google-tango-vs-microsoft-hololens-augment-my-reality ● Made with AR http://www.madewitharkit.com/ ● ARKit WWDC https://developer.apple.com/videos/play/wwdc2017/602/ ● Google VR/AR https://vr.google.com/ https://developers.google.com/ar/discover/ ● CoreML+ Vision https://developer.apple.com/documentation/vision/classifying_images_with_vision_and_core_ml ● Depth API https://developer.apple.com/videos/play/wwdc2017/507/ ● Recent news https://augmented.reality.news/ ● https://venturebeat.com/2018/04/13/sony-is-considering-new-psvr-livestreaming-and-mixed-reality-features/ ● New things of Arkit 2.0 https://medium.com/vladimir-novick/new-cool-things-you-can-do-with-arkit-2-0-and-react-native- 29cae163849f ● ARKIT 101 https://mobile-ar.reality.news/collection/apple-arkit/