Workshop sulle spatial anchors

Editor's Notes

• #4: Why mobile AR when we have next gen AR devices like HL2 Gartner quote: By 2020, 100 million consumers will shop in AR online and in-store and 5G will play a huge role says Gartner. ​https://www.gartner.com/en/newsroom/press-releases/2019-04-01-gartner-says-100-million-consumers-will-shop-in-augme ​ Persistent content enabled by Anchors will play an important role here as well for advanced in-store experiences 26% active devices were able to use either ARKit + ARCore. September 2018  Now one third Google I/O - 400mio devices We all love the advanced features of HL but mobile AR is Low entry, wide reach  everyone has a smartphone
• #5: Why Azure? Devices have their own, disconnected AR view of the world Cloud brings it together Virtual content persists and sharing across devices Relies on device’s digital view of the world How do devices view/see the world?
• #6: Abstract concept  Examples and scenarios we see with our customers HL since 2015  Dozens of apps / use cases Due to NDA generalized use cases we see commonly
• #7: Own coord system for location in real-world A common frame of reference to place digital content in the same physical location where it stays at the same position and orientation relative to the real-world environment.
• #8: Location in real-world ensures virtual content stays at the same pos/orient even if you move Anchor is not the content, just a real-world location where content can be anchored HoloLens, ARKit, ARCore and Magic Leap (PCF) A common frame of reference to place digital content in the same physical location where it stays at the same position and orientation relative to the real-world environment.
• #9: What is a Cloud Spatial Anchor - Why Cloud? Normalizes different devices anchors/mappings Multiple users. sharing across devices with persistence A common frame of reference for enabling multiple users to place digital content in the same physical location where it is persisted and can be seen on different devices at the same position and orientation relative to the real-world environment.
• #10: Persistence Hot research topic, Startups as well as big players ASA is the only implementation that provides persisted, x-plat, shared anchors  Paris later tech details Here more examples: shared, x-plat + persistence A common frame of reference for enabling multiple users to place digital content in the same physical location where it is persisted and can be seen on different devices at the same position and orientation relative to the real-world environment.
• #11: Real-time IoT sensor data on real-world equipment Multi-user, cross-platform Biz val: Real-time insights into data much faster than searching through pages of information on complex control room panels
• #12: Dynamic warehouse navigating for fulfilling a order Biz val: Increased efficiency and less packaging errors Photo from Wikipedia: https://commons.wikimedia.org/wiki/File:Pantos_Logistics_-_Warehouse_picture.jpg
• #13: Enterprise is nice but there’s a ton of fun consumer use cases as well Mobile AR market size
• #14: Festival navigation like Coachella https://twitter.com/rschu/status/1117339333969817600 Theme park navigation concept video used with permission: ​ https://vimeo.com/319985238 ​ “Sure, You can use it. At the moment it's a marker based/Arkit/Core combination, but love to see what  Azure Spatial Anchors can do to further this concept” Also: holographic characters in its amusement park for guests in line to hang out with them as they’re waiting to board the ride Biz val: Engaging customers with innovative tech and improve their CX
• #16: Swiss 'gabarits’ / Baugespann / Profile Show the form of a future construction with poles, Architecture vis of building before they build to get citizens involved Design reviews Biz val: making tedious, laborious and time-intensive work into a natural, intuitive and more collaborative experience. Better vis means better commitment; reduce cost First Photo from: https://upload.wikimedia.org/wikipedia/commons/4/43/Sch%C3%B6nb%C3%BChlring_in_Luzern.jpg Second photo by my friend Laurent Bugnion who lives in Siwtzerland
• #17: Location-based: ASA screams games  Multi-user, cross-platform Scavenger hunt; Hide and seek games, social engaging Biz val: Democratize AR game dev – now any developer can build the next Pokémon Go, ​ Fruit hunt  dropped fruity anchors during last MVP Summit Won’t slip on virtual banana
• #18: Talk about Azure Spatial Anchors, abbreviate to ASA New Azure service lets you place spatial anchors and save them to the cloud
• #19: Before we begin, want to briefly touch on what a cloud spatial anchor is again – important for the rest of this talk If you place a spatial anchor, you now have a reference between your device and a point the real world. You can render a hologram there, And when you want to share this point, you save it to ASA. When the next person comes, they get that location, and they can see the hologram as well. Share a position in the real world that can be persisted and can be seen on different devices.
• #20: Want to share a picture that I’m familiar with – the lobby of my building.
• #21: And this video is what Azure Spatial Anchors sees when it's looking at the lobby.   In green are feature points, which are distinguishing points in the real world. Your badge would be full of feature points because there are lots of high visual contrast.   In red are camera frames, which are the pictures we took with the camera from different points in space.   As you wave your phone around, we get more and more green feature points. When you place an anchor, it gets tied to those feature points. And when you look for an anchor, we use the feature points that we've collected to know where the anchor should be placed. I want to note that all of this is done on the device, so no camera frames are leaving your device.
• #22: Building an app for collaboration in the real world Building on that to do wayfinding Finally, some UX best considerations
• #23: Support HL, iOS, Android. Can develop on Unity or write native apps. All platforms are first class citizens. And our KPI matrix that considers all platforms, and their relationships to each other. Today, we’re highlighting iOS Swift and Android Java. But you can also build for iOS and Android.
• #24: So the app we built is a collaborative experience. Like we saw in Rene's factory floor example, where they placed training guidance on a car being assembled, this app lets you leave notes on places of interest and then later retrieve them.   Before this demo, we placed a note on the table. When we start the app, we're going to locate it. We'll then place a new note that we can later retrieve. Right now, they're collecting the camera frames and searching for anchors. We found the note! And now we'll place a note for the next person to come see. If we restart the device, we can now see that we found the new anchor again!
• #26: I'm now going to give you a quick look at the architecture. We'll then look at some key concepts for the ASA API before coming back to it in more detail.   We have client code, Java and Swift in this case. It talks to two things.   ASA, which we use to save and locate anchors.   Next, an API. Using Azure App Service, we have an ASP.NET API that is connected to a CosmosDB backend. We use this to store information about our anchors which we need to retrieve them from Azure Spatial Anchors. Because it’s not really important what we use in the sharing service, I’m just going to call it the sharing service, and you can think of it as a database.   Now I'm going to look at some API key concepts, and then we'll revisit this diagram with a little more context.
• #27: CloudSpatialAnchorSession – you give the session camera frames (one note – you don’t need to do this on HoloLens because we have the map) CreateAnchorAsync – no ID when you create, only when it’s been created you’ll get a callback with a CloudSpatialAnchor that has an ID assigned. CreateWatcher You get back a CloudSpatialAnchor, and it has a link to an underlying AR platform anchor. If you are on iOS, you will get back an ARAnchor, even if you are on a HoloLens. That’s one of the really cool things we do – you don’t have to do any math or conversions and you get to work with the anchor that is native to the platform you are on. So if we look at the big picture, we see that when we create an anchor we are given back an ID. And when we want to find an anchor, we need to tell ASA the ID of the anchor we want to find. That is why we need a database – we store IDs in it.
• #28: I'm now going to give you a quick look at the architecture. We'll then look at some key concepts for the ASA API before coming back to it in more detail.   We have client code, Java and Swift in this case. It talks to two things.   ASA, which we use to save and locate anchors.   Next, an API. Using Azure App Service, we have an ASP.NET API that is connected to a CosmosDB backend. We use this to store information about our anchors which we need to retrieve them from Azure Spatial Anchors. Because it’s not really important what we use in the sharing service, I’m just going to call it the sharing service, and you can think of it as a database.   Now I'm going to look at some API key concepts, and then we'll revisit this diagram with a little more context.
• #29: The sharing service is a way for us to store IDs. You need to share IDs because different users at different times will want to look for an anchor. I now
• #30: I’m elippsis in place of code to simplify things. In this case, sceneView and scene are Sceneform concepts. All this code it on GitHub, and I’d love it if you looked into it more there. Then, we need to give the camera frames to ASA, so we add an update listener and call ProcessFrame. If you’re not familiar with this sytax, a listener is similar-ish to a delegate or callback, in that it will get called by SceneForm whenever there is a new camera frame So what this does it call session.processframe everytime there’s a new camera frame On HoloLens you don't need to do this because we have the spatial map. SessionUpdated So we use sessionUpdated to tell us about how much environmental information we have, and whether it’s enough to save an anchor Next, let’s create an anchor. To do so, we need to add a callback that will be invoked when you tap on the screen. TapArPlaneListener is a ARCore concept that lets us register a method to be called when you tap on a plane. Let’s write the handle tap method to place an anchor
• #31: First, we create the local anchor – creating this Anchor object is an ARCore concept, and we do it by using the HitResult, i.e. the place in the real world we tapped.
• #33: We’ll now add another listener that will get called when we locate an anchor. Now that we’ve located an anchor, we can get its location. We get in back as an ARCore Anchor, even if it was created on HoloLens or iOS. So that’s our app – I hope you can now initialize a session, create an anchor, and then locate anchors. All this code is on GitHub, and I’ll share the Url at the end of the session.
• #35: You’ve seen how to locate an anchor by ID. You initialize the criteria, set an array of IDs, and create a watcher. But you can also locate anchors that are connected to, and nearby, an anchor you have already located. We’ll write LocateNearbyAnchors to do so. It’s passed in a CloudSpatialAnchor that has already been located. Then create NearAnchorCriteria, and set the anchor as the source anchor. Then we’ll create the AnchorLocateCriteria we’ve seen before, but this time set the NearAnchor as the nearAnchorCriteria object. Let’s look at what this will do.
• #38: I want to first touch briefly on feature points again.
• #39: Well, first, you want to try to get feature points. A white wall will have less of those green dots, so it will be harder to locate an anchor that was placed without capturing many feature points.   Second, you want to consider the scenario, because it will lead your UX design.   Are you placing a note at the top of a hiking trail to tell people about the route? If so, you'll want to make sure you capture from 360 so that the next person coming along doesn't need to be in the same place you were   Or are you displaying a data visualization on a conference room table? Maybe you should encourage users to place the anchor sitting down, or in an accessible and central spot.   Are you leaving notes around a museum? Maybe it doesn’t matter if the user captures a 360, and instead they should focus on the painting and panning around it.
• #40: When you’re finding anchors, I also encourage you to design based on the scenario. Internally, we’ve seen two patterns for finding anchors emerge. One is the target scenario – where you’re able to show the user a picture of the area they should aim at. Let’s give an example - If you were designing an app for a museum, you might have an admin mode, where only certain people get to place anchors. In that case, you might be able to get them to take a picture of each painting to help guide users to find the anchor later. Another thing we’ve seen is the room scenario, where the anchor could be anywhere. An example of this is ap escape the room app we built, where anchors could be all over. In that case, you want to give UX that will encourage the user to pan all around.
• #41: So, with that, I want to say that we’re live! We launched at the end of February, and our team is so happy and thankful to be here. We’ve loved watching what people are building – and we hope to continue seeing Rene’s amazing videos, as well as all of yours!