Mixed Reality Interfaces and Product Management

Editor's Notes

• #3: We need to define something that is Valuable, Usable, Feasible and Viable. Valuable: Deliver Value, benefit or utility to the customer. Usable: The end-consumer is able to get that value easily and emotionally satisfying way. Feasible: Feasible given all our constraints around resources, funding, internal capabilities, external capabilities and so on... Feasible not just for our organization but also for all partners, who are involved in delivering that value. All three combined should be viable. Viable for the customers, our organization, and partners e.g. channel partners. This combination keep on changing. As new Technologies make feasible what was earlier not feasible, they Improve usability. As certain needs are met certain new one emerge making products more or less valuable. The viability changes are our capabilities improves or prices drop. As Product managers our challenge is to stay on top of them.
• #4: The objective on the talk is share developments, underlying technology components and some use cases in the space of AR/VR. What this is making feasible and what impact it has on usability. For each us as Product Managers, to examine whether this can add or enhance our current offerings. Can it enables us to creates a competitive advantage in current market or enable us to enter new markets/customer segment by meeting unmet customer needs? This is Product Mentor talk. Talk briefly about the Process of making that assessment. It that applies to any new technology. So what is Augmented Reality/VR
• #5: A big component of this is a New Compute Interface. That redefines… I call this Spatial User Interface In your current actual physical view, that your eyes getting, when you are watching this on your PC/Mac or mobile. The PC/mobile is a 2D screen in your total physical view. The 2nd key trait is Hands free. And it looks something like this
• #6: This is how the interface looks like. To understand it a little better….. Lets look at the Compute Interface Evolution
• #7: Lets look at the Compute Interface Evolution Note that A single device can and usually does support multiple interfaces. Augmented Reality is not just Spatial User Interface but it also includes applying voice interface, A 3D Spatial voice and GUI in space. Lets understand the components behind this and how it is coming together. I will start with simple case and build it up. Lets start with Virtual Reality.
• #8: Lets start with the simpler virtual reality.
• #9: To get to 3D we need to two images one for each eye. Stereoscopic 3D You need for two separate views one for each eye. Lets see it in action. You tube 360. glasses icon. Show pan and tilt, movements. Head Tracking –– Three sensors gyroscope, accelerometer and compass/magnetometer Notice the forward or backward motion is not tracked in this case. You need bring it close to your eyes to eliminate the actual view. And so you need lenses to zoom it. Stop Sharing.. To take to the next level by Show the lens/GearVR The Display moves with your head… you get 360 degree view To get a good experience you need high resolution and you need to refresh the frames. A minimum frame rate of around 60 frames per second is needed to users feeling sick. The current crop of VR headsets go way beyond this - Oculus is capable of 90fps, for instance, Sony's PlayStation VR manages 120fps. Computation intensive for CPU and GPU => Lot of power and heat, battery life.. Start Sharing..
• #10: Describe in own words.. Switch to camera
• #11: One way to solve this is to connect to a more powerful processor. A PC or gaming console. Add controller. More immersive experience. The variation are smoother. The Resolution is higher, the Frame refresh rate is high (Sony Play Station can do 120 fps) 9 LED lights to track head movement Challenge: Tethered to Cable: Cost. Sony Recent brought the price down to $400.
• #12: Oculus Touch is a set of wireless controllers, designed to make you feel like you're using your own hands in VR. There is also a matrix of sensors on each controller to detect gestures such as pointing and waving.  Eye tracking It's not available An infrared sensor monitor's your eyes inside the headset so FOVE knows where your eyes are looking in virtual reality. The main advantage of this is to make depth of field more realistic. And address the bandwidth challenge. Position Tracking: Oculus Development Platform : Develop not just games, but also Apps, Social VR Experiences wherein individuals can create their Virtual Avatars and interact with others in Virtual Room. They can together talk, share a whiteboard, view a video or play 3D games all in the virtual room.
• #13: Content A host of players are developing solutions and innovations in this space. Large studio like Walt Disney to mom and Pop videography that record events like weddings are looking at creating content in 3D.
• #14: Now lets look at the more complex augmented reality.
• #15: It has been around from some time. We have seen it all. Describe in own words. The idea is to augment the actual view with computer generated graphics. For those of you outside of US, not familiar with the game … the orange and blue lines This is simple case where there is one single POV where the camera is.
• #16: Pokemon Go What is really happening is Anchoring/tracking to an object? A dollar bill in this case. Take the feed of the camera and super-impose the image on an anchor. This is called the Marker based tracking. Recognizing an image the app knows before hand. Here notice that motion tracking is achieved due to the marker. There is another approach Called Markerless Tracking but we not quite there which is I did on my Samsumg Note. Google is taking this forward with Tango.
• #17: Depth Sensing computer vision to enable mobile devices to detect their position relative to the world around them without using GPS or other external signals. by illuminating an area with modulated IR light. By measuring the phase change of the reflected signal the distance can be accurately determined for every pixel in the sensor creating a 3D depth map It store this map. Lenovo Phab 2 Pro and the Asus Zenfone AR. Priced : $500 not more expensive An example of this.. Range is few meters upto 10-15 meters. And usually in indoor scenario. Time of Flight ToF Depth Sensor 5Hz (time of flight camera). 3D time of flight operates by illuminating an area with modulated IR light. By measuring the phase change of the reflected signal the distance can be accurately determined for every pixel in the sensor creating a 3D depth map of the subject or scene.
• #18: It gives motion tracking and remembers it.
• #19: It is in 3D if you move but it still on 2D Screen in your view. We need to a get to where the entire physical view is augmented.
• #20: Enter Smart glasses. Form factor Now you have two images, one for each eye to get 3D. Head tracking to see the computer generated graphics from different POVs. They have full on board compute resources. Diplay sensors, battery, network connectivity. There is no cable. They also include Sensors 6 DOF tracking. All other Sensors?? ODG R-9 enterprise $1800, R-8 $1000. Picture is R-8 ODG did not have gesture control but a track pad on the glasses, unlike the Hololens.
• #21: How is it coming together. It is coming together by Optics Unlike in the phone where the camera stream is augmented here the actucal view is augmented. MagicLeap. basic tradeoffs between resolution, field of view, eye box, image quality, hardware weight/fit, aesthetic form factor, FOV human eyes: 200x100 degree FOV Flat: Google Glass gets 13 degree FOV, and Epson BT-300 gets 23 degree FOV with 1280x720 resolution. Pros: Light, small, relatively affordable ($500–700ish).  waveguide hologram to progressively extract a collimated image guided by total internal reflection (TIR) in a waveguide pipe.  Pros: Potentially better FOV and resolution on medium sized devices. Cons: Expensive (estimated $3k+), technology still actively in development, maybe black magic. Cons: Very limited FOV and display, difficult to improve. Curved: Meta: “almost 90 degree FOV” and a 2560x1440 pixel display split between both eyes.  Pros: High FOV and resolution, relatively affordable ($900ish). Cons: Bulky, low angular resolution, material quality risk
• #22: Even without this there are at There are at least 40 different models of models across about 10 vendors? These are mainstream at least not yet… for various reasons.
• #23: 25+ AR Platforms with varying levels of functionality. For example, in our example of the stool, we need to look up the details and order it, Software Challenges: Inconsistent buttons What does pinch, swipe mean? Variations across Scaling and managing smart glasses. Most play in android. Application authoring tools, Tools Content : CAD designs, Object Recognition. So After doing all this….
• #24: We are not quite there yet but that is where we moving towards. Lets look at where it sits in the big picture…
• #26: Consumer vs. Enterprise play.
• #27: Here are some of he challenges. For taking them mainstream and for wide spread adoption. If you notice most of these challenges while critical for consumer adoption are not so crucial for Business adoption. AR is getting traction on enterprises. I believe that AR will continue to get adopted by Enterprises. AR adopted will be driven by enterprises. While VR is likely to be driven largely by Gaming and Events.
• #28: However there are a lot of smart people with lot of money are working to overcome these challenges. They are in fact competing to solve this before others. E.g. ~ 4B venture funding has gone into the space over the last 3 years. + Investments from large players 100s of patents are filled every year over the last three years. Google Tango/Day Dream, Facebook: Oculus Microsoft : Speculate has invested over 1B on Hololens with Wave guide Optics. Apple: Rumored to launch AR in 2017. Amazon: Filed application to FCC to test new devices. On the Price do not let the price hold you back. Can you get glasses that meet your specific needs at the required price. What can you get on existing phones? Can you offer it by providing the glasses yourself? One can argue that why now we can wait till it matures. Sure. The benefit of defining it now. Is to assess what incremental value it can add. How exactly you can advantage of the usability improvements. And even it turns out it is not feasible or viable. You have a view of what needs to change to make it feasible/viable. And then you can track and as soon as it meet. You can get into the market get the first move advantage. And that is one possible way to get Competitive advantage.
• #31: Execute a procedure.. There are uses cases in the entire physical object life cycle. Not just machines, but could also be organs e.g. doing a surgery.
• #32: Navigation While driving – BMW use the windshield of the car as the combiner. It makes some of the challenges in smart glasses form factor like weight, battery life .. irrelevant. It is not just normal driving …. Think of assistance it can provide during poor visibility conditions like fog. Depth sensing to assist a Fireman to guide in smoke. Example of aircraft assistance. Shop 411.
• #33: The list is by no mean exhaustive. I still keep on learning and thinking about new use cases…. Hopefully, this triggered some creative juices for you to define your own uses for your own organizations.
• #34: Valuable Not just to customers but also to your organization, your partners e.g. channel partners. Usable by end consumers Feasible: Not just technical but also business. Feasible given your constraints around resources, funding, capabilities your and if you need partners then those of your partners. They keep on changing as new Technologies make feasible what was earlier not feasible. Improve usability. As certain needs are met certain new one emerge or become more or less valuable. To share developments in the space of Mixed Reality AR/VR. What is this making feasible and what impact it has on usability. For us as Product Managers. to determine if this add/enhance our offerings; enables us to creates a competitive advantage in product market mix or enable us to enter new markets/customer segment. Enables us to meet unmet customer needs?
• #53: IMU A combination of Accelerometer, gyroscope and compass. Provide 6 degree of motion to track the head movement, position The gyroscopes measures 360 degree rotation, accelerometers measure movement along the XYZ axes and magnetometers can determine orientation towards a magnetic field. 
• #55: More Pictures ?
• #56: More Pictures ?
• #57: More Pictures ?
• #58: See if we can add more questions. Marketing use cases? If multiple use cases emerge then you have to prioritize. Document the use case. Look for the right Smart Glasses Look for the right AR platform. Is this something that is mature now or you need to wait and monitor? Is there anything you can do to be prepared. Back end systems? Can ipad