Programming Augmented Reality - Xamarin Evolve
1 like490 views
The document outlines the fundamentals of programming augmented reality (AR) for mobile applications, discussing the integration of real-world data with computer-generated input. It details the necessary components for AR, such as accessing video camera feeds, location, and orientation sensors, while also providing technical guidance for implementing AR features. Additionally, it touches on various applications of AR technology in gaming, social interaction, and industrial uses, as well as listing numerous available SDKs for developers.
![Receiving Location Updates
lman = new CLLocationManager {
DesiredAccuracy = CLLocation.AccuracyBest,
};
lman.LocationsUpdated += (sender, e) => {
var loc = e.Locations [0];
Timestamp = loc.Timestamp;
Location = new Location (loc.Coordinate.Latitude, loc.Coordinate.Longitude, loc.Altitude);
HorizontalAccuracy = loc.HorizontalAccuracy;
VerticalAccuracy = loc.VerticalAccuracy;
};
lman.StartUpdatingLocation ();](https://image.slidesharecdn.com/frankkruegeraugmentedrealitypresentationfinal-180730153614/85/Programming-Augmented-Reality-Xamarin-Evolve-15-320.jpg)
Programming Augmented Reality - Xamarin Evolve
- 1. Programming Augmented Reality: Make your own sci-fi Mobile App Dev @praeclarum Frank A. Krueger
- 2. Augmented Reality a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data
- 5. Why Augmented Reality? • Presents data in a way that we are accustomed to • Can present a lot of data in multiple ways Useful • Remember the Terminator, he was fun right? • Extends our senses • Physical games are more fun Fun • Takes advantage of distinguishing platform elements Mobile Only
- 6. Anatomy World Simulation Video Camera Location & Orientation Augmented View
- 7. Anatomy World Simulation Video Camera Location & Orientation Augmented View This is where the magic happens AR library
- 8. Video Camera
- 9. Video Camera ■ Many resolutions (width and height) and framerates ■ Image filtering controls ■ Field of View ■ High-performance code - be careful! ■ 8 x 106 pixels * 30 Hz = 240,000,000 pixels / second It all starts here
- 10. Directly Accessing the Camera (iOS) session = new AVCaptureSession (); session.SessionPreset = AVCaptureSession.PresetMedium; device = AVCaptureDevice.DefaultDeviceWithMediaType (AVMediaType.Video); device.LockForConfiguration(out error); device.ActiveVideoMinFrameDuration = new CMTime (1, 30); device.UnlockForConfiguration(); input = AVCaptureDeviceInput.FromDevice (device, out error); session.AddInput (input);
- 11. Getting Images (iOS) output = new AVCaptureVideoDataOutput (); output.VideoSettings = new AVVideoSettings (CVPixelFormatType.CV32BGRA); queue = new DispatchQueue ("VideoCameraQueue"); output.SetSampleBufferDelegateAndQueue (new VideoCameraDelegate { Camera = this } , queue); session.AddOutput (output); class VideoCameraDelegate : AVCaptureVideoDataOutputSampleBufferDelegate { public override void DidOutputSampleBuffer (AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection) { try { var frame = ImageFromSampleBuffer (sampleBuffer); Camera.OnFrameCaptured (frame); sampleBuffer.Dispose (); } catch (Exception ex) { Debug.WriteLine (ex); } } }
- 12. Field of View public float FieldOfView { get { return device.ActiveFormat.VideoFieldOfView; } } Field of View (FOV) is the Visible Angle
- 14. Location Sensor ■ Our friends Latitude & Longitude ■ Altitude above sea level (sometimes) ■ 10m resolution, on a good day ■ “Better” with Wi-fi ■ “Not great” indoors ■ (There are alternatives such as triangulation from known points) Global Spherical Coordinates
- 15. Receiving Location Updates lman = new CLLocationManager { DesiredAccuracy = CLLocation.AccuracyBest, }; lman.LocationsUpdated += (sender, e) => { var loc = e.Locations [0]; Timestamp = loc.Timestamp; Location = new Location (loc.Coordinate.Latitude, loc.Coordinate.Longitude, loc.Altitude); HorizontalAccuracy = loc.HorizontalAccuracy; VerticalAccuracy = loc.VerticalAccuracy; }; lman.StartUpdatingLocation ();
- 16. Orientation Sensor ■ Pitch, Roll, Yaw (3 numbers) ■ Quaternion (4 numbers) ■ 4x4 Transformation Matrix (16 numbers) A Variety of Formats Pitch Yaw Roll ■ Need a “reference orientation” or known orientation from which to measure It’s relative…
- 17. Receiving Orientation Updates mman = new CMMotionManager { ShowsDeviceMovementDisplay = true, }; mman.StartDeviceMotionUpdates ( // CMAttitudeReferenceFrame.XArbitraryZVertical, // CMAttitudeReferenceFrame.XArbitraryCorrectedZVertical, // CMAttitudeReferenceFrame.XMagneticNorthZVertical, CMAttitudeReferenceFrame.XTrueNorthZVertical, new NSOperationQueue (), (motion, error) => { Orientation = ToMatrix4d (motion.Attitude.RotationMatrix); } );
- 18. Augmented View
- 19. How do you augment th Video Camera images?
- 20. Augmentation Techniques Overlay 2D Annotations • Heads Up Display • Terminator • Use OpenGL, SceneKit, MonoGame, … • Mixing virtual reality with augmented Show 3D Virtual Objects Distort Camera Image • Enhance recognized and correlated objects • Hot / Cold Maps • Pixel perfect • No lag
- 21. Aligning the Augmented View to the Video Camera A Pinhole Camera Model • Simple • Used by 3D renderers • Only depends on Field of View • “Close enough” to the video camera In a 3D Cartesian World • Use 3D X, Y, Z coordinates instead of Latitude, Longitude, Altitude • Orient the camera by applying the orientation transform Projection Matrix ModelView Matrix
- 22. Pinhole Camera Model Projection Matrix ModelView Matrix 3D Worl d Point 2D View Point x x = (after a perspective divide) Video Camera Location & Orientation
- 23. Projection Matrix public static Matrix4d GetProjectionMatrix (VideoCamera videoCamera, UIImage frame) { return Matrix4d.CreatePerspectiveFieldOfView ( fovy: videoCamera.FieldOfView, aspect: frame.Size.Width / frame.Size.Height, zNear: 0.01, zFar: 4700); }
- 24. ModelView Matrix - Two Techniques • Very easy • Not accurate • High latency • Best for outdoor unpredictable environments Location & Orientation Sensors • Very difficult • Very accurate • No latency • Best for indoor or controlled environments Image Recognition
- 25. ModelView Matrix from Location & Orientation public static Matrix4d GetModelView (Location location, Matrix4d orientation) { // 1. Calculate position in 3D cartesian world // 2. Find “up" // 3. Orient to face the north pole // 4. Apply the device orientation }
- 26. ModelView Matrix: Location to 3D Cartesian • Center of earth is at (0, 0, 0) • Does not rotate, the universe around it does • Physically accurate • X, Y, Z are generally unintelligible Earth-centered, Earth-fixed • Flatten the earth like a map • X axis is East/West • Y axis is North/South • Z axis is Altitude North, East, Up
- 27. ModelView Matrix: Location to 3D Cartesian (ECEF) // // 1. Calculate position in 3D cartesian world // var pos = location.Position; public Vector3d Position { get { var omega = ToRad * Longitude; var phi = ToRad * Latitude; var r = RadiusOfEarth + Altitude; return new Vector3d ( r * Math.Cos(phi) * Math.Cos(omega), r * Math.Cos(phi) * Math.Sin(omega), r * Math.Sin(phi)); } } • (X, Y, Z)
- 28. ModelView Matrix: Up // // 2. Find "up" // var up = location.Position; up.Normalize (); • (0, 0, 0) • (X, Y, Z) Up
- 29. ModelView Matrix: Look at the North Pole // // 3. Orient to face the north pole // var northPos = Location.NorthPole.Position; var northZAxis = (pos - northPos); northZAxis.Normalize (); var northYAxis = up; var northXAxis = Vector3d.Cross (northYAxis, northZAxis); northXAxis.Normalize (); northZAxis = Vector3d.Cross (northXAxis, northYAxis); northZAxis.Normalize (); var lookNorthI = new Matrix4d ( new Vector4d(northXAxis), new Vector4d(northYAxis), new Vector4d(northZAxis), Vector4d.UnitW); Location NorthPole = new Location (90, 0, 0); True North • (X, Y, Z)
- 30. ModelView Matrix: Orient // // 4. Apply the device orientation // var newOrient = new Matrix4d ( -orientation.Column1, orientation.Column2, -orientation.Column0, Vector4d.UnitW); var newOrientI = newOrient; newOrientI.Transpose (); var modelViewI = (newOrientI * lookNorthI); modelViewI.Row3 = new Vector4d (pos.X, pos.Y, pos.Z, 1); var modelView = modelViewI; modelView.Invert (); return modelView;
- 31. Location to 2D View Projection Matrix ModelView Matrix 3D Worl d Point 2D View Point x x = (after a perspective divide)
- 32. Location to 2D View public PointF LocationToView (Location location) { // Move location to 3D earth var pos3d = new Vector4d (location.Position, 1); // Camera model var m = Matrix4d.Mult (modelViewMatrix, projectionMatrix); // Project into homogeneous 2D point var pos2h = Vector4d.Transform (pos3d, m); // Perform the perspective divide var pos2 = pos2h / pos2h.W; // Stretch into our view return new PointF ( (float)((pos2.X + 1) * 0.5) * viewSize.Width, (float)((-pos2.Y + 1) * 0.5) * viewSize.Height ); }
- 33. Demo
- 34. Same Math Can be Used to Make 3D void SetOpenGLCamera () { GL.MatrixMode (All.Projection); GL.LoadMatrix (ref projectionMatrix.Row0.X); GL.MatrixMode (All.Modelview); GL.LoadMatrix (ref modelViewMatrix.Row0.X); }
- 35. Demo
- 36. World Simulation
- 37. It’s up to you The world’s full of geo tagged data
- 38. Applications Games • Massively Multiplayer Online Live Action Role Playing Games (MMOLARPGs) • Treasure Hunt • Intercontinental Worms • Friend locator • Friend holograms • Tourist attractions Social • Physically secure digital goods • Architectural projections • Airplane & satellite tracking • Boat tracking Industrial
- 39. DroidAR, ARmedia, Vuforia, ARLab, Beyond Reality Face, D'Fusion, instantreality, Metaio SDK, Viewdle, Xloudia, ARPA, ALVAR, AndAR, AR23D, ARToolkit, Aurasma, Awila, BeyondAR, Catchoom, Cortexia, Google Goggles, IN2AR, Koozyt, layar, LibreGeoSocial, mixare, NyARToolkit, Obvious Engine, PanicAR, PointCloud, popcode, PRAugmentedReality, PTAM, Qoncept AR, Robocortex, SLARToolkit, snaptell, SSTT, String, Studierstube Tracker, UART, Wikitude, xpose visual search, yvision, Zenitum Feature Tracker 45+ Mobile SDKs to Choose From http://socialcompare.com/en/comparison/augmented-reality-sdks
- 40. Thank you Frank A. Krueger @praeclarum https://github.com/praeclarum/ARDemo