Augmented Reality Application Development_Vuforia.pdf
- 1. Unit II Unit Title: Overview of AR Development AR Plugins Overview - Types of Plugins - Plugins Setup - Vuforia and AR Foundation Setup. AR Plugins Unity provides powerful tools to make rich, deeply engaging augmented reality (AR) experiences that intelligently interact with the real world Unity’s custom resources bring vision to life. Our industry-leading real-time 3D development platform pair tools purpose-built for augmented reality creators and a unified workflow across devices, let to focus on creating AR experience. Augmented Reality (AR) involves a new set of design challenges compared to VR or traditional real-time 3D applications. By definition, an augmented reality app overlays its content on the real world around the user. To place an object in the real world, you must first determine where to place for it. For example, you may want to place a virtual painting on a physical wall. If you place a virtual potted plant, you might want it on a physical table or the floor. An AR app receives information about the world from the user’s device, such as the locations of planar surfaces, the detection of objects, people, faces, and so on; and must decide how to use this information to create a good experience for the user. Unity provides powerful AR tools to create rich, deeply engaging augmented reality experiences that intelligently interact with the real world. Unity’s AR feature set provides a starting point for augmented reality development, spanning from head-mounted displays to mobile. Unity provides plugins to develop AR applications. Various types of plugins are 1. XR integration tool kit 2. AR Foundation 3. AR Core 4. Easy AR 5. AR Kit 6. Vuforia XR Interaction Toolkit The XR Interaction Toolkit package is an interaction system for creating immersive experiences. No need to code object interactions from scratch – add interactivity to AR apps by dropping components into your scene. XR Interaction Toolkit supports interaction tasks such as cross-
- 2. platform XR controller input, haptic feedback through XR controllers, visual feedback, and more. The XR Interaction Toolkit package is a high-level, component-based, interaction system for creating VR and AR experiences. It provides a framework that makes 3D and UI interactions available from Unity input events. The core of this system is a set of base Interactor and Interactable components, and an Interaction Manager that ties these two types of components together. It also contains components that you can use for locomotion and drawing visuals. XR Interaction Toolkit contains a set of components that support the following Interaction tasks: Cross-platform XR controller input: Meta Quest (Oculus), OpenXR, Windows Mixed Reality, and more. Basic object hover, select and grab Haptic feedback through XR controllers Visual feedback (tint/line rendering) to indicate possible and active interactions Basic canvas UI interaction with XR controllers Utility for interacting with XR Origin, a VR camera rig for handling stationary and room- scale VR experiences To use the AR interaction components in the package, you must have the AR Foundation package in your Project. The AR functionality provided by the XR Interaction Toolkit includes: AR gesture system to map screen touches to gesture events AR interactable can place virtual objects in the real world AR gesture interactor and interactables to translate gestures such as place, select, translate, rotate, and scale into object manipulation AR annotations to inform users about AR objects placed in the real world Finally, its possible to simulate all of the interactions with the XR Device Simulator in case you don't have the hardware for the project you are working on, or just want to test interactions without entering the headset. Requirements This version of the XR Interaction Toolkit is compatible with the following versions of the Unity Editor:
- 3. 2021.3 and later Dependencies The XR Interaction Toolkit package has several dependencies which are automatically added to project when installing: Input System (com.unity.inputsystem) - The Input System allows your users to control your game or app using a device, touch, or gestures - Unity supports input through two separate systems, one older, and one newer. - The older system, which is built-in to the editor, is called the Input Manager. The Input Manager is part of the core Unity platform and is the default, if you do not install this Input System Package. - This Input System package is a newer, more flexible system, which allows you to use any kind of Input Device to control your Unity content. It's intended to be a replacement for Unity's classic Input Manager. It is referred to as "The Input System Package", or just "The Input System". To use it, you must install it into your project using the Package Manager. - During the installation process for the Input System package, the installer offers to automatically deactivate the older built-in system. Unity UI (com.unity.ugui) - Unity UI is a UI toolkit for developing user interfaces for games and applications. It is a GameObject-based UI system that uses Components and the Game View to arrange, position, and style user interfaces.You cannot use Unity UI to create or change user interfaces in the Unity Editor. - This allow to create a Canvas, positioning and animating elements, defining user interactions, and sizing layouts automatically. XR Core Utilities (com.unity.xr.core-utils) - The XR Core Utilities package contains a variety of classes and extension methods which are commonly used and shared between multiple XR packages. XR Legacy Input Helpers (com.unity.xr.legacyinputhelpers) - The com.unity.xr.legacyinputhelpers package contains a number of useful helpers for building XR Projects. These include the Tracked Pose Driver and the Input Asset XR Bindings Seed Utility.
- 4. Built-in modules o Audio o IMGUI o Physics Optional dependencies To enable additional AR interaction components included in the package, AR Foundation (com.unity.xr.arfoundation) must be added to your project using Package Manager. To enable additional properties in some behaviors, the Animation module must be added to your project using Package Manager. AR Foundation A framework purpose-built for AR development allowing you to develop your app once, then deploy it across multiple mobile and wearable devices. Build and ship robust apps through AR Foundation’s core features from industry-leading XR platforms and frameworks: 1. Meta Quest 2. Apple Vision Pro 3. ARCore 4. ARKit 5. HoloLens Download the plugin from : com.unity.xr.arfoundation Plugins Setup: Follow these steps to install the AR Foundation Unity package. a. Open an existing Unity project, or create a new 3D project. b. Navigate to Window > Package Manager. c. Next to Packages, select Unity Registry.
- 5. d. In the search bar, type "AR Foundation". e. Click Install. f. The AR Foundation package presents an interface for Unity developers to use, but doesn't implement any AR features itself. To use AR Foundation on a target device, you'll also need to install separate packages and enable the corresponding plugin for each respective platform.
- 6. To setup ARCore Plugin in Project: The ARCore XR Plugin package allows you to build augmented reality apps for Android devices. Follow these steps to install it in your Unity project if your application targets Android devices. Use the same version as the AR Foundation package to avoid any compatibility issues. In your project, navigate to Window > Package Manager. Next to Packages, select Unity Registry. In the search bar, type "ARCore XR plugin". Click Install.
- 7. Navigate to Edit > Project Settings. In XR Plug-in Management, open the Android tab and enable ARCore. To set up the AR Session https://developers.google.com/ar/develop/unity-arf/getting-started-ar- foundation#configure_scene
- 8. Vuforia Vuforia Engine is a software development kit (SDK) for creating Augmented Reality apps. With the SDK, you add advanced computer vision functionality to your application, allowing it to recognize images, objects, and spaces with intuitive options to configure your app to interact with the real world. Vuforia Engine supports AR app development for Android, iOS, Magic Leap, and UWP devices. Vuforia Features Vuforia Engine offers tracking a variety of objects and spaces that can be categorized as Images, Objects, and Environments. Use this overview to quickly get started with the solution and Vuforia features best fitting your needs. Features are, 1. Image Target 2. Multitargets 3. VuMarks 4. Cylinder Targets 5. Barcode Scanner 6. Cloud Recognition Service 7. Model Targets 8. Area Targets 9. Ground Plane 10. Vuforia Fusion 11. Device Tracking 12. Driver Framework 13. Recording and Playback 14. External Camera Image Targets Attach content onto flat images, such as print media and product packaging Instant Image Targets – Create Image Targets at runtime or from a database. Best Practices – Learn to make ideal image compositions for Image Targets. Device Database – Learn how to create the Device Databases in the Target Manager Multi Targets Use more than one Image Target and arrange them into regular geometric shapes (e.g. boxes) or in any arbitrary arrangement of planar surfaces with Multi Targets. Best Practices – Learn to prepare the images to properly be uploaded in the Target Manager. Cylinder Targets Recognize images wrapped onto objects that are cylindrical or close to in shape (e.g. beverage bottles, coffee cups, soda cans).
- 9. Best Practices – Learn to prepare the images to properly be uploaded in the Target Manager. Calculate the cylinder’s shape – Learn how to adjust the image size so it is easily wrapped. VuMarks These are customized markers that can encode a range of data formats. They support both unique identification and tracking for AR applications. Design Guide – Learn the essentials on all the design considerations to make a VuMark. o Make VuMarks in Adobe Illustrator – Follow our guide on making a VuMark, from start to finish. VuMark Unity Guide – Follow this guide to get started with the VuMarks in Unity. Barcode Scanner Detect a large variety of barcodes from your AR experience application Best Practices – Familiarize yourself with the conditions for detecting and reading barcodes. Barcode Unity Guide – Follow this guide to easily set up Barcode Reader. Cloud Recognition Service Recognize a large set of images and frequently update the database with new images. Cloud VS Device Databases – Learn the differences between the two database types and select the solution best fitting your needs. Vuforia Web Services – With the VWS API you can manage these large image databases in the cloud efficiently and you can automate your workflows in your own Content Management System. VuMark API generation – Learn more on using VuMark with Cloud Recognition. Model Targets Allow you to recognize objects by shape using pre-existing 3D models. Place AR content on a wide variety of items like industrial equipment, vehicles, toys, and home appliances. See the Comparison of Object Tracking Methods for help selecting the right tracking technology for your product. Create Model Targets - Use the Model Target Generator to quickly create a Model Target. Vuforia Creator App – Use the Creator app to test your Model Targets. Model Targets in Unity – Get introduced to Model Targets in Unity and add your augmentations to the object. Best Practices for CAD Models – Adjust your digital model to these principles and practices. Area Targets Augment real environments that you scan using the Vuforia Creator App or a commercially available 3D scanner. Create accurately aligned persistent content into a wide variety of commercial, public, and fun places to enrich spaces with augmented experiences.
- 10. Scanning an Environment – Learn the best practices on scanning an environment with each supported device. Vuforia Creator App – Use the Creator app to capture, generate, and test your Area Targets. Area Target Generator – Get this tool if you have a scan from one of the supported professional scanners. o Create Area Target – Follow this guide to make your first Area Target from a professional-equipment-made scan. o Area Targets Test App – Test your Area Targets on your device by following this guide Area Targets in Unity – Proceed to customize and add augmentations in the space with this guide. Ground Plane Enables you to place content on horizontal surfaces in the environment like tables and floors. Ground Plane in Unity – Follow this guide to get introduced to setting up Ground Plane in Unity. Vuforia Fusion Designed to provide the best possible AR experience on a wide range of devices. Fusion senses the capabilities of the underlying device (such as ARKit/ARCore) and fuses them with Vuforia Engine's features, allowing developers to rely on a single Vuforia API for an optimal AR experience. ARCore – See this page for setting up ARCore with Vuforia Engine in Unity. Device Tracking Vuforia Engine offers robust and accurate tracking by using the Device Pose Observer that includes tracking a Vuforia target even when the object or content is no longer in the camera view. Note that some Vuforia features are required to run on a Device Pose supported device. Enable Device Tracking in Unity – Learn more on enabling this capability. Pose Status and Status Info – The Device Pose Observer returns tracking states that you can use to optimize the user experience and tracking use cases. o Continued AR Experiences – Learn more on using the states to resume from interrupted and paused AR sessions. Recording and Playback Record and replay your AR session to test, experience, and improve your AR development workflow with the Session Recorder. Use the API or the ready-made SessionRecorder GameObject in Unity to record your Vuforia targets and continue to develop even when those targets or spaces are unavailable. Session Recorder API – Learn more about the API and recorder module. Vuforia Creator App – Use the Creator app to record sessions for your development.
- 11. Driver Framework Allows developers to provide and consume data from external systems through Vuforia Engine. Developers can also use the Driver Framework to record sessions and create custom drivers for their custom devices. Custom Driver – Learn here to create your own Custom Driver. o Vuforia Support for Custom Devices using Vuforia Driver – See this article for more information on using Vuforia Driver on custom devices. External Camera Access video data from a camera outside of the one in a phone or tablet when creating AR experiences. The external camera is used in extension of the Driver Framework. Camera Calibration – Learn more on setting up an external camera and adding it to the Vuforia Engine input. Setting Up Vuforia Engine a. Adding Vuforia Engine to a Unity projct Create an empty Unity project or open one of your existing Unity projects. 1. In the Unity Hub, create a New Project and select the Unity version currently supported by Vuforia Engine. 2. Select the 3D or 3D (URP) core template and enter a name for your Unity project. NOTE: Only the Vuforia Engine Extension is supported in URP. The Vuforia Core Samples and the Vuforia VFX Library are incompatible with URP since they contain unsupported assets.
- 12. 3. Create Project to open the empty Unity Project. b. Add Vuforia Engine Extension The Vuforia Engine Extension is a Unity Asset Package that can be imported into your Unity project. 1. Download the extension with the installer script from the Developer portal: Vuforia Engine Extension for Unity. Or, choose one of the alternative import methods documented in Vuforia Engine Package Hosting for Unity. 2. Import the downloaded Unity Asset Package from the menu Assets -> Import Package -> Custom Package. Importing the package will let a script make the necessary changes in your Unity project to install or upgrade to the latest Vuforia Engine SDK. See which version you currently have in your project in the Package Manager in the Window tab. To manually check if new updates are available, go to Help -> Vuforia Engine -> Check for Updates:
- 13. c. Vuforia Engine Setup The Vuforia Engine is visible in the GameObject Menu after it is imported. A missing menu could indicate that the Vuforia Engine needs to be imported. Each scene that tracks and renders augmented content requires an ARCamera GameObject. It supports apps for both handheld devices and digital eyewear. Steps:
- 14. 1. Add an ARCamera GameObject from the Vuforia Engine menu. 2. Select the ARCamera and Open Vuforia Configuration (CMD+Shift+V or Ctrl+Shift+V) from the Inspector. 3. Add a Vuforia Developer license key in the App License Key field. Add License opens the License Manager in a browser window. For a guide on getting a license key, see Engine Developer Portal Licenses. NOTE: Although measures are taken to store your license key securely, we recommend utilizing additional encryption solutions to further enhance your key's security. Tip: You can delete the default Main Camera after adding an ARCamera. The ARCamera also contains a Camera component. The Main Camera is unnecessary unless you use it to render a specific camera overlay, e.g., user interface or minimap. d. Adding Vuforia Engine Features After activating Vuforia Engine in Unity, you can add features from the Vuforia Engine menu to your project from the Unity GameObject Menu. You can also add the Vuforia Core Samples from Unity’s Asset Store. The Core Samples include and demonstrate many of the Vuforia Engine Features. The steps below continue to set up a simple Vuforia AR scene. Steps:
- 15. 1. Navigate to the Vuforia Engine Menu and select Image Target. or any other targets you wish to use. The Features Overview linked just above provides guides for using other targets. Each Vuforia Engine GameObject is configured in the Inspector. When a target is added, it will appear in the Hierarchy. 2. Select the Image Target GameObject from the Hierarchy and choose from the Type dropdown either a. From Image – Please see Instant Image Targets for further steps. b. From database – Image Target databases can be created in the Target Manager. c. Cloud Reco – Cloud Databases are also configured in the Target Manager. See Cloud Recognition for more information. 3. In this example, we select From Database. If there is no database with Image Targets in the project, Vuforia Engine will ask if you wish to Import a Default Image Target Database in a pop-up window.
- 16. 4. Click Import. 5. On most Vuforia targets, you can set an Optimize Tracking For mode that improves tracking for particular use cases. Select between Default and AR_Controller. Model Targets have an additional mode: Low_Feature_Objects. See Optimizing Model Target Tracking for information on the modes. TIP: Always maintain an accurate scale between the Vuforia targets and the physical prints or objects. Using targets that deviate in size from the real-life object might impact the tracking quality. See Virtual Scene Scale in Unity for more information. All Vuforia Targets include the following: A Vuforia <target type> Behaviour component lets you choose your target from a database and configure the target’s properties. A <target> ObserverEventHandler that lets you assign events and render conditions based on the tracking information returned by its Status and StatusInfo. A Target preview that lets you control whether the target preview should be visible in the scene during authoring. On many Vuforia Targets, you can Add Occlusion Object and Add Target Representation. They are created as a child of the target.
- 17. The Occlusion Object adds a GameObject of the model with an occlusion material that lets your object visually obstruct digital content. The Target Representation adds a GameObject of the model with the original material and mesh of the target. It helps apply shaders and colliders or highlight parts of the tracked object. NOTE: Virtual Scene Scale will not re-scale the occlusion object or target representation. Add the objects again after applying a value to the Virtual Scene Scale Factor. NOTE: Area Targets are created with a pre-computed preview mesh. The settings to add occlusion or colliders are available in the AreaTargetBehaviour component. See Area Targets in Unity for details. e. Upgrade your Vuforia target previews From Vuforia Engine SDK 10.8 and forward, the Vuforia target preview representations are changed to improve the development workflow. Upgrading your project to the latest SDK will display a message on your Vuforia Targets asking you to Upgrade This Target or Open Upgrade Dialog. Click Upgrade This Target to upgrade the target to the new representation setup. See Adding Vuforia Engine Features to see how that looks. IMPORTANT: Upgrading alters the Vuforia target by removing the old preview that is a child of the Vuforia target. Pointers and references to the old preview should be updated to use a Target Representation instead. Click the Open Upgrade Dialog to update all former previews in the scene or all former previews in all scenes in the project.
- 18. f. Adding Digital Assets Add digital content to your target as children of the target. This way, augmented content is only shown when that target is tracked.
- 19. 1. Add content as a child of the target by dragging your content on top of the target. o Add a simple Cube from GameObject -> 3D Object -> Cube or place content from Unity’s Asset Store or the Core Samples as pictured above. 2. See also the Vuforia VFX Library for free assets and shaders to style your AR app. g. Print Physical Targets A Vuforia target is, in most cases, a physical print or object. You can find these in /Editor/Vuforia/ForPrint as PDFs if you have imported the Core Samples. All printable Vuforia targets are also available here: Printable Target PDFs. Print without scaling.
- 20. h. Playing the Scene To see your scene in action, you can use the Vuforia Engine Play Mode in the Game View, which you can activate by clicking the Play button. Use Play Mode to evaluate and rapidly prototype your scene(s) without deploying to a device. Play Mode is configured in the Vuforia Configuration section, and you may use a webcam, simulator mode, or recording mode to test your Vuforia targets. See Vuforia Play Mode in Unity for more information. i. Building and Running your App Vuforia Engine-powered Unity apps are built and run like other Unity apps for Android, iOS, and UWP. When switching platforms between iOS and Android in the Build Settings, remember that Vuforia Engine sets and unsets the Render Over Native UI automatically.