How to Pitch Your First AR Project
3 likes982 views
The document is a guide on pitching augmented reality (AR) projects, emphasizing the importance of understanding users' needs, the risks involved in innovation, and identifying the right problems for AR solutions. It details the significance of mobile AR, discusses various sensors and machine learning applications, and outlines different user behavior categories to design effective experiences. The document also addresses the challenges and limitations of mobile AR, encouraging designers to think critically about content types and integration into real environments.
How to Pitch Your First AR Project
- 1. How to Pitch Your First AR Project @goatsandbacon Hello friends, Thank you so much for having me. I’m really excited to be here and to talk abut how you can pitch your first Augmented Reality project. What I want to do is share examples and processes that can help you scope and conceive an AR experience. A show of hands people here who have made or shipped an AR project? Okay great, lets share notes after this talk.
- 2. WHAT IS IT THOUGH? So before I get started, I want to address something. There’s a lot of debate around what this industry that we are in called? What field does AR fall under? To be honest, at this moment there’s no right or wrong answer. I think language is still evolving and once we start seeing more devices and adoption, we’ll have a better idea of what the best thing to unify on is. In that spirit, there may be language I use in this talk that might change too.
- 3. Spatial Computing INTRODUCTION For the moment Spatial Computing seems to feel the most snug so that’s what I’m going with. We think spatially and space is where we as humans have evolved. Our work involves using computers to extend our presence and capabilities in spaces around us so… spatial computing just works. This format is also inclusive of VR which is why I like it.
- 4. Exploring the horizons of technology requires courage because research carries risk” - IVAN SUTHERLAND http://cseweb.ucsd.edu/~wgg/smli_ps-1.pdf “ So Ivan Sutherland who is the father of computer graphics, he’s the reason all of us have jobs. And he wrote this phenomenal essay on Technology and Courage which has become my foundation. The reality about designing in this or any new format really, Is that there are risks involved that you need to mentally prepare for. Technology is moving at an incredibly fast pace and there are a lot of unknowns. “The risks that are difficult to quantify, that are different than money and resources, they are social and emotional risks, risks to reputation and to pride; they are risks that are felt but difficult to identify and describe.” Basically if you want innovate, If you want to truly innovate and be one step ahead, theres no room for pride or ego.
- 5. “My boss wants me to make an AR thing… What should I make?” That being said, there’s are some pretty OBVIOUS risks you can probably avoid. This is a sentiment I’ve ran across several times over the past few years. There is no ‘AR Thing’. There are problems that need solving, and these problems have constraints that may be overcome by thinking spatially. The first step in finding out if AR is the right medium is by identifying the users and their needs.Do these problems involve immersing the user in real time, assisting them in space or physically engaging them? Are there any physical constraints that currently prevent them from being successful? If so, then there’s a good chance that Augmented Reality can add value to the solution. Having your users experience rather than observe may sharply increase the chances of them understanding and retaining information.
- 6. Understanding Mobile AR BREAKDOWN For this talk I will specifically focus on Mobile AR - the reason for this is that most if not all of your audience will have a phone. This is the most scalable medium that’s in circulation right now. By getting ourselves familiar with this medium first, we become primed for head mounted displays and any other device that will come in the future. However, a mobile phone was not necessarily designed for AR experiences and therefore has a lot of points of friction.
- 7. Front Facing (Notch) World Facing Camera Machine Learning Capability Front Back A. Infrared Camera (Reads Data) B. Floor Illuminator (Face/Eyes Detection) C. Proximity Sensor (Activity/Human Detection) D. Ambient Light Sensor E. Speaker F. Microphone G. True Depth 7MP Camera H. Dot Projector (Face Recognition) Dual Lenses Image Stability Neural Engine Surface/Image Detection GPU (Realtime 3D Rendering) A B C D E F G H iPHONE XS But we really should understand how the phone works beyond just the operating system or interface. Hardware is easy to overlook when focusing on interface design and experience. It’s also not something we are taught about in design school. But if you’re working in new and emerging tech, knowing how things work can help inspire and even bring innovation to this space. Just a general awareness is enough to get some inspiration Hardware is also generally easier to scope and predict than user behaviors. It is vital for designers to explore beyond current technological constraints so they can help lead the technology forward.
- 8. Depth Sensor Gyroscope MagnetoMeter Accelerometer Light Sensor EXAMPLES OF SENSORS The most common method of ingesting information is through a camera. However, the most accurate way is to use sensors. A sensor is a piece of technology that detects information from its surroundings and then responds back with data. Really consider how a sensor can enhance your users experience. Depth Sensor: Calculates depth and distance. Gyroscope: Detects the angle and position of your phone. MagnetoMeter: Essentially a compass that can always tell where north is. Accelerometer: Detects change in velocity, movement, and rotation. Light Sensor: Measures light intensity and brightness. Having these in your toolkit will really help you consider more possibilities
- 9. EXAMPLES OF MACHINE LEARNING Facial Recognition Visual/Object Recognition Personalization/ Suggestions Predictions Chatbots It’s one thing to gather data but another to make it meaningful. For example, calculating the distance between a phone and a door can be easy with a sensor. However, trying to identify the door can be extremely difficult. The computer must understand what makes a door different from a wall or a window? It must also comprehend the different types of doors that it is likely to encounter. Facial Recognition is a super common ML found in camera apps and face filters. Visual and Object recognition identifies not just things around us but visual cues such as lighting conditions Personalization is probably the most common ML type you encounter day to day, it’s how your profile is shaped on social media Predictions is also very common, especially in apps like Lyft and Uber where the app gives you an estimated time of arrival Chatbots is essentially conversational UI that can learn through repetition and become more customized for unique cases, super common for support roles.
- 10. Defining Purpose & Value RESEARCH Properly researching and defining the problem space you are building for helps you in several ways. It provides you some evidence and data to base your assumptions off of. It helps you narrow your possibilities and avoid remaking something that already exists, It also might also make you rethink your hypothesis and if its the right one. Most of all it really connects you to the user you are designing for.
- 11. DEFINE USER BEHAVIOR Interact with Content IKEA Place Follow a Narrative Ghostbusters World Capture Data Tap Measure A really easy place to start is to figure out what the focus of your users behavior will be - is success based on them just interacting with something, like in Ikea place, where they view and place furniture to decide that yeah, I would buy this. Then you should scope for an interactive experience where the user will be able to manipulate content and have some form of control. In a narrative first experience, the success of the experience relies on the user performing a sequence of actions in order to reach a final goal or destination. Narrative first experiences are most often used for gaming and story based experiences, they also rely a lot on device orientation, location and maps. Or if Ghost bustin’ makes me feel good. In a capture first experience, the success of the experience relies on the user ultimately capturing content like a video or an image. A capture first experience is the basis of most sharing and platform based applications. It involves getting the user to create an artifact they can then take with them out of the app or in a different mode.
- 12. ENTERPRISE / EDUCATION Learn / Train iNaturalist Preview/ Try Before you Buy WannaKicks Instruct / Assemble Golfscope After you’ve figured out what your user is expected to do, ask yourself, what category will this fall into? These are all examples of real applications in the market currently in the App store. There is tons of metrics and good data on these use cases that you can use to really sell your vision. I’d say we are at a moment where it can be possible to do a viable competitive analysis or audit. You can really zoom in on one of these several categories and start from there. An example is iNaturalist that lets you learn about wildlife around you Preview/Try Before you buy like Wannakicks that lets you try on sneakers before you spend $1000 bucks There’s also instruct or assemble apps like Golfscope that help you improve your golf game.
- 13. CONSUMER Social Facebook Messenger Gaming Pokemon GO Brand Engagement Miller Lite by Trigger Facebook Messenger offers a ton of social interactions where you can use the front facing camera to play with friends. Pokemon GO - This is a great example of why research is important, The fact is that most players in Pokemon GO actually disable the camera AR feature when catching a Pokemon because it actually makes the game harder - however there’s TONS of other AR capabilities that are worth looking at such as the actual weather influencing the Pokemon you see, or location based data. Then we have brand engagement like this miller lite campaign made by Trigger for St Patricks day where a little friend pops up and serenades you as you drink.
- 14. UTILITY/ TOOLS Measurement/ Accuracy KLM Airlines Location/Guide Google Maps Creative / Authoring Facebook Stories KLM Airlines lets a user double check to make sure their carry on fits in the overhead compartment., Google maps is great for people like me who will forever struggle with left and right, also which way Missisauga is. Facebook Stories activating a real-time drawing system where you can design digital paintings and then pose them throughout the real-world.
- 15. ART & DELIGHT Face Meme Dan Miller Play the City Tool of North America Childish Gambino Playmoji Google Playground The there are Apps that are just fun and bring delight. These types of applications and experiences also tend to be the most experimental and really push the medium to its potential.
- 16. Language & Foundation CONTENT So after you’ve determined what problem space you are looking to work in. Paying attention to the language you use and how you describe things is important on so many levels, words have meaning and if you’re all on the same page it creates alignment with you and the rest of the stakeholders. You’re sort of in uncharted waters so make the effort to be on the same page. One less confused person goes a long way in your product cycle.
- 17. So The entire point of AR graphics is to overlay data in space, That means it’s a 3D experience even if you’re working with 2D graphics. If your experience isn’t visual, it’s still happening in real time. Traditionally, in 3D software you would craft your scene and then to properly view it you would render it out after you make something. AR as a medium cannot afford any lag or delay so it needs to happen in real time. Get yourself comfortable with the term Real Time 3D because it’s a known authoring convention that has a ton of process and documentation that can help inform you and make smarter decisions.
- 18. Static Content that is still and lacks movement and interaction Animated Content that moves on a timeline or follows a sequence 3D Object with width, height and depth. Dynamic Adaptive content that changes with interaction or over time Procedural Content generated automatically or algorithmically CONTENT TYPES The following are examples of some of the more popular content types used in spatial computing. These content types are not exclusive and can combine in many different ways. However, it is essential to understand these formats so the designer can properly articulate what they are trying to do.
- 19. Dynamic 2D Flexible Glass Dynamic 3D Fixed Space DESCRIBING BEHAVIOR Type State Location State LocationType When mapping out behaviors and relationships, it is helpful to be specific about where and how to treat the content. Try to be as precise in describing the experience to get alignment amongst stakeholders. A good rule of thumb is to call out the type of content, the state and the location.
- 20. ADDITIONAL RESOURCES Designers Guide to AR https://medium.com/@goatsandbacon 3D For Designers www.3dfordesigners.com I’ve actually written quite extensively about content types which you can read on Medium. If you’re an absolute beginner to 3D, I would recommend 3D for Designers, It was Created and taught by design industry veteran Devon Ko, it includes both free lessons and a comprehensive 3D design fundamentals course. I’m also a TA on this course so come say hi!
- 21. Discover, Prototype & Design TOOLS Once you’ve figured out what type of app you’re going to make, the next step is to figure out to how your app will look and what features you’d like to invest in.
- 22. DISCOVER: GAMES https://80.lv/articles/the-design-secrets-of-breath-of-the-wild/ Zelda: Breathe of the Wild Nintendo Overwatch Blizzard Conduct AR Northplay In the discovery process you will do yourself a great disservice if you don’t look at 3D games. 3D games are probably the most similar in design treatment and challenges since in both 3d games and AR. your layering real time 3D data over a camera controlled by the user. There’s a lot of visual treatments done in gaming that can inspire your interfaces - for example overwatch uses color burns to make sure their UI works on both light and dark surfaces Another reason to look at games is for the game theory and tricks they use to make a more compelling game play - Matt Walker who is Production Manager for Capcom translated a series of tweets by the game director and senior lead artist for Zelda Breathe of the Wild. He talks about the “Triangle Rule”. By using triangle shaped obstacles they complete 2 objectives- gives players a choice as to whether to go straight over the triangle, or around, and it obscures the player’s view, so designers can utilize them to surprise players, make them wonder what they’ll find on the other side.
- 23. RAPID PROTOTYPE Storyboard Invest in a clear story from beginning to end to determine a feature. 3D Room Simulations Use 360 videos, 3D models or a sequence of snapshots to simulate an environment Animated Clickthrus Use timed transitions using Framer or Adobe XD to create clickthrus on your phone to get a sense of ergonomics and interactivity. One of the challenges in making static prototypes is that things are always changing in your camera feed, and so using methods that have animation or follow a timeline and narrative can help communicate the changing nature of AR and your interface.
- 24. TRADITIONAL COMPOSITING After Effects Import phone footage and use 3D Camera tracker and camera solve to create 2.5D simulation Cinema 4D Import image sequence into Motion tracker to generate 3D camera information If you can composite or do some motion graphics its even easier to conceptualize how UI might look on a moving camera, If you’re not as familiar, there are tons of tutorials online to help you do motion tracking. Tracking software is super helpful. Tracking basically means to overlay position size and rotation data of something over a video so that it looks like it belongs in the same scene. A lot of principles of making things look like they belong in video compositing can be inherited in AR.
- 25. DISCOVER : PLAY You have to play to discover opportunity. I find it super helpful to challenge myself with random pieces of footage that I find around me. It’s a super low pressure way to just create and learn.
- 26. REAL AR AUTHORING AR Foundations (Unity) iOS & Android Project Aero (Adobe) Currently in Beta Torch AR iOS There’s really nothing quite like the real thing - There are a few options out there to prototype and author experiences. There are tons of tutorials and information online on how to get started with AR foundations in Unity - The upside to mocking up in Unity is that your entire app that you plan on publishing can be made once in Unity and be built out to ARKit and ARcore.
- 27. Limitations of Mobile AR HEADSUP So we’ve discussed figuring out what your user wants to do, what category you want to focus on, how you can start prototyping and getting a feel for your idea. But there are a few areas of friction that I’ve personally encountered that you are likely to encounter too.
- 28. Semantic Understanding Visual Inconsistency Platform File Extensions Relocalization CURRENT HURDLES ? Triangle The promise of AR is its integration into the real world, yet we’ve seen very limited examples of that. Right now it’s tricky to get the world to interact back with you. These are a few of the hurdles I’ve faced. Semantic understanding basically means does the device know exactly what something is, how it should behave? This technology is not mature enough to be reliable for anything that isn’t a basic model. For example ARkit and ArCore don’t tell you what the floor is. Visual Inconsistency : Things just feel like they float and don’t quite look right, things don’t occlude or match the expected perspective. Platform: Some things that are specific to a device or platform and not really in your hands or control - take the time to learn about these and if you can, the company roadmaps. File Extensions: the industry has divided itself in terms of support. For example, Apple exclusively supports and encourages the use of .USDZ for ARKit applications, however Chrome does not support this format on the web and promotes .glTF. If you are considering commissioning or buying a 3D asset, my recommendation would be an .fbx since you can covert it into both .glTF and .USDZ based on your needs. (Relocalization is where the device knows its position and orientation in space, that helps create a persistent experience that then allows for returning and multiplay experience - so it means things are where you had left them or someone else can visit an experience you were just in in the exact space
- 29. Architects of a New Reality LEVEL UP I want to challenge you to think beyond what the current apps that I’ve shared are. The amount of information and access to information we will have will grow and so how can you prepare yourself to create the next generation of apps? This is a phrase I’ve used before in Context design and it makes a lot of sense to consider yourself an architect since you essentially design spaces
- 30. This is Ellen, a 3D model that you can download for free from the Unity asset store under the 3D game kit.
- 31. CONTEXT OF A TABLE Elevated from Floor Small Table Size Horizontal Plane Lets say you want to have your 3D model appear on a small table. So what is a table context? Context is basically the circumstances that form a setting for an event or idea. If the following requirements are met, I want Ellen to spawn on the table.
- 32. The reality is that you have no control over what a users room could look like, at all. This is basically the fuzzy area you have to design for.
- 33. But if you can design different bits of contexts and figure out a way to make your experience work in an unknown environment
- 34. You can figure out how to make it work in any environment - and really extend the value of your design. This is true immersion because you adapt contextually and responsively to the space around the user.
- 35. Mixed Augmented Reality Studio (Project MARS) * Currently in Preview UNITY LABS PRESENTS This is a great way to talk about what our team is working on in Unity Labs.
- 36. CONTEXT FIRST AUTHORING Context Content What would otherwise take a lot of time and math we are making into easy to use visual tools so you have less mental overhead when embracing this new type of authoring.
- 37. PROJECT MARS This is what the app allows you today. In MARS you basically have objects that you give affordances to, and then in the app you can preview & iterate against simulated & real data. We give you a set of rooms that you design against and can customize. This makes you create as many contexts as you need to match as many spaces with the desired experience.
- 38. PROJECT MARS Here is something that was built in MARS today - What’s great is that as a user starts scanning, you can also design an experience that adapts and changes as they get more of a room and different surfaces. This sort of procedural authoring could change AR from being these bite sized experiences to something you actually live and interact in. If you’re interested in trying it out for yourself, definitely talk to me after. I’ll also have my email up at the end.
- 39. When human systems and tool systems align - it leads to an accelerating rate of progress” - DOUGLAS ENGELBART GROUP OF EXPERTS HARDWARE/ SOFTWARE DATA/ INFORMATION/ KNOWLEDGE COLLECTIVE INTELLIGENCE “ I kinda want to end this talk on a note where I need your help for very selfish reasons - the more we can help to improve our collective intelligence, the better it is for me.. I mean us. For real though, I am deeply interested in how humans and tools evolve, I strongly believe that how we all work together on important challenges is a measure of our progress and how effective we as a society are. I think the pace in which tools evolve can be really good indicators of progress. DOUGLAS ENGELBART