The Trouble with All Those Boxes: Designing for Ecosystems Instead of Screens

Editor's Notes

• #3: …We all design-- and I mean technical design as well as UXdesign -- based on boxes. We love our boxes.
• #4: Whether they are big, metal boxes moved by forklift or the little plastic boxes in our pockets, we tend to forget that they are connected, shared and used to do work in the real world.
• #5: And because it’s all about the computer, we forget to “think like a human.” We design features, systems, tasks and content that fit into little boxes. Into computer screens, and flowcharts that show paths between little boxes of information. We design processes that progress in predictable, linear ways. In ways that work like businesses, and like computers. That think of the the system without considering the user, and present information in little boxes of content, on glowing screens. No, the guidelines I have heard even here today of making simple processes don’t work because users don’t have this diagram, and don’t think like that anyway.
• #6: And by the way, we call these “systems,” but we don’t really mean it. Systems are inter-related, multi-part, and generalized. Systems are toolboxes with drawers full of capabilities. When most of us are building a website or application, we’re building tools. Tools do one thing, in more or less one way.
• #7: There have been plenty of complex information and computational systems for a long time, but networks changed everything. If we think about networks, we consider them as connections between machines, delivering packets (more boxes on the diagrams) of data between them. Network connectivity is just another feature of the device.
• #8: But that’s not quite right. Networks do not connect computers. Networks don’t even exist, really. And thinking of networks as packet delivery mechanisms doesn’t really help. Networks are instead just ways of making our computers bigger. Of making the world be one, large, distributed computing and storage system. You don’t use the cloud, you are part of it.
• #9: When you think like this, you should start noticing the boundaries between channels or device classes start breaking down.
• #10: Think about that.  How would you design your product if it wasn’t for an iPhone screen, but for the whole world? I am not kidding, or weaving crazy tales. How does, say, Facebook work? Or Twitter?
• #11: They work on the Web, both mobile and desktop. On apps and widgets on handsets, and computers. On smart TVs, and Twitter especially is famously over SMS. In email. Don’t underestimate the power of email. Via push messaging platforms. Via APIs. As services. These, and others, are services that can be used by their owners, or in some cases by anyone, to offer the product in all sorts of ways. In ways that could not be predicted when they started. But they had one key winning attribute: that theydon’t need to predict in detail how it will be used, on what platform. They can just build servicesso we can build interfaces (or let others) to any platform, any time they want to. Yes, that’s a kiosk. And the featurephone? Last time someone could calculate the numbers (they are hidden now) 82 million regular users of the J2ME app on FEATUREPHONES. Less than half the Android or iOS platform, but that’s a lot of people; don’t you wish you had a spare 80 million users? It is a lot of opportunity when 80% of the world is still on featurephones.
• #12: Lest you think everyone big and successful is doing it right and not missing a beat, let me use a counter-example. I don’t use Flicker much from mobile because it fails me. When I click a link in an email, or Google search, I most often get… the desktop website inside the browser. I am not even so worried about them not having a mobile site:They need a custom URI. They need to launch the app they are so proud of every time someone thinks “Flickr” on mobile. They are thinking of boxes, instead of systems and have checked the box for “mobile app” without thinking about the whole ecosystem or how we use this stuff. The other day, Dave Malouf sent me some tweets about how he couldn’t clearly follow someone’s Flipboard, as it… I can’t even remember. But does something that presents you a blank page if you don’t press the right button in some popup. We are smarter than this.
• #13: Or back to Flickrhere, they are chasing the competition onboxes. Check the box that says “we offer filters!” Why? Does this meet some specific need of THEIR users? And why is it only on the mobile app? It’s not on the desktop, so is clearly not a service but a little widget residing locally in the apps. I think they even have different filters between the iOS and Android apps.
• #14: Some people will go even further. Our happy glowing screens on all our beloved devices are actually getting in the way. I’ve been told by someone very clever about networks “what we’re doing is trying to translocate information, and all the machine does is impair that process. Possibly infinitely. You have a choice over where it goes, and that’s it.” Networks can be thought of only as delay-engines, and all our access platforms (our phones, computers and web browsers) as methods to divorce us further from the innate truth and meaning of the information.
• #15: It doesn’t mean you don’t do the best possible job when designing interfaces. But first, you allow arbitrary access, so there’s customer choice; they get to pick the platform they are most comfortable with.And you need to push to platforms that fulfill these needs. Any platform. Screen-free platforms, like this are critical not just for special classes of uses, but to assist everyone in using technology all the time. We get “temporarily disabled” so need vibration and voice readouts. And users become exposed to your information from several sources, so it becomes almost ambient. Think about where and how your information can do the most good.
• #16: We should never expose the edges of the system like this, much less require the user to tap the screen (button or not) to have the system try again. We need to design and build with an understanding… no, an expectation of delay, and difficulty. Both technical delays and because of our lives. Users get interrupted, and switch platforms. Networks fail, and stall, and delay. We have to cache data, allow working offline, build for multi-channel, multi-device use cases. We have to automatically synch and not cause errors when there are mismatches. We should build resilient systems that encourage sharing so users can resume tasks when they get home, or get to work and sit down.
• #17: If you don’t get what I mean, or why, let’s take a specific case, maps. This is not about how Apple tried to get me lost driving in Europe. And I did the screencaps a couple months back so it may be a bit out of date. You might need maps when you are far into the boonies. Or you want to know where you are going next while in a tunnel, in one of the no-network area on an underground subway. A month or two ago, I used an iPhone on my bike mount driving around San Francisco, but didn’t bother to put in a SIM. I have lots of phones, most don’t have plans so I carried another one with me as well. I just cached the maps and used GPS alone. Implementations vary, in ways that are useful to illustrate this: Google maps on iOS does… nothing. If you are off network, you get a useless blur. Google maps on Android does a better job of having a “basemap,” a permanently cached global map, but you can maybe find half a dozen towns per state, and maybe none per country in sub-saharan Africa. They do have a service to manually cache areas, but you have to plan ahead. And can only save so many of these. iOS Maps has, despite other flaws, probably the best overall solution: It automatically caches. If you have looked at the area lately, it’ll keep it stored offline. So if you plan your trip, then go there soon enough it’s saved.
• #18: If you do any of this adaptive stuff wrong, or don’t do it at all, users will notice. I like to use map caching as an easy-to-see example, but generally, users are intolerant of bad systems. There is an increasing body of actual research that everyday users do not use, for example, tablet apps that are obviously just phone apps stretched to fit the screen (photo). Or, take Facebook’s hybrid app failure. This was not a nerdy developer and designer issue, but everyday people rejected a major service and stopped using the app. We know this because they admitted to a sustained 4-fold increase in use almost overnight, when they launched the first native app. Do not dismiss these concerns as nerdy, niggling disagreements. We’re all nerds now, and your users care.
• #19: Let me show you another example. Pebble, the eWatch, changed my life. Not really. But it’s really a shockingly interesting product and I’ll tell you about it if you care sometime,except there are oddities and pitfalls. The first use experience is a perfect example. There are no instructions on the box per se, it has just a website address, and the device says almost nothing when you turn it on. The link could have been emailed to me. Better would be that they help me get the app installed a few days before the watch arrives, also with email links or something similar. The watch has not enough info, and the app doesn’t make much of anything clear. You have to read the web page, which might as well be a printed VCR manual. It’s long, it’s tedious, it’s jargon-tastic.
• #20: Okay…I think most of you are nodding, or have similar thoughts from the exercise, because we already know how to do all this the right way. And I suspect everything else I’ll tell you in the next few slides you know also. Connecting the two is what is key. Some of these concepts might be new, but most of all I am talking about creating cultures that encourage building great products. There are a lot of principles and practices and concepts and examples and pithy phrases I could give you. I do change my mind and modify this periodically, but today I think if we could all get to a common understanding of these three points, we could do a lot better for our products, our clients and our users.
• #21: Design for Every Screen is my attempt at a clever catchphrase. Of course, I don’t just mean screens, but “platforms” somehowsounds less great and they are sorta not real either so that would be worth arguing also. It’s essentially whatJakob Nielsen calls the “Transmedia User Experience,“ which is even less catchy, so I don’t feel so bad. You can also think of it as designing agnostic of any interface (maybe even with NO interface), or designing for services.
• #22: This is also my counter to things like how “Mobile First,” which isn’t bad, butis too often misinterpreted to mean a different excusive choice.While it's better than “desktop Web first,” Mobile Firstis a fundamentally Web-centric thought process, and is too focused on screen size; it's the kind of thing that meshes with RWD, whereas there's a lot more to design and development. If you like Responsive, skip it and think Adaptive instead.
• #23: Pragmatically, tactically, there are too many mobiles also. Are you supposed to start with the lowest capability mobile device, like here where I've got a search dialogue. It works well enough for that, but…
• #24: …the capabilities of more advanced mobile platforms allow much more interesting interactions, which have nothing to do with size, and not much to do with input method.If I designed the one first and progressively enhanced, I am not sure I'd have come up with these.
• #25: In the end, I meandesigning for people. About half my work is for Big Dumb Companies, for legacy systems and services and to integrate with or build on existing products. It means when you do the discovery work you already do, like seeing how people shop for greeting cards (photo) you don’t just take it as input but turn that directly into task flows and architectures. You don’t make assumptions about presentation layer, platform or technology, but you let the user needs (and business needs, and so on) but let those evolve organically from the design process.
• #26: When you write, or draw concepts, they should talk about services, data, sensors, networks and users. Not screens. Don’t just put your goals and objectives on the wall, or on the first page of the deck, but bake them into the diagram. This is what I think of as a Task Map, and my Customer Journey Maps look similar. With no start, no stop and no happy path. Lots of circles.
• #27: Storyboard. Think about processes from entirely the user’s point of view. Storyboards are a great way to force yourself to think about users, their context, and their behaviors off the screen.
• #28: Of course, you can also do screens the same way. Draw comps parallel to the storyboard, or like this with bits of the user and context. And, you don’t need to settle on a platform for this. Here, it’s SMS, notifications and apps. Sketched out as part of the user task.
• #29: Detailed IA diagrams, task flows, system models like this are just a natural extension of these concepts. Draw these as evolutions from the concept, storyboard and other phases. Designagnostically, as we already do with IA principles and deliverables, then branch (by module, not page!!!) to the platforms you identify is the way to make sure you design the best experience for each of those platforms. I call these Blueprints to differentiate from the similar task flow IA diagrams that you make as you branch this to each platform.
• #30: Never let project constraints keep you from conceiving of future interfaces, to make sure the architecture, the whole solution, is future friendly. When thinking of systems, or creating blueprint level diagrams and concepts, “never say never.” Okay, this particular smart TV widget didn’t launch yet. But it’s on the roadmap, and everyone from the CEO to the data architects knows it is.
• #31: Summary of
• #32: Mobile is More. Not less.
• #33: The standard line, which I am /still hearing people say/ is that mobile is smaller and less capable, so you must “ruthlessly cut” features, content, etc. This means, in comparison to desktop. A tablet is in between, so you cut half the features, I guess.
• #34: WrongThat assumes desktops are the ultimate. Everything else is less. Mobile is smaller.
• #35: But there’s more to it. Every device is different, and every device is good in its own way. For scale, size hardly even matters in many cases. Angular resolution and viewing distance cause a lot of this to disappear.
• #36: Not disappear in the sense of merge, but because the capabilities and interactions are matched to the way they are used. Embrace these differences.
• #37: These differences in device capabilities are great. This is stuff that gets me excited! Mobiles are in no way like desktop computers. And they are not like cars. And your smart thermostat, or watch are not like each other or anything else.
• #38: In a lot more ways more than the size of the screen. Or the input method. Most of all due to connectivity and sensors. Many devices are taking the mobile cue of being very aware of their environment, and their user. And also, critically, more seamless access to the features most computers have like connectivity. Data entry, for example, can be made trivial:by removing it. Since the device is reliably connected, and full of sensors, it canalready knowwhat the user wants.
• #39: I know this isn’t a mobile phone, but it’s a great example that’s easy to see and understand, and is a good way to talk about how no-interface and pre-emptive tasking is probably the future anyway. It’s a hand-washing station. You press a button for water, then a button for soap, then a button for water, then a button for air, then dry your hands on your shirt. They each come out different places, so you move your hands to different positions. Which is not enough to turn it you. You really do have to press a button. Wet, soapy, etc. In this day and age. Unsanitary? Mostly, un-necessary. Sensors are (seriously) cheaper than buttons and can make the experience a LOT better.
• #40: We have to design, each platform, every platform, to take advantage of the features inherent to it, and the unique environment, and user expectations.There are sexier items like location, but sharing is my favorite example of mobile vs. web. Not the concept, but the implementation where you just press “Share” is brilliantly good stuff. It appeals to the way people cognitively address tasks like this. If you are building email forms into your desktop website, that’s correct. That’s the standard. But for mobile web and apps, that’s wrong. You link to installed applications because that’s the expectation and method of operation of mobiles.
• #41: Do whatever is right for whatever system you and your users work with.TV, and kiosks and telematics and so on will all have their good points. And these are ALL here. Millions of Smart TV downloads have already been sold. You’ve all seen you can now develop for cars. Mobile handsets and tablets are, relatively, easy to understand. You have to stop thinking of mobiles as tiny, crippled desktops, with weird touch interfaces that mean you can’t use hover states. You need to embrace the network connectivity, the portability, the presence, the sensors, the intelligence, the integration, the personalization.
• #42: Time to really draw. Take the concepts and states in your task flow, and draw the interfaces.Draw everything at device scale. I have a few sheets of phone outlines to help with this, but you can just draw boxes on your page that are close to scale instead. Do not forget interactions; show paths and motion and delay. Design by widgets. Think not about screens, but components. How can this component be used other places. This can be a good team exercise, as one draws a widget someone else is marking on the task flow diagram all the views or states it will apply to. Use colored markers and letters or something to annotate the relationship.
• #43: …Who here has heard of Resilience Engineering, as an engineering and security practice? It is used to keep services like Google, Facebook, Etsy, Flickr, Yahoo, or Amazon online. At a deep engineering level they follow practices and procedures to assure their systems are not brittle, and avoid failure or fail gracefully and can be fixed easily. Well, we rarely need to design end-user systems for true catastrophes, but for any digital product there are many failure modes we simply are not accounting for. And we really should.
• #44: Resilience is usually defined as the ability of a system to absorb disruptions without tipping over to a new kind of order. A building when exposed to too much lateral ground movement settles into a state of “pile of rubble on the ground” unless you design it to resist this disruption adequately. We approach all design of technical systems from the engineering perspective, and assume we can codify the process, so predict failure points. We can’t. Our systems are embedded in other technical systems, embedded into complex systems, like the user carrying it around, who may be subject to rain, and traffic delays, and screaming babies.
• #45: I say there’s something called Resilience Design. Or there should be. Here’s a simple example… I also still wear normal watches. One is a dive watch, because it’s shiny, not because I am a diver or anything. It is one of those with a twisty ring around the outside. That part with the numbers twists around. If you don't know, and I didn't until recently, this is used as a simple timer. But on mine, and on all dive watches (vs. Aviators watches), the ring only goes one way. The clicky detent lets it go counter-clockwise, only. Why? … Because it's for timing remaining air. The ring might get bumped and change it's setting. Having it show less time might be inconvenient, but going the other way might kill you. And, you don't even need to know this. It just works. That's the sort of brilliantly-simple answer I am talking about with resilient design.
• #46: This [watch] might seem like it is dirt simple and free-standing, but it works like that because of it is part of a system. You wear it. You do things that require timing. Resilient structures are designed as parts of interconnected networks, not as free standing items, and not hierarchies or linear processes. We need to design our systems to work with users’ messy lives. To account for arbitrary complexity, chaos, and failure.
• #47: Failure means anything you didn’t predict. It means planning on users missing the target on their small touchscreen. So it’s not catastrophic, and maybe so the whole system works better anyway. It means you never say the word “happy path” again, much less design for it. Users do not enter your site (or app) from the “home page” anymore. Stop designing like that. (This is a site I run. These are real numbers.) I don’t pay much attention to the way the home page works anymore.
• #48: More specifically, and maybe easier to get your hands on, is that people cannot click perfectly. There’s a talk tomorrow morning about these figures, but if you put two click targets too close, people will hit the wrong one way too much. This Twitter client is typical. If you try to click these links, you can miss and hit the user line instead.
• #49: This other Twitter client – just to prove that it has nothing to do with the type of product – solves it easily. These look like links, but clicking anywhere on the line opens up this drawer with options.Sure, it’s a second click, but it’s better than accidents, and once the user is accustomed to it then it works real fast.
• #50: Say you are building a store locator function as part of your tablet app. You don’t ask users to type their location in, do you? Because the device knows already. I don’t even put in “Search” buttons for things like this (not that I claim to have built this Square wallet app I am showing off here), and at least for maps, most designers don’t. When you open the feature, you get a map. With results already fetched in the background so there’s no delay.
• #51: Even when you allow typing, you do it intelligently. You think about what users know, care about and tolerate. This IS something I am working on (it’s not launched just yet so don’t complain about the details; there are probably open bugs on them). It automatically detects location and populates it. Note here it’s only got coarse location so we don’t use the centroid of the area and call it a street address. We use the data we have correctly, within the constraints we know. But it also lets the user type. Not lat/long figures, not precise addresses. Who knows the postal code when visiting a store? Nope, we use the Google Places API which lets you type in place names. Stores, landmarks, stuff that people think of the way they think about it. Yes, it offers autocomplete options, so the user can type just a few characters sometimes, and tap to pick from the list. And this didn’t even muddle our data structure as any result the user accepts turns into a geocoded address in the location field and a store name in a field elsewhere.
• #52: I have mentioned modularity a few times, and wanted to briefly touch on how important that is. Remember all those complex, interconnected systems? It is mathematically impractical to address every use case. When considering the available combinations of choices a user could make on many systems, combinatorial variations are in the billions. I did some rough math on one project and determined the documentation and analysis would take longer than the history of the universe to that time to complete. This one [image] isn’t that bad, but is a real example of a not-very-complex website a few years back. We can discuss development and QA efficiencies of modularity, but in design it gives you additional advantages of evaluation. If there are six states of a search module, that’s easy to consider, and adding one is no big deal. Designing and specifying those states per page is orders of magnitude harder. Evaluating them with all plausible information types can be impossible.
• #53: Methods: Principles are great, but how do we do this? 1) Evaluate by widgets. systems /thinking/ is great, but evaluating at a systems level is literally impossible. So, for each widget or module,figure out what all modes it operates in, including error states and other sub-optimal conditions. 2) Tolerance. How much will the user, based on their abilities, their context, and their desire to complete the task tolerate failure? Thinking of this, you'll see that failure can be not system errors; required registration before using a website is part of this because the user doesn't want to do it, the registration system gets in their way, and since they don't want to do it, they have low tolerance. Then you end up with reduced registration… You could even use HF methods to calculate the difficulty of selection and input, and the cognitive load, and if you work in a life/health/safety field I'd do that, but I never bother. Low, medium, high with a brief explanation is good. 3) Severity. How bad is the error or unexpected condition? You can even assign numbers to it, which can be pretty accurate if you do it as a team exercise, with everyone doing an expert review /by persona/, you can also cross it with the tolerance level per widget and get a hint as to what the specific pitfalls might be, so you can fix the most troublesome conditions.
• #54: Take your existing designs, the task flow and the interfaces, and use these techniques to review for failure. The evaluation process cannot be so formal in the short time we have available, so just take whatever comes to mind quickly, then modify the design to fix it. Do you need to change the whole flow, or would changing a button label or position be enough?
• #55: In case you haven’t picked up on that, or I accidentally implied these are steps in a process, they aren’t. You should use all these during your design process, but routinely. If not hourly, then check for resilience and consider platform specifics daily. Eventually, you can do this automatically. It becomes second nature. To me, I am serious when I say this is developing a philosophy of design.
• #56: So, whose confused, disagrees or doesn’t see how it applies to them?
• #57: APPENDIX: Slides that didn’t make it due to time and flow. But I might use as examples during Q&A. If this multi-platform stuff worries you, or if you are someone who gripes about fragmentation, you are doing it wrong. First, we’re UX designers. The users are making choices of these platforms, and all these device sizes. They are not being tricked into it, they are not cheapskates no matter what Gizmodo says. You must to respect user choice of device and platform. Use analytics and not opinion to make your decisions. Implementation-wise, another platform is a simple, easy extension to your core /service/. I’ll say it again: most of us should not be designing and building apps or sites, but services that happen to have a variety of front ends. How easy is it to extend? I know people who have launched on another platform by adding one person to the team. One Android developer, some designers you already had on staff, and literally weeks later there’s a beta version to play with. I know people who have had alpha ports to a new platform within ONE DAY.
• #58: APPENDIX: Slides that didn’t make it due to time and flow. But I might use as examples during Q&A. If you think your favorite platform is not subject to fragmentation, you are just ignoring the people who can’t use your product. iOS has different, but broad issues of device and OS compatibility. Sometimes, depending on the product you are making, worse ones than Android.