Taking Your Product Development to the Next Level with Full Stack

Editor's Notes

• #4: Welcome to Take Your Product to the Next Level with Full Stack — the final breakout session at Opticon! I’m Jamie Connolly, Senior Product Manager at Optimizely. I lead our Full Stack product, as well as the core app platform that serves both Web and Full Stack. I’ve been with the company for six and a half years — this is my sixth Opticon. I’m incredibly excited to be here today, and to speak with you all about this subject.
• #5: This presentation covers four topics: - Why we at Optimizely believe product experimentation is so important — i.e. the reasons we built Full Stack - What’s required from a product experimentation platform - How Full Stack works, and how our customers use it - And finally, our roadmap – the features we’ve built — and are building now — and the reasons those features are important
• #6: But first, I want to begin with a celebration Today is Full Stack’s third birthday!
• #10: Let’s consider an example — AirBnB’s website, or your company’s website - The set of things that you con test on the frontend is limited … - But, there’s so much more beyond the frontend - Search, recommendation, the algorithms that you might use to personalize your site, or help your users discover your products. The business logic that defines your customer experience
• #11: And for many businesses, your website is just one of many customer touchpoints. You may have mobile experiences, OTT or IOT experiences, perhaps a kiosk in your physical store. Your call centers / support channels, the algorithms that route product through your warehouse and into customer’s hands. Ultimately, then frontend of your website is just the tip of the iceberg. It’s an important tip – acquiring new business is essential. But if you’re only experimenting on the look and feel of your acquisition funnel, you’re just scratching the surface.
• #13: Let’s look at Optimizely, as a specific example. If you count lines in our codebase, the marketing website — i.e. our acquisition funnel — represents about 13% of our customer experience. Our product covers the remaining 87% And, this estimate is conservative — I didn’t differentiate between server-side marketing website code and client-side marketing website code. If I had, the distribution would be even more skewed. While this is just a rough approximation, and while your business may be different from ours, it provides a loose sense of just how much you’re missing if you aren’t experimenting in product
• #19: If you aren’t familiar with Full Stack, I want to introduce you to a few companies who are having success today, how they’re experimenting in their products Examples of their use cases Don’t talk about pricing
• #20: It's worth mentioning that there are several different kinds of rollout you can do. One of the most common ones is to have something first enabled just in a local environment for a single developer And then roll it out to a staging environment.So whether that's a preproduction environment or a staging, you can actually use these rollouts to launch something in there. And then to go from staging out to maybe a small set of Beta users. So there's a different kind of rollout that's actually targeted to a certain audience or a certain whitelist of people that you've specifically decided should see your new stuff. We do this all the time. As a B2B company, we find friendly customers that wanna give new features a try, even some of our bigger companies might be more sensitive. We're not quite ready to try out a new feature until it's more fully baked. From there, once you've validated in the Beta, you can do a gradual rollout where you slowly ramp up traffic to a larger audience and roll back if anything goes wrong. And then and only then we can do that 100% rollout where everybody has a new feature. And the key idea here is that we can mix and match these techniques. It's not that one of these is the right answer. I think for each different business and each different product or feature within that, you wanna choose what the right rollout strategy is. But I would argue the right rollout strategy is almost never to just flip the switch. The gradual process is usually much more reliable.
• #21: If you aren’t familiar with Full Stack, I want to introduce you to a few companies who are having success today, How they’re experimenting in their products Examples of their use cases
• #22: **** Call BBC out as “one of our longest tenured Full Stack customers *** This was already highlighted in last year’s keynote Either call that out o r replace Anyone who say it last year might think we’re struggling for new success stories
• #23: This is a sample quote slide
• #30: This is a sample quote slide
• #31: - Emphasize that this means we’re in their critical path, and that they trust us to be in their critical path
• #33: We just looked at several successful in-product experimenters, and the aggregate growth of product experimentation across our customer. I want to close with one more example, this time of a customer who’s just getting started. ATB Financial is an 80 year old Canadian bank that recently launched Brightside, a new business — and entirely digital bank. Brightside’s goal is to bring ATB into the modern world — providing a mobile banking service that fits the preferences and patterns of young customers. ATB uses Rollouts, the free version of Full Stack, with the following goal — to wrap every new feature in a feature flag, both a safeguard and to facilitate data-driven development. One customer that’s adopting Rollouts to help transform the way they ship product is ATB Financial. ATB is a Canadian financial institution that’s been around over 80 years and has over 5K employees. To respond to their customers needs and trends in the market, they decided to build a new type of bank. An all-digital, mobile-first bank and in order to roll out it safely they choose Optimizely Rollouts Established Canadian bank launching an all-digital new business.
• #34: And they’ve been successful. Kyla Workman, a Brightside Product Owner, said the following:
• #35: As I mentioned a moment ago, this is was achieved entirely with Rollouts — our free product. If you aren’t experimenting in product today, I would strongly encourage you to give Rollouts a try. It’s very powerful and valuable on its own, and, once you’re ready, the transition from Rollouts to Full Stack is seamless. If you or you company are new to this idea — or if you’re looking to ship faster with less risk, we’ve made getting started incredible easy via Optimizely rollouts. Rollouts is a free product build on the Full Stack platform — with the key features required for a business to get started. I do wanna mention Optimizely Rollouts. This is actually our newest product that we just launched just about a month ago and it's a free solution for doing feature flagging and rollouts. So everything we just talked about, about Canary Releases, silent launches, Feature Toggling. You can do yourself in your own application all for free. You don't need to pay for any new products, so I would really encourage you to check out optimizely.com/rollouts and see if this could be a fit for your team. Please encourage your developers to try this out. We are real believers in this way of working. That's why we've actually made this free and open source. We want anybody to be able to experience the power of Feature Flag in Rollouts. And we see that as a great on-ramp towards adopting these other pieces of experimentation. Optimizely Rollouts is available in 10 different languages, including all the major mobile frameworks as well as Java, JavaScript, C#, Python, PHP, Ruby, Node, you name it, all the major languages. You can start implementing features in Rollouts all for free and see if that's a good fit for your development team.
• #36: This is a sample quote slide
• #37: Alright, so – that’s my pitch — why we think product experimentation is important, what we’ve built to democratize the process, and a bit about how customers are using the product today. For the remainder of our session, I want to talk about the product. Specifically our roadmap — that things we’ve built, and the things we’re building.
• #38: Our roadmap is governed by three principles: developer ease-of-use, flexibility, and scalability. We want to make the product easy to adopt, from setup, to first flag, to first experiment The product must be flexible — both in terms of the technologies it supports, and the use cases it unlocks And finally, it needs to scale – both in terms of processes and technical requirements
• #39: It's worth mentioning that there are several different kinds of rollout you can do. One of the most common ones is to have something first enabled just in a local environment for a single developer And then roll it out to a staging environment.So whether that's a preproduction environment or a staging, you can actually use these rollouts to launch something in there. And then to go from staging out to maybe a small set of Beta users. So there's a different kind of rollout that's actually targeted to a certain audience or a certain whitelist of people that you've specifically decided should see your new stuff. We do this all the time. As a B2B company, we find friendly customers that wanna give new features a try, even some of our bigger companies might be more sensitive. We're not quite ready to try out a new feature until it's more fully baked. From there, once you've validated in the Beta, you can do a gradual rollout where you slowly ramp up traffic to a larger audience and roll back if anything goes wrong. And then and only then we can do that 100% rollout where everybody has a new feature. And the key idea here is that we can mix and match these techniques. It's not that one of these is the right answer. I think for each different business and each different product or feature within that, you wanna choose what the right rollout strategy is. But I would argue the right rollout strategy is almost never to just flip the switch. The gradual process is usually much more reliable.
• #40: Everything we’re building aligns with one of these principles. There’s a lot on this slide — I don’t expect you to memorize all of it — but I share it to convey how we’re thinking about the future of in-product experimentation Everything you see here is either live today or in progress. This will all be available within the next few months Let’s dig into a few examples
• #41: First, I want to talk about developer ease of use — making the process of getting up and running delightful
• #42: Recall the architecture slide we spoke about a few minutes ago. I want to highlight one specific piece of this diagram — the datafile, and the process by which it’s downloaded to your SDK Historically, we left the process of managing your datafile up to our customers. We did this for several reason — e.g. to give folks the flexibility to integrate datafile management with their particular networking constraints. But leaving this to customers was — excuse my French — a pain in the ass. It meant the developer who’s just getting started had to implement a simple feature before creating their first test or flag.
• #43: Here’s an example of a datafile manager — or, at least, part of a datafile manager. There are more than a 100 lines of code here — quite a bit of investment to just get started. This investment, however, is no longer necessary. Over the past few months, we’ve added automatic datafile management to every SDK…
• #44: …replacing the code on the left with a one-line initialization. Installing your SDK is now as simple as a copy and paste.
• #45: We validated automatic datafile management by conducting a usability study with twenty engineers. Among first group, who got started without automatic datafile management, only 1 of 10 were able to create and deploy a flag in <30min Among the second group, all ten succeeded. This, in my mind, was a huge win
• #47: Lucas Reis, Senior Softare Engineer at Compass, validated the feature, stating that installing the Java SDK with automatic datafile management was simple and smooth. And, this is just one example of several features we’ve added that make the implementing engineer’s life much easier. For example, we’ve also rewritten our event processor, which eliminates the need to implement a custom event dispatcher – a second hurdle that, like datafile management, new customers were historically required to clear. With changes like these, we make the process of getting up and running dramatically easier.
• #48: Next, let’s talk about governance and process. Early on, you may be experimenting with a small team; Everyone knows eachother, what’s happening when and why
• #49: But, as you scale, this becomes less and less true. Multiple teams, multiple parts of your stack, multiple locations. As this happens, you need governance — an area where Full Stack has historically fallen short.
• #50: Over the past three years, the most requested feature missing from our application is Change History – a record of the changes user’s make to their experiments and flags. I’m happy to report that this gap is nearly closed.
• #51: What we’re looking at now is a real screenshot of an in-progress solution — in this case, the Change History view of one of Optimizely’s favorite customers, Dunder-Mifflin Paper. Let’s consider an example. Imagine that I’m Michael Scott, and I’m curious about a recent experiment.
• #52: Specifically, an experiment that launched in early August. So, I filter my Change History to show only the relevant experiments.
• #53: And I find the test I’m looking for
• #54: I can then drill into the details of exactly what was changed. In this case, the experiment key and experiment description were updated, and the traffic allocation was changed — from 50/50 to 60/40 This same sort’ve detail will be available for every change — along with the user, timestamp, and other associated details. We’re also going to make this data exportable, so that you can integrate your change history with other systems, to provide a comprehensive audit log of your tests and flag.s
• #55: All of this is in progress now, and coming very soon. We expect to begin giving customers early access in late October.
• #56: We’re also making improvements that will help your scale your experimentation and feature management processes. For example, we’re now looking at the change history for a somewhat famous paper company, Dunder-Mifflin. And we want to view changes that occured over a three week period in August.
• #57: PUNCHLINE: Flexible
• #58: Many of you may remember Snapchat's rollout of their new app about a year ago where there was a total backlash on the Snapchat redesign. People hated the new user experience. They didn't like the new navigation. Kim Kardashian loudly announced she was moving to Instagram because of the better UI. And it really blew up in Snapchat's face and in the process drew a big drop in their share price as well because users were flocking away to a different application. There's another case where this experimentation mindset could have really helped avoid this problem. We could have actually caught this challenge much earlier and done much better usability testing or rolled out this new experience to a small segment of users to see how they'd react before blowing sort of the brand on this big launch to the entire customer base at once.
• #59: Earlier in this presentation — and in the keynote yesterday — we spoke about the concept of a rollout, i.e. the process of using a feature flag to control the release of a feature. Rollouts are a form of protection against bugs, outages, and other issues, like this example — in which Amazon accidentally made all of their products available for next to nothing. One of my favorite examples, this is one from a few years ago, Amazon had a glitch in its marketplace where anybody could buy product for a penny. Obviously it didn't last long or Amazon would have been cleaned out, but for a while there, their billing system was screwed up and every product just cost one cent. And so Amazon had a problem where people were going in and buying, you know, hundreds of X boxes for a penny each, so just paying a few dollars for it. And the worst part was they couldn't just cancel these orders because they'd already gone out to third party suppliers for all these things. And so it was a huge mess to untangle all based on a simple software bug that was rounding the wrong way. And we see these all over the place. Sometimes it's these kind of more concrete software errors, but sometimes it's more of a design challenge.
• #60: At Opticon two years ago we announced feature management, which allows customers to conduct rollouts — to gradually expose users to…
• #61: 5%
• #62: 25%
• #63: 50%
• #64: And eventually 100% of your audience. This kind of safeguard is tremendously powerful, and has become very popular across our customer base. But the example we’re looking at here is quite simple — just naively exposing percentages of our users to a feature over time. Oftentimes, you may want to release your feature using more specific rules — to specific users at specific times.
• #65: A famous example was described in this economist article a couple of years ago. Are there any Kiwis in the audience? If yes, I apologize, because we’re going to pick on you for a couple of minutes. The process described here involves launching your feature to a specific country first — in this case, New Zealand. You can think fo New Zealand as a stand-in for any customer group — beta users, or users with a specific subscription — the idea is that we’re isolating audiences during our feature release. And I'll just throw out one more use case that I think is fascinating and something that we can all take to heart for our own development which is the increasing use of New Zealand as a target for doing stage rollouts. So many companies like Facebook, Yahoo, and Microsoft have started taking new risky product launches and rolling them out to just people in New Zealand before they roll it out to other bigger countries. And they choose New Zealand because it's a pretty small country. It's pretty isolated from other countries where these companies operate. It speaks English. And so the market is somewhat similar to places like the United States or the UK. And a lot of people there are socially connected with technology. And so what these companies will do is, for example, Facebook will make a bold change to the newsfeed in just New Zealand. And they'll study that entire population to see how it works. And only if they're happy with New Zealand will they continue to roll out to say Australia and then the UK and then the United States. And that way, again, they've contained the damage of the blast radius of this thing they're doing. So I'd encourage all of you to think about what is your New Zealand? Maybe it is New Zealand, maybe it's Beta testers, maybe it's just a random slice of your traffic, but what's that smaller segment that we can actually try out these ideas on?
• #66: I’m excited to announce that we’re bringing this capability to Full Stack. In this example, we’re rolling our feature our to four different geos — NZ, USA, France, and Spain.
• #67: We begin with our poor friends in New Zealand, launching to 5% of users, then
• #68: 25%
• #69: 25%
• #70: And eventually 100%
• #71: And we only go to 100% once we've validated that rollout at smaller scale. The beauty of this is that if something does go wrong, and let's be honest, with many software launches, something does go wrong...
• #72: *** Clarify what feature variables and that they’re configurable remotely And we only go to 100% once we've validated that rollout at smaller scale. The beauty of this is that if something does go wrong, and let's be honest, with many software launches, something does go wrong...
• #73: And we only go to 100% once we've validated that rollout at smaller scale. The beauty of this is that if something does go wrong, and let's be honest, with many software launches, something does go wrong...
• #74: It's worth mentioning that there are several different kinds of rollout you can do. One of the most common ones is to have something first enabled just in a local environment for a single developer And then roll it out to a staging environment.So whether that's a preproduction environment or a staging, you can actually use these rollouts to launch something in there. And then to go from staging out to maybe a small set of Beta users. So there's a different kind of rollout that's actually targeted to a certain audience or a certain whitelist of people that you've specifically decided should see your new stuff. We do this all the time. As a B2B company, we find friendly customers that wanna give new features a try, even some of our bigger companies might be more sensitive. We're not quite ready to try out a new feature until it's more fully baked. From there, once you've validated in the Beta, you can do a gradual rollout where you slowly ramp up traffic to a larger audience and roll back if anything goes wrong. And then and only then we can do that 100% rollout where everybody has a new feature. And the key idea here is that we can mix and match these techniques. It's not that one of these is the right answer. I think for each different business and each different product or feature within that, you wanna choose what the right rollout strategy is. But I would argue the right rollout strategy is almost never to just flip the switch. The gradual process is usually much more reliable.
• #75: It's worth mentioning that there are several different kinds of rollout you can do. One of the most common ones is to have something first enabled just in a local environment for a single developer And then roll it out to a staging environment.So whether that's a preproduction environment or a staging, you can actually use these rollouts to launch something in there. And then to go from staging out to maybe a small set of Beta users. So there's a different kind of rollout that's actually targeted to a certain audience or a certain whitelist of people that you've specifically decided should see your new stuff. We do this all the time. As a B2B company, we find friendly customers that wanna give new features a try, even some of our bigger companies might be more sensitive. We're not quite ready to try out a new feature until it's more fully baked. From there, once you've validated in the Beta, you can do a gradual rollout where you slowly ramp up traffic to a larger audience and roll back if anything goes wrong. And then and only then we can do that 100% rollout where everybody has a new feature. And the key idea here is that we can mix and match these techniques. It's not that one of these is the right answer. I think for each different business and each different product or feature within that, you wanna choose what the right rollout strategy is. But I would argue the right rollout strategy is almost never to just flip the switch. The gradual process is usually much more reliable.
• #76: It's worth mentioning that there are several different kinds of rollout you can do. One of the most common ones is to have something first enabled just in a local environment for a single developer And then roll it out to a staging environment.So whether that's a preproduction environment or a staging, you can actually use these rollouts to launch something in there. And then to go from staging out to maybe a small set of Beta users. So there's a different kind of rollout that's actually targeted to a certain audience or a certain whitelist of people that you've specifically decided should see your new stuff. We do this all the time. As a B2B company, we find friendly customers that wanna give new features a try, even some of our bigger companies might be more sensitive. We're not quite ready to try out a new feature until it's more fully baked. From there, once you've validated in the Beta, you can do a gradual rollout where you slowly ramp up traffic to a larger audience and roll back if anything goes wrong. And then and only then we can do that 100% rollout where everybody has a new feature. And the key idea here is that we can mix and match these techniques. It's not that one of these is the right answer. I think for each different business and each different product or feature within that, you wanna choose what the right rollout strategy is. But I would argue the right rollout strategy is almost never to just flip the switch. The gradual process is usually much more reliable.
• #77: We’re supporting 10 years worth of different packages entirely w/ Full Stack Targeted Rollouts
• #78: We’re supporting 10 years worth of different packages entirely w/ Full Stack Targeted Rollouts
• #79: We’re supporting 10 years worth of different packages entirely w/ Full Stack Targeted Rollouts
• #80: We’re supporting 10 years worth of different packages entirely w/ Full Stack Targeted Rollouts
• #81: PUNCHLINE: Scalable
• #82: Build order: old SDKs, Swift, React, React Native, Go Swift — working in the iOS language of choice, modernizing 2nd most popular sdk React — most popular frontend language, again supporting patterns developers prefer React Native — customers have used the JS SDK in React Native apps, but, we just talked about ease of use, so we want to make the React Native developer’s life easy Go — finally, over the past 18 months, Go is (by far) the most requested new SDK; it’s in beta now with GA coming very soon
• #83: Finally, I want to talk about the feature that I’m most excited about — Full Stack as a Service — something that we believe will be transformative. Recall the companies we spoke about at the beginning Netflix, Facebook, Google aren’t just implementing an SDK in a single service They’re building massive-scale platforms that serve thousands of tests across thousands of users Our customers are starting to do the same thing.
• #84: Consider a hypothetical architecture that contains multiple services — an approach that is increasingly popular among companies today.
• #85: One possible approach to installing Full Stack in such an architecture is to embed an SDK in each service — in each application But, this approach has downsides It adds significant maintenance overhead Each team is installing and maintaining their own Optimizely instance It - don’t be negative about many SDKs — just explain reasons for different architectural
• #86: - Significantly less engineering investment than doing this out of the box - Reduce engineering effort, not no engineering effort
• #87: Now, you may be asking yourself — why am I looking at a slide full of LEGOs? Well, two reasons — First, for the non-technical folks in the room, the prior explanation may have been a bit inaccessible. A different way to think about it — and an analogy that I like — is to think about product experimentation as a collection of building blocks. Full Stack historically – and the solutions provided by our competitors — gave you a kit of tools that you could use to implement the platform of your choice. This works — often quite well — but it leaves the task of assembly to the user. This example solution is technical, and perhaps Be clear about the purpose of metaphor
• #88: Full Stack as a Service replaces that toolkit of disassembled LEGOs with with a much more complete solution — something that looks much more like the finished product. And our plan is for that finished product to be grand — like the Taj Mahal Full Stack Service will do everything your SDK does, and a whole lot more. By providing you with a service, rather than an SDK, we can implement sophisticated features that aren’t possible with a simple library. We’re tremendously excited about this — we have big plans.
• #89: - Significantly less engineering investment than doing this out of the box - Reduce engineering effort, not no engineering effort
• #90: Stubhub’s customer experience is driven by algorithms they develop in house For example, the recommended events you see when you visit Stubhub.com are created by the data science team. They take into account your location, prior purchase and browsing history on the site, time until the event, and many other factors to determine whether to show you sports, music, or other events, which teams, and which events. Every time they make a change to this algorithm, they are testing how it impacts their conversion rates.
• #92: One more customer example — I mentioned a moment ago that many customers have chosen to implement Full Stack as a service StubHub is one such customer. And they’ve leveraged that architecture to tremendous gain. StubHub uses Full Stack to experiment on everything Stubhub’s customer experience is driven by algorithms they develop in house For example, the recommended events you see when you visit Stubhub.com are created by the data science team. They take into account your location, prior purchase and browsing history on the site, time until the event, and many other factors to determine whether to show you sports, music, or other events, which teams, and which events. Every time they make a change to this algorithm, they are testing how it impacts their conversion rates.
• #93: This is a sample quote slide
• #94: Michael Alley, Sr. Product Manager at StubHub, has this to say:
• #99: PUNCHLINE: So, you want to get started?
• #100: We see Rollouts as a start of a journey and that's why we're making it free, so you can actually start shipping faster with less risk. I'd also encourage you to think about whether you're ready to drive that measurable product impact. So really being able to see the results of those rollouts and measure how different groups are performing. And if so, I'd encourage you to check out Optimizely's Full Stack experimentation as well.
• #101: - Be explicit that I’m not committing to deliver dates - should we remove this
• #102: - Be explicit that I’m not committing to deliver dates - should we remove this