Anatomy of a Feature Release
Download as PPTX, PDF0 likes1,287 views
The document outlines the process of integrating user testing, data-driven development, and A/B testing into the feature release lifecycle, emphasizing that each feature is important. It describes a structured approach to development that involves testing, analyzing KPIs, and iterative releases to optimize features. Key recommendations include setting realistic expectations, creating a structured testing process, involving the team in analysis, and ensuring robust tracking of KPIs and user interactions.
Anatomy of a Feature Release
- 1. Anatomy of a Feature Release How integrating User Testing, Data Driven Development & A/B Testing into your development lifecycle means you never have to give up on a new feature. (well, almost never)
- 2. Every feature counts Feature E Feature D Feature C Feature B Feature A
- 4. Now IDEA User test Design Prototype Release @ 5% Build Fix bugs Analyze QA No bugs * * Well, no relevant bugs KPIs up User test ROLL OUT! (slowly) Increase to 50/50 KPIs down Analyze discrepancies Release @ 50% Optimize A/B Test Test winner vs control
- 5. Design IDEA Left vs Right Design Form Open vs Closed User test Tabs vs Arrow Colors & Style
- 6. =
- 7. Prototype Prototype User test Umm. Yeah. We skipped this step. Big mistake.
- 8. Build & Release (Round 1) Build Release @ 5% Fix bugs Analyze QA KPIs down
- 9. Fix & Release (Round 2) Release @ 5% Fix bugs (again) Analyze QA (again) KPIs down
- 10. Tweak & Release (Round 3) No bugs* Deep Dive Analysis Release @ 50% Optimize A/B Test Winner Winner (against control) Is it this? Or this? * Well, no relevant bugs
- 11. So we optimized… These (along with a few others) VS. This
- 12. Winner Winner!
- 13. No one likes a Frankensite
- 14. Recap IDEA User test Design Prototype Release @ 5% Build Fix bugs Analyze QA No bugs * * Well, no relevant bugs KPIs up User test ROLL OUT! (slowly) Increase to 50/50 KPIs down Analyze discrepancies Release @ 50% Optimize A/B Test Test winner vs control
- 15. How to make data driven feature releases a reality Set expectations • A data driven release can take a long time! A decent estimation for a major new feature is ½ the time it took build. Create a process • How will you test? When will decisions be made? When will fixes & tweaks go out? Have the tools • Make sure your testing framework can support this. We use an internal system for major splits and Optimizely for front end tweaks / optimizations. Involve the team • Involve the whole team when analyzing results - sometimes the problem is obvious, sometimes it is buried. You need a multitude of viewpoints to make this work. Track EVERYTHING • Have tracking in place for all KPIs and relevant user interactions
- 16. Relax, you’ll make it happen @zorstag pretty much everywhere Even LinkedIn: www.linkedin.com/in/zorstag/ Model Credits: Sundance & Cassie
Editor's Notes
- #3: A wasted launch isn’t just about the time, money and resources that go into building the feature. Each feature is a step towards a bigger goal and vision, and while you can pivot and adjust if a feature fails, its definitely easier if they all work out.
- #4: By just giving up on a new feature you can’t learn why it failed and use that learning for the next feature. Was it not something the users wanted? Was it confusing? Why exactly did it fail?
- #5: Don’t worry, we’re going to break this down.
- #6: In the design phase, figure out what questions you can ask as early as possible. Who you should ask: - anyone you can find - Friends / family - any relevant users you can get your hands on - random people on the street or at the mall. Who you should not ask: - Stakeholders - Team membersRelevant tools or apps: - 5secondtest.com
- #7: Yes, crowdsourcing design can be scary, but it worked out well for us.
- #8: The idea here is simple, Build out a sample of the page with static data, using HTML 5 you can store sample data browser side. You can also use Square UI or Axure with high fidelity mocks to create basic user flows and interaction behaviors that can be tested. We NOW use remoteusertesting.com which allows you to create scripts to walk through prototypes and user flows at every stage. When you are doing prototype testing make sure to always test new work flows against existing workflows, where relevant.
- #9: We released to 5% of users and religiously measured out KPIs daily. We have a number of KPIs, but the sake of this use case we’ll focus on Email contacts to properties, which are pretty important to us and our clients. In our first round of testing, Email contacts were down by over 10%. We dialed back down and did some hard core QA. We deep dove on Hot Leads across all browsers and all major devices, smart phone, tablets, etc. We found a number of bugs, fixed them and went out again.
- #10: We were still down. We found a few more bugs with the email contacts, but nothing significant, so we poured through the error logs and discovered an inconsistent error that was only occurring with the new form. We didn’t think it was happening enough to cause the full drop, but we fixed it and tried again. Note: pouring through error logs is much easier if you have Splunk. Also, Make sure you are adding descriptive error logging from the beginning, to be able to troubleshoot more easily.
- #11: And we were still down, but less, only about 5% over legacy. We felt pretty confident there were not more bugs that were influencing this particular workflow. So, we deep dove into the tracking and analytics, in particular we looked at how often user’s viewed a property page and how often they clicked the Check Availability button and compared that to how often they opened the form and submitted it. The View / Open ratio was about 8% under the old site, where as the Open / Submit ratio was 10% over! So, we were definitely looking at an issue with getting users to open the form.
- #12: We used optimizely to test the front end tweaks to the new site. We made a list of all the changes we could make and started with the ones we thought would be most impactful.
- #13: And we ended up with a 12% lift! And after another 3 months and countless other optimizations across the site, we rolled out 100% and never looked back.
- #14: When you are optimizing specific areas of the site it is important to stop and assess how the changes you make are affecting other areas of the site. If you change a button color on one page, make sure you test or change similar buttons throughout the rest of the site to maintain consistency. If you are constantly testing and optimizing (as you should be) you should stop and assess key pages and workflows periodically to ensure you are maintaining a cohesive experience throughout.
- #15: Makes more sense now, right?
- #16: Sorry for all the text (I hate text), but the purpose of this slide is to Pin it up in your cube or maybe just leave it on Sr. Management’s desk, slide it under the door….mail it annonymously, etc. In tech you must keep moving forward to succeed. It is your responsibility as a Product Manager to make sure that every new feature your company invests time and resources in is a success. Be a champion and never give up (except when it makes sense to).