Overcoming the Challenges of Experimentation on a Service Oriented Architecture
0 likes261 views
Lupin Campos and Stephen Schwahn discuss the challenges of implementing experimentation in service-oriented architectures, specifically within the context of their registration process at Greenlight Financial Technology. They describe their approaches using Optimizely for A/B testing, introducing an experimentation ID service to maintain user identity consistency throughout various contexts. The document highlights their improvements in registration conversion rates and overall experimentation strategies across multiple services.
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
- 1. Overcoming the Challenges of Experimentation on a Service Oriented Architecture Lupin Campos & Stephen Schwahn September 2020
- 2. Hi! We are Lupin and Steph :) Lupin Campos Director of Product Growth Tech Startup Founder (Funnely) Software Development Company Founder (Leanbirds & Condiminds) Former Venture Partner @ 500Startups Steph Schwahn Growth Team Tech Lead Tech Startup Founder (Crescendo) Mobile App Development Company Founder (Plutonium Apps) Georgia Tech Alumni, graduated with high honors.
- 3. So far: - 1 complete backend refactor - 2 frontend refactors - 36 Registration flow experiments (web + fullstack) - 40% Improvement on Registration Conversion Rate
- 4. 1Let’s Get Into Context A quick note on who we are, what we do, and the challenges that we found in the way.
- 5. This is our registration Process... Phone Number Role Verification Code Email & Password Child Info Personal Info Sync Bank Acc (Website)
- 6. ….And here are the services behind it. A simplified version, you don’t want to see the entire thing. Trust us.
- 7. 1Service Oriented Architectures with Optimizely (the wrong way)
- 8. Greenlight Financial Technology, Inc. SOA with Optimizely Fullstack on Frontend Web App Marketing Site Mobile App Edge Service 1 Edge Service 2 Nested Service 1 Nested Service 2 Helper Service 1 Backend
- 9. Greenlight Financial Technology, Inc. SOA with Optimizely Fullstack on Frontend Web App Marketing Site Mobile App Edge Service 1 Edge Service 2 Nested Service 1 Nested Service 2 Helper Service 1 Backend Variant: Variant 1 Variable: FlowId = 6
- 10. Greenlight Financial Technology, Inc. SOA with Optimizely Fullstack Web App Marketing Site Mobile App Edge Service 1 Edge Service 2 Nested Service 1 Nested Service 2 Helper Service 1 Backend User_id: '1234_4321'
- 11. Greenlight Financial Technology, Inc. SOA with Optimizely Fullstack Web App Marketing Site Mobile App Edge Service 1 Edge Service 2 Nested Service 1 Nested Service 2 Helper Service 1 Backend User_id: '1234_4321' 30s 40s 400s 200s 5s
- 12. Greenlight Financial Technology, Inc. SOA with Optimizely Fullstack Web App Marketing Site Mobile App Edge Service 1 Edge Service 2 Nested Service 1 Nested Service 2 Helper Service 1 Backend User_id: '1234_4321' 30s 40s 400s 200s 5s
- 13. 1Service Oriented Architectures with Optimizely (our way)
- 14. Greenlight Financial Technology, Inc. Solution: SOA with Optimizely Agent Marketing Site Mobile App Edge Service 1 Edge Service 2 Nested Service 1 Nested Service 2 Helper Service 1 Backend Feature or Experiment Key: 'featureTest’ User_id: '1234_4321' Optimizely Agent 30s Web App
- 15. Greenlight Financial Technology, Inc. In Practice…. Client 1 Client 2 Client n . . . Ambassador (API Gateway) Pod (3-30 running based on traffic) Optimizely Service (Node.js) Optimizely Agentlocalhost:8080 Datafile sync (every 30s) localhost:3000 Exposes Optimizely Service on service port 80 Service grpc:80 pod-ip:3000 NodeJS Container Go Container (sidecar) Datafile pod-ip:3000pod-ip:3000
- 16. 1Let’s talk about “User ID” And how do we pass it everywhere!
- 17. Greenlight Financial Technology, Inc. Used for bucketing MurmurHash(“user_1234”) Rollout to 60%
- 18. Greenlight Financial Technology, Inc. Used for bucketing MurmurHash(“user_5678”) Rollout to 60%
- 19. Greenlight Financial Technology, Inc. What if we don’t have the user id? MurmurHash(“steph@greenlight.me”) ? Rollout to 60%
- 20. Greenlight Financial Technology, Inc. Introducing the Experimentation ID! MurmurHash(“exp_user_1234”) Rollout to 60%
- 21. 1Introducing Experimentation-ID Service How we keep a consistent user id regardless of context
- 22. Greenlight Financial Technology, Inc. Experimentation -ID service
- 23. Greenlight Financial Technology, Inc. Optimizely Service orchestrates
- 24. Greenlight Financial Technology, Inc. New API Contract! export type Context = UserIdContext | EmailContext | AnonymousIdContext | RegistrationIdContext | DeviceInfoContext export type EmailContext = { email: string } export type AnonymousIdContext = { anonymousId: string } export type DeviceInfoContext = { deviceOs: string, deviceToken: string } export type UserIdContext = { userId: string } export type RegistrationIdContext = { registrationId: string } export function isFeatureEnabled(featureKey: string, context: Context, attributes?: Attributes) export function getEnabledFeatures(context: Context, attributes?: Attributes) ...
- 25. 1Truly Full Stack Experimentation Through the ENTIRE user journey, from the marketing landing page, through registration services, and all the way to fundamental service primitives
- 27. Science of delivering it right