2 6 dev ops - paas a-b testing
0 likes74 views
This document discusses A/B testing using Azure DevOps. It provides an agenda for a DevOps conference that includes a session on A/B testing. It then discusses the key aspects of A/B testing with Azure including using deployment slots to route traffic to different versions, using traffic manager to configure traffic percentages, and monitoring tests using App Insights and annotating releases. It also demonstrates configuring an A/B test routing rule using PowerShell.
2 6 dev ops - paas a-b testing
- 1. DEVOPS - A/B TESTING 8.12 OKKO OULASVIRTA, KOMPOZURE OY
- 2. AGENDA – DAY 2 • 9:00 Recap • 9:30 Azure SQL • 10:15 DBaaS + NoSQL • 11:00 Azure Governance • 11:30 Azure AI • 12:00 Lunch • 13:00 Azure AI • 14:00 Coffee • 14:30 DevOps - A/B testing • 15:30 Closing words
- 3. PERFECT DEFINITION OF DONE Live in production, collecting telemetry that examines the hypothesis which motivated the deployment.
- 4. BUILD-MEASURE-LEARN We believe {customer segment} wants {product/feature} because {value prop} Toproveordisprovetheabove,theteamwillconductthe followingexperiment(s):… Theaboveexperiment(s)provedordisprovedthehypothesisby impactingthefollowingmetric(s):…
- 5. APP SERVICE PLAN S1+ FOR A/B TESTING • Use deployment slots to route some percentage of the traffic to different url than production • Slot name is suffix to CNAME part of the website url: o app.domain.fi app-slotname.domain.fi o production site can be swapped with any of slots in matter of few seconds • Traffic manager is used to configure how much of the traffic goes to which slot – other than production
- 6. APPINSIGHTS IN VSTS • Free marketplace extension by Microsoft • No (yet) live monitoring and only subset of AppInsights offering • Shown as dashboard widgets • Use release annotations to track deployment to production • AppInsights application ID and Key required for configuration
- 7. DEMO Get-AzureRmSubscription -SubscriptionName "OKKO-MSDN-DEV" | Select-AzureRmSubscription; $r = New-Object Microsoft.WindowsAzure.Commands.Utilities.Websites.Services.WebEntities.RampUpRule; $r.ActionHostName = "oulasvirta-eshop-prd-ab.azurewebsites.net"; $r.ReroutePercentage = 10; $r.Name = "ab" $r.ChangeIntervalInMinutes = 10; $r.ChangeStep = 5; $r.MinReroutePercentage = 1; $r.MaxReroutePercentage = 30; Set-AzureWebsite "oulasvirta-eshop-prd" -Slot Production -RoutingRules $r;
- 8. A/B TEST CONFIGURATION • Use Azure PS (classic) to set RoutingRule Azure web site % of traffic on A/B testing (deployment slot) • Monitor… with App Insights and other tooling • Use Annotate Release task to mark entry of your A/B testing period • Use release task to Swap deployment slots