Serverless in production, an experience report (Going Serverless, 28 Feb 2018)
- 1. in production an experience report an experience report what you should know before you go to production ServerlessServerless
- 8. apr, 2016
- 11. 10PM traffic
- 13. low utilisation to leave room for spikes EC2 scaling is slow, so scale earlier
- 14. lots of $$$ for unused resources
- 15. up to 30 mins for deployment deployment required downtime
- 16. - Dan North “lead time to someone saying thank you is the only reputation metric that matters.”
- 18. “what would good look like for us?”
- 19. be small be fast have zero downtime have no lock-step DEPLOYMENTS SHOULD...
- 20. FEATURES SHOULD... be deployable independently be loosely-coupled
- 21. WE WANT TO... minimise cost for unused resources minimise ops effort reduce tech mess deliver visible improvements faster
- 22. nov, 2016
- 23. 170 Lambda functions in prod 1.2 GB deployment packages in prod 95% cost saving vs EC2 15x no. of prod releases per month
- 24. time is a good fit
- 25. 1st function in prod! time is a good fit
- 26. ? time is a good fit 1st function in prod!
- 28. Practices ToolsPrinciples what is good? how to make it good? with what?
- 30. 170 functions ? ? time is a good fit 1st function in prod!
- 33. rebuilt search
- 34. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearch
- 35. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda
- 37. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery
- 38. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery 1 developer, 2 days design production (his 1st serverless project)
- 39. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery “nothing ever got done this fast at Skype!” - Chris Twamley
- 40. - Dan North “lead time to someone saying thank you is the only reputation metric that matters.”
- 50. BigQuery
- 51. BigQuery
- 64. TESTING
- 66. 1.Unit do our objects do the right thing? are they easy to work with? Level of Testing
- 68. 1.Unit 2.Integration does our code work against code we can’t change? Level of Testing
- 69. handler
- 70. handler test by invoking the handler
- 71. Level of Testing 1.Unit 2.Integration 3.Acceptance does the whole system work?
- 73. “…We find that tests that mock external libraries often need to be complex to get the code into the right state for the functionality we need to exercise. The mess in such tests is telling us that the design isn’t right but, instead of fixing the problem by improving the code, we have to carry the extra complexity in both code and test…” Don’t Mock Types You Can’t Change
- 74. “…The second risk is that we have to be sure that the behaviour we stub or mock matches what the external library will actually do… Even if we get it right once, we have to make sure that the tests remain valid when we upgrade the libraries…” Don’t Mock Types You Can’t Change
- 75. Don’t Mock Types You Can’t Change Services
- 78. Unit Tests
- 80. what unit tests will not tell you… is Lambda proxy configured correctly? is the API resources configured correctly? is IAM policy set up correctly? is the request mapping set up correctly? is the table created? are we assuming the correct schema? is our request correct?
- 85. Paul Johnston
- 86. “…Wherever possible, an acceptance test should exercise the system end-to-end without directly calling its internal code. An end-to-end test interacts with the system only from the outside: through its interface…” Testing End-to-End
- 87. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda
- 88. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda Test Input
- 89. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda Validate Test Input
- 92. HOW observation
- 95. CI + CD PIPELINE
- 96. Yan
- 97. “…We prefer to have the end-to-end tests exercise both the system and the process by which it’s built and deployed… This sounds like a lot of effort (it is), but has to be done anyway repeatedly during the software’s lifetime…” Testing End-to-End
- 98. Yan
- 99. Jenkins build config deploys and tests unit + integration tests deploy acceptance tests
- 105. build.sh allows repeatable builds on both local & CI
- 107. Auto Auto Manual
- 109. LOGGING
- 111. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now?
- 112. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now? UTC Timestamp API Gateway Request Id your log message
- 113. Yan
- 114. CloudWatch Logs
- 115. CloudWatch Logs AWS Lambda ELK stack
- 116. …
- 117. CloudWatch Events
- 120. DISTRIBUTED TRACING
- 122. a user
- 123. where could the problem be?
- 124. correlation IDs* * eg. request-id, user-id, yubl-id, etc.
- 126. kinesis client http client sns client
- 133. • invocation Count • error Count • latency • throttling • granular to the minute • support custom metrics
- 134. • same metrics as CW • better dashboard • support custom metrics https://www.datadoghq.com/blog/monitoring-lambda-functions-datadog/
- 138. my code
- 139. my code
- 140. internet internet press button something happens my code
- 144. console.log(“hydrating yubls from db…”); console.log(“fetching user info from user-api”); console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”); console.log(“MONITORING|1489795335|8|count|yubls-served”); metric s logs timestamp metric value metric name metric type
- 145. CloudWatch Logs AWS Lambda ELK stack logs metrics CloudWatch
- 147. CONFIG MANAGEMENT
- 150. Environment variables make it hard to share configurations across functions. me
- 151. me Environment variables make it hard to implement fine-grained access to sensitive info.
- 156. SSM Parameter Store
- 159. SSM Parameter Store HTTPS role-based access encrypted in-flight
- 160. SSM Parameter Store encrypt role-based access
- 161. SSM Parameter Store encrypted at-rest
- 162. HTTPS SSM Parameter Store encrypted in-flight role-based access
- 166. That’s it, folks! Thank you all :-D
- 167. API Gateway and Kinesis Authentication & authorisation (IAM, Cognito) Testing Running & Debugging functions locally Log aggregation Monitoring & Alerting X-Ray Correlation IDs CI/CD Performance and Cost optimisation Error Handling Configuration management VPC Security Leading practices (API Gateway, Kinesis, Lambda) Canary deployments http://bit.ly/production-ready-serverless