re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdf
0 likes13 views
The document discusses the success and lessons learned from AWS Lambda Powertools, which reached 10 million downloads, highlighting community growth and customer experiences. It emphasizes the importance of balancing operational excellence with community needs while adopting best practices. Key takeaways include the need for empathy in practice, ongoing refinement, and collaboration within the community.
![© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Structured logging: Without Powertools
Raw
[INFO] 2022-12-02T22:.. 1c8df7d3… Hello world](https://image.slidesharecdn.com/opn306-awslambdapowertoolslessons10mdownloads-230607080013-3d4cb5c5/85/re-Invent-OPN306-AWS-Lambda-Powertools-Lessons-10M-downloads-pdf-82-320.jpg)
![© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Structured logging: Without Powertools
[INFO] 2022-12-02T22:.. … {“message”: “Hello world”}
Semi
structured](https://image.slidesharecdn.com/opn306-awslambdapowertoolslessons10mdownloads-230607080013-3d4cb5c5/85/re-Invent-OPN306-AWS-Lambda-Powertools-Lessons-10M-downloads-pdf-83-320.jpg)
re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdf
- 1. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Lambda Powertools: Lessons from the road to 10 million downloads O P N 3 0 6 Heitor Lessa (he/him) Principal Solutions Architect, Developer Acceleration (DevAx) AWS
- 3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Serverless Lens in AWS WA Tool
- 4. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Serverless Lens in AWS WA Tool Most common risks
- 5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Findings come in late Plan/design Build Ship Operate
- 6. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Findings come in late Plan/design Build Ship Operate Readiness assessment
- 7. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Findings come in late Plan/design Build Ship Operate Readiness assessment
- 8. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Findings come in late Plan/design Build Ship Operate Today, let’s fix here
- 9. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Lambda Powertools Logger Metrics Tracer + Many more A toolkit to ease leading practices adoption
- 10. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Lambda Powertools OPS1 solution OPS 1: How do you evaluate your Serverless application’s health?
- 11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Lambda Powertools: Lessons from the road to 10 million downloads O P N 3 0 6 Heitor Lessa (he/him) Principal Solutions Architect, Developer Acceleration (DevAx) AWS
- 12. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Today Lessons Up next
- 13. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Today
- 14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Today: Serverless Lens mapping (51%) Question Practice SEC 1 Scope access based on identity’s metadata SEC 2 Design smaller, single-purpose functions SEC 2 Use temporary credentials between resources and components SEC 3 Store secrets that are used in your code securely REL 1 Use mechanisms to protect nonscalable resources REL 2 Manage transaction, partial, and intermittent failures REL 2 Manage duplicate and unwanted events REL 2 Orchestrate long-running transactions OPS 1 Use distributed tracing, and code is instrumented with additional context OPS 1 Use structured and centralized logging OPS 1 Use application, business, and operations metrics OPS 2 Use infrastructure as code (IaC) and stages isolated in separate environments OPS 2 Use configuration management PERF 1 Measure and optimize function startup time PERF 1 Take advantage of concurrency via async and stream-based function invocations PERF 1 Optimize access patterns and apply caching where applicable COST 1 Minimize external calls and function code initialization COST 1 Optimize logging output and its retention COST 1 Use cost-aware usage patterns in code Total 19 Powertools assistance
- 15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Today: Scale in numbers 6.5 billion 161% invocations MoM growth YoY
- 16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Today: How did we get here? Our customers grew Observability
- 17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Today: How did we get here? Batch partial failure REST/GraphQL API Input/Output validation Config management Handling secrets Idempotency Observability BYO middleware Self-documented schemas Feature flags Data extraction Caching Our customers grew The community grew too ~40% came from the community Streaming
- 18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Today: How did we get here? Our customers grew The community grew too Data engineers Developers Platform eng. Security eng. APIs, event-driven architecture ETL, data validation Automation, abstractions AppSec, guardrails
- 19. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons: I am because we are
- 20. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Release notes Challenge
- 21. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Release notes Lesson
- 22. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Release notes Lesson
- 23. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Release notes Lesson
- 24. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Ambiguity Language idioms? Breaking changes? Can I run on Fargate too? What about X dependency? When to cut your losses? Best vs. emerging vs. leading practice When too many opinions are harmful Challenge
- 25. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Ambiguity Lesson Tenets AWS Lambda only Eases the adoption of best practices Keep it lean We strive for backwards compatibility We work backwards from the community Progressive
- 26. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Ambiguity Lesson Thanks to our OpenSearch friends
- 27. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Ambiguity Lesson
- 28. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Ambiguity Lesson
- 29. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Contribution Bugfix? Sample apps? New features? What about docs? hacktoberfest? What is a contribution after all? Challenge Case study? Automation improvements? Discussion?
- 30. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Contribution Lesson
- 31. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Contribution Lesson pre-commit make pr + Shortened feedback loop and consistency at scale
- 32. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Contribution lesson Expose conventions and definitions early in the process
- 33. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Tech writing Challenge Like coding, it’s another competency . . . that can be learned
- 34. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Tech writing Lesson Co-authorship & editing mechanisms Line editing, copywriting, 1:1, etc.
- 35. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Tech writing Lesson Strategic Writing for UX by Torrey Podmajersky Voice chart, style guide, diverse audience, etc.
- 36. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge Docs growth Gatsby Apollo theme💔
- 37. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge Docs growth mkdocs-material ❤
- 38. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge From a few to 300+ Examples growth
- 39. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Gradual lint and realistic uplift Lesson Examples growth
- 40. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge Consistency Table of contents inconsistency
- 41. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Lesson Consistency Gradual and sustainable consistency
- 42. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Balancing act Challenge “Why don’t you just”
- 43. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge Open source Work Personal life Balancing act
- 44. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge New features Operational excellence Documentation Community Balancing act
- 45. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge New features Operational excellence Documentation Community Balancing act
- 46. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Challenge New features Operational excellence Documentation Community Help! Balancing act
- 47. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Lesson Balancing act Working in Public The Making and Maintenance of Open Source Software by Nadia Eghbal
- 48. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Lesson The Hard Parts of Open Source “Everything is done in haste, at fever pitch.” Balancing act
- 49. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Lesson Ops excellence Roadmap Contribution guidelines Balancing act
- 50. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Lesson Balancing act
- 51. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Lesson Balancing act
- 52. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Transparency Challenge
- 53. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Transparency Lesson
- 54. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Roadmap and daily operations revamped Transparency Lesson CloudWatch Feature request: Data Class for CloudWatch Logs passed through Kinesis Data Streams #1488
- 55. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Migration to Discord led to a healthier community Transparency Lesson
- 56. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Release overhead Challenge Source Build Tests PyPi Docs SAR GitHub PyPi pipeline Security Complexity Perf Test Prod API User guide Beta Prod Extras
- 57. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Release overhead Challenge Source Build Beta Wave N Beta Canary Prod Wave N Prod Canary Test Test Lambda layer pipeline for 16 Regions Internal pipeline was disconnected from GitHub processes
- 58. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Release overhead Challenge Source Build Beta Wave N Beta Canary Prod Wave N Prod Canary Lambda Layer pipeline for 16 regions GitHub/Internal sync Trigger internal pipeline Source Build N tests PyPi Docs GitHub actions – PyPi pipeline SAR ~3 hours 5 minutes 50 Lambda layer deployments per release (stage + canary + SAR)
- 59. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Lesson Release overhead 178 Lambda Layer deployments per release (stage + canary + SAR)
- 60. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons End-to-end test Challenge Speed? Step-through debugging? Parallelization? Isolated test runs? Entry bar for infrastructure as code? Frequency? Cost? Run a single test/infra?
- 61. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons End-to-end test Lesson
- 62. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons End-to-end test Lesson
- 63. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons End-to-end test Lesson
- 64. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons End-to-end test Lesson Data fetchers are not customer-facing (yet!)
- 65. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons End-to-end test Lesson Parallelization workflow
- 66. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Impact Stars? # of case studies? How can we track unobtrusively? Downloads? Forks? Challenge # of language bindings?
- 67. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Impact Lesson Community Lambda layer Belonging Unobtrusive tracking ~40% of Powertools features Customer and Partner’s feedback
- 68. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Impact Lesson
- 69. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Up next: What the near future holds
- 70. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Up next Feature parity Java Python TypeScript .NET
- 71. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Up next Python Java TypeScript .NET Tracer x x x x Logger x x x x Metrics x x x x Event handler (REST) x Event handler (GraphQL) x Idempotency x x Parameters x x Batch processing x x Validation x x Event source data classes x x x Parser x Feature flags x Streaming x JMESPath (data extraction) x x Amazon SQS large message handling x Feature parity
- 72. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon CloudWatch X-Ray Amazon CloudWatch and AWS X-Ray Up next Observability providers
- 73. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Up next Event source JSON schema
- 74. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Up next: Long-term Lambda Powertools Testing Beyond Lambda AWS Fargate, AWS Glue, etc. Event Builder, data fetcher (E2E test) /You tell us/ Making your experience smoother
- 75. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary
- 76. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary Key lessons learned Customers grew So did the community Balancing act Ops excellence is business as usual Empathy Practice and refine what we preach
- 77. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Resources All links here
- 78. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! Please complete the session survey in the mobile app Heitor Lessa (he/him) @heitor_lessa
- 79. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Appendix
- 80. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging
- 81. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Recap Raw Semi-structured Canonical Structured
- 82. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Without Powertools Raw [INFO] 2022-12-02T22:.. 1c8df7d3… Hello world
- 83. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Without Powertools [INFO] 2022-12-02T22:.. … {“message”: “Hello world”} Semi structured
- 84. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Without Powertools Canonical at=INFO msg=Aha! request_latency=0.1
- 85. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Without Powertools Structured
- 86. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Without Powertools Structured
- 87. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Powertools Structured
- 88. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Powertools Structured
- 89. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Powertools Structured
- 90. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Structured logging: Powertools Structured
- 91. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing
- 92. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. API Gateway Lambda Fargate DynamoDB API Gateway S3 bucket Distributed tracing: Recap
- 93. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Trace API Gateway Lambda Fargate DynamoDB API Gateway S3 bucket Distributed tracing: Recap Legend
- 94. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Trace API Gateway Lambda Fargate DynamoDB API Gateway S3 bucket Distributed tracing: Recap Legend Segment
- 95. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Trace API Gateway Lambda Fargate DynamoDB API Gateway S3 bucket Distributed tracing: Recap Legend Segment Sub-segment
- 96. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Trace API Gateway Lambda Fargate DynamoDB API Gateway S3 bucket Distributed tracing: Recap Legend Segment Sub-segment Type Key Value Namespace Annotation UserId a1c8k... Metadata Cart {….} Checkout Trace metadata
- 97. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing: Without Powertools Tracing
- 98. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing: Without Powertools Tracing
- 99. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing: Without Powertools Tracing
- 100. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing: Without Powertools Tracing
- 101. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing: Without Powertools Tracing
- 102. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing: Without Powertools Tracing
- 103. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed tracing: Powertools Tracing
- 104. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics
- 105. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics: Recap Namespace Metric Unit Dimensions Metrics
- 106. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics: Without Powertools Metrics
- 107. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics: Powertools Metrics
- 108. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics: Powertools Metrics
- 109. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics: Powertools Metrics
- 110. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda Powertools: Humble beginnings Recap