Amazon DevOps Guru for the Serverless Applications at AWS Community Day Benelux 2022
- 1. DevOps Guru for the Serverless applications Vadym Kazulkin, ip.labs, AWS Community Day NL, October 3 2022
- 2. Contact Vadym Kazulkin ip.labs GmbH Bonn, Germany Co-Organizer of the Java User Group Bonn v.kazulkin@gmail.com @VKazulkin https://www.linkedin.com/in/vadymkazulkin https://www.iplabs.de/
- 5. What is AWS DevOps Guru Amazon DevOps Guru is a service powered by machine learning (ML) that is designed to make it easy to improve an application’s operational performance and availability DevOps Guru helps detect behaviors that deviate from normal operating patterns so you can identify operational issues long before they impact your customers • increased latency • error rates (timeouts, throttles) • resource constraints (exceeding AWS account limits) https://aws.amazon.com/devops-guru
- 6. Benefits of DevOps Guru https://aws.amazon.com/devops-guru
- 7. How DevOps Guru work https://aws.amazon.com/devops-guru
- 8. Automated reasoning's scientific frontiers https://www.amazon.science/blog/automated-reasonings-scientific-frontiers
- 9. DevOps Guru Example Application https://github.com/Vadym79/DevOpsGuruWorkshopDemo inspired by https://github.com/aws-samples/serverless-java-frameworks-samples
- 10. DevOps Guru Live Demo of Your Choice
- 11. DevOps Guru Set Up
- 13. DevOps Guru Reactive Insights
- 14. • Warm up the application (takes between 1 and 24 hours) to create a base line • Design test experiment to provoke errors and latency increase • Reduce the service quote of the AWS service (API Gateway, Lambda, DynamoDB) • Set very low service quotas for the sake of reducing AWS costs • Add latency artificially • Stress test with Hey Tool to run into the operational issues • See if the DevOps Guru recognized the operational issues • Remediate the operational issues by increasing service quote, removing the artificial latency or stopping the stress test • See whether DevOps Guru closes the incident when it’s resolved DevOps Guru Examples | CONFIDENTIAL 14 https://github.com/rakyll/hey
- 15. DevOps Guru: Recognize operational issues in DynamoDB
- 16. DevOps Guru Examples: DynamoDB Throttling
- 17. DevOps Guru Examples: DynamoDB Throttling stress test and empty burst credits hey -q 20 -z 15m -c 20 -H "X-API-Key: XXXa6XXXX " https://XXX.execute-api.eu-central 1.amazonaws.com/prod/products/1
- 18. DevOps Guru Examples: DynamoDB Throttling
- 19. DevOps Guru Examples: DynamoDB Throttling
- 20. DevOps Guru Examples: DynamoDB Throttling
- 21. DevOps Guru Examples: DynamoDB Throttling
- 22. DevOps Guru Examples: DynamoDB Throttling
- 23. DevOps Guru: Recognize operational issues in API Gateway
- 24. DevOps Guru Examples: API Gateway HTTP 429 „too many requests“ Error
- 25. DevOps Guru Examples: API Gateway HTTP 404 „Not Found“ Error Query for not existing product id ,e.g. 200 hey -q 1 -z 15m -c 1 -H "X-API-Key: XXXa6XXXX" https://XXX.execute-api.eu-central- 1.amazonaws.com/prod/products/200
- 26. DevOps Guru Examples: API Gateway 4XX Error
- 27. DevOps Guru: Recognize operational issues in Lambda
- 28. DevOps Guru Examples: Lambda Throttling 1 hey -q 5 -z 15m -c 5 -H "X-API-Key: XXXa6XXXX" https://XXX.execute- api.eu-central-1.amazonaws.com/prod/products/2
- 29. DevOps Guru Examples: Lambda Throttling 1
- 30. DevOps Guru Examples: Lambda Throttling 2 hey -q 1 -z 15m -c 5000 -H "X-API-Key: XXXa6XXXX" https://XXX.execute-api.eu-central- 1.amazonaws.com/prod/products/2 or reach Lambda concurrency execution account limit (without reaching function concurrency limit) hey -q 1 -z 15m -c 950 https://AAA.execute-api.eu-central amazonaws.com/prod/func1 ... until hey -q 1 -z 15m -c 950 https://BBB.execute-api.eu-central amazonaws.com/prod/func10 hey -q 1 -z 15m -c 950 -H "X-API-Key: XXXa6XXXX" https://XXX.execute-api.eu-central- 1.amazonaws.com/prod/products/2
- 31. DevOps Guru Examples: Lambda Throttling 2
- 32. DevOps Guru Examples: Lambda Timeout Error Add 31 sec latency in the code of the Lambda function
- 33. DevOps Guru Examples: Lambda Timeout Error Java runtime requires 256 MB memory to start and execute this function
- 34. DevOps Guru Examples: Lambda Timeout Error
- 35. DevOps Guru Examples: Lambda Increased Latency Temporary add 28 sec latency in the code of the Lambda function
- 36. DevOps Guru Examples: Lambda Increased Latency
- 37. DevOps Guru: Recognize operational issues in SQS
- 38. DevOps Guru Examples: Operational Issues in SQS Temporary add 26 sec latency in the code of the Lambda function
- 39. DevOps Guru Examples: Operational Issues in SQS
- 40. DevOps Guru Proactive Insights
- 41. DevOps Guru Proactive Examples: Lambda timeout exceeds recommended SQS visibility
- 42. DevOps Guru Proactive Examples: Lambda timeout exceeds recommended SQS visibility
- 43. DevOps Guru Proactive Examples: SQS triggered Lambda does not have a DLQ
- 44. DevOps Guru Proactive Examples: Lambda function has concurrency spillover hey -q 1 -z 30m -c 9 -m DELETE -H "X-API-Key: XXXa6XXXX" -H "Content-Type: application/json;charset=utf-8" https://XXX.execute-api.eu-central-1.amazonaws.com/prod/products/11
- 45. • Table or Account Level read/write capacity for DynamoDB consumption reaching account limit • Triggered when the account consumed capacity is approaching table or account-level limits during a period of time Other operational issues and the proactive insights 1/2 | CONFIDENTIAL 45 https://aws.amazon.com/de/blogs/aws/automatically-detect-operational-issues-in-lambda-functions-with-amazon-devops-guru-for-serverless/
- 46. • DynamoDB table consumed capacity reaching AutoScaling Maximum parameter limit • Triggered when table consumed capacity is reaching AutoScaling Max parameters limit over a period. Other operational issues and the proactive insights 2/2 | CONFIDENTIAL 46 https://aws.amazon.com/de/blogs/aws/automatically-detect-operational-issues-in-lambda-functions-with-amazon-devops-guru-for-serverless/
- 47. DevOps Guru integration in incident management tools https://aws.amazon.com/devops-guru • OPsCenter • PagerDuty • Atlassian Opsgenie
- 48. DevOps Guru Integration Settings
- 49. DevOps Guru Examples: DynamoDB Throttling
- 50. DevOps Guru Integration with OpsCenter
- 51. DevOps Guru Integration with OpsCenter
- 52. DevOps Guru Integration with OpsCenter OpsItem View
- 53. DevOps Guru Integration with OpCenter Incident Manager View
- 54. DevOps Guru Integration with OpCenter Incident Manager View
- 55. DevOps Guru Integration with OpCenter Incident Confugration
- 56. DevOps Guru Integration with OpCenter Incident Contacts
- 57. DevOps Guru Integration with PagerDuty https://www.pagerduty.com/docs/guides/amazon-devops-guru-integration-guide/
- 58. DevOps Guru Integration with PagerDuty Enter „Integration URL“ generated by PagerDuty
- 59. DevOps Guru Integration with PagerDuty
- 60. DevOps Guru PagerDuty Incidents
- 61. DevOps Guru Supported Services https://aws.amazon.com/de/devops-guru/pricing/
- 62. DevOps Guru Cost Estimator https://aws.amazon.com/de/devops-guru/pricing/
- 63. DevOps Guru Conclusions, Obeservations, Suggestions 1/4 • All errors have been correctly recognized so far • It took several (at least 7) minutes to create an incident after anomaly appeared • Correctly no insights created for the temporary incidents • Short time Lambda, DynamoDB and API Gateway Throttling • Recommendations for the insight reason could be more precise • No differentiation between Lambda throttling because of reaching individual function concurrency limit or the total AWS account concurrency limit
- 64. DevOps Guru Conclusions, Obeservations, Suggestions 2/4 • HTTP 4XX Errors • Time to create DevOps Guru insight is relatively big (more than 10 minutes). Maybe because of the medium severity • Are not fine granular splitted between different error codes (404, 429) which have totally different meaning and cause • No reference which Lambda function behind the API Gateway cause these errors (important for 404 error)
- 65. DevOps Guru Conclusions, Obeservations, Suggestions 3/4 • Lambda duration anomalous insights (Duration p90) • took huge time to create (sometimes more than 30 minutes). Maybe because of the medium severity • DevOps Guru Proactive Insights • Stay ongoing long time after the insight occurred only once • Not always expire quickly after being fixed • Missed some important ones, like not used Lambda Provisioned Concurrency for a long period of time
- 66. DevOps Guru Conclusions, Obeservations, Suggestions 4/4 • Log Groups haven’t always been displayed within DevOps Guru Insight • Missing Link to Tracing ( e.g. AWS X-Ray) • Easier integration into AWS Systems Manager Incident Manager
- 67. DevOps Guru for RDS https://aws.amazon.com/devops-guru/features/devops-guru-for-rds/ https://aws.amazon.com/de/blogs/devops/leverage-devops-guru-for-rds-to-detect-anomalies-and-resolve-operational-issues/
- 69. www.iplabs.de Accelerate Your Photo Business Get in Touch