The present and future of Serverless observability
5 likes5,090 views
The document discusses the concept of observability in distributed systems, contrasting it with traditional monitoring, and emphasizing the importance of tracking individual user experiences and events. It outlines the four pillars of observability: monitoring, alerting/visualization, distributed tracing infrastructure, and log aggregation/analytics, while describing new challenges such as the inability to install agents or manage background processing in serverless architectures. The narrative reflects on how effective observability tools and practices can enhance understanding and debugging of complex systems.
![user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-127-320.jpg)
![user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-128-320.jpg)
![user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
create-auth0-user](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-130-320.jpg)
![user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
click here to go to code create-auth0-user](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-131-320.jpg)
![Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
create-auth0-user](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-135-320.jpg)
![Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
!](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-136-320.jpg)
![!
Function [X] just performed
an unexpected write against
DynamoDB table [Y].
Should I…
ignore it from now on
shut it down!!](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-163-320.jpg)
![Function [X]’s performance
has degraded since yesterday -
99% latency has gone up by
47% from 100ms to 147ms.
!](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-167-320.jpg)
![!
Function [X] can run faster &
cheaper if you increase its
memory allocation.
Should I…
ignore it from now on
update setting](https://image.slidesharecdn.com/thepresentandfutureofserverlessobservability-180215104813/85/The-present-and-future-of-Serverless-observability-168-320.jpg)
The present and future of Serverless observability
- 2. hi, my name is Yan.
- 3. hi, my name is Yan.
- 4. hi, my name is Yan. AWS user since 2009
- 10. mm… I wonder what’s going on here…
- 11. what is observability? how is it different from monitoring?
- 12. Monitoring watching out for known failure modes in the system, e.g. network I/O, CPU, memory usage, …
- 13. Observability being able to debug the system, and gain insights into the system’s behaviour
- 14. However, I would argue that the health of the system no longer matters. We've entered an era where what matters is the health of each individual event, or each individual user's experience, or each shopping cart's experience (or other high cardinality dimensions). With distributed systems you don't care about the health of the system, you care about the health of the event or the slice. ”http://bit.ly/2E2QngU- Charity Majors “
- 15. However, I would argue that the health of the system no longer matters. We've entered an era where what matters is the health of each individual event, or each individual user's experience, or each shopping cart's experience (or other high cardinality dimensions). With distributed systems you don't care about the health of the system, you care about the health of the event or the slice. ”http://bit.ly/2E2QngU- Charity Majors “
- 16. These are the four pillars of the Observability Engineering team’s charter: • Monitoring • Alerting/visualization • Distributed systems tracing infrastructure • Log aggregation/analytics “ ” http://bit.ly/2DnjyuW- Observability Engineering at Twitter
- 17. Observability is useful even outside of incidents and outages
- 18. microservices death stars circa 2015
- 23. microservices death stars circa 2015 I got this!
- 25. new challenges
- 26. new challenges
- 27. NO ACCESS to underlying OS
- 29. •nowhere to install agents/daemons new challenges
- 30. user request user request user request user request user request user request user request critical paths: minimise user-facing latency handler handler handler handler handler handler handler
- 31. user request user request user request user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead
- 32. user request user request user request user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead NO background processing except what platform provides
- 33. •no background processing •nowhere to install agents/daemons new challenges
- 34. EC2 concurrency used to be handled by your code
- 35. EC2 Lambda Lambda Lambda Lambda Lambda now, it’s handled by the AWS Lambda platform
- 36. EC2 logs & metrics used to be batched here
- 37. EC2 Lambda Lambda Lambda Lambda Lambda now, they are batched in each concurrent execution, at best…
- 40. HIGHER concurrency to log aggregation/telemetry system
- 41. •higher concurrency to telemetry system •nowhere to install agents/daemons •no background processing new challenges
- 43. Lambda data is batched between invocations
- 44. Lambda idle data is batched between invocations
- 45. Lambda idle garbage collectiondata is batched between invocations
- 46. Lambda idle garbage collectiondata is batched between invocations HIGH chance of data loss
- 47. •high chance of data loss (if batching) •nowhere to install agents/daemons •no background processing •higher concurrency to telemetry system new challenges
- 48. Lambda
- 51. my code send metrics internet internet press button something happens
- 54. ? functions are often chained together via asynchronous invocations
- 56. ? SNS Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 SES tracing ASYNCHRONOUS invocations through so many different event sources is difficult
- 57. •asynchronous invocations •nowhere to install agents/daemons •no background processing •higher concurrency to telemetry system •high chance of data loss (if batching) new challenges
- 58. the Present
- 59. These are the four pillars of the Observability Engineering team’s charter: • Monitoring • Alerting/visualization • Distributed systems tracing infrastructure • Log aggregation/analytics “ ” http://bit.ly/2DnjyuW- Observability Engineering at Twitter
- 62. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now?
- 63. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now? UTC Timestamp Request Id your log message
- 65. one log group per function one log stream for each concurrent invocation
- 66. logs are not easily searchable in CloudWatch Logs me
- 67. CloudWatch Logs
- 68. CloudWatch Logs AWS Lambda ELK stack
- 69. …
- 70. CloudWatch Logs
- 74. •no background processing •nowhere to install agents/daemons new challenges
- 75. my code send metrics internet internet press button something happens
- 76. those extra 10-20ms for sending custom metrics would compound when you have microservices and multiple APIs are called within one slice of user event
- 77. Amazon found every 100ms of latency cost them 1% in sales. http://bit.ly/2EXPfbA
- 78. console.log(“hydrating yubls from db…”); console.log(“fetching user info from user-api”); console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”); console.log(“MONITORING|1489795335|8|count|yubls-served”); timestamp metric value metric type metric namemetrics logs
- 79. CloudWatch Logs AWS Lambda ELK stack logs m etrics CloudWatch
- 82. API Gateway send custom metrics asynchronously
- 83. SNS KinesisS3API Gateway … send custom metrics asynchronously send custom metrics as part of function invocation
- 85. X-Ray
- 88. do not span over API Gateway
- 89. narrow focus on a function good for homing in on performance issues for a particular function, but offers little to help you build intuition about how your system operates as a whole.
- 90. However, I would argue that the health of the system no longer matters. We've entered an era where what matters is the health of each individual event, or each individual user's experience, or each shopping cart's experience (or other high cardinality dimensions). With distributed systems you don't care about the health of the system, you care about the health of the event or the slice. ”http://bit.ly/2E2QngU- Charity Majors “
- 91. follow the data
- 93. don’t span over async invocations good for identifying dependencies of a function, but not good enough for tracing the entire call chain as user request/data flows through the system via async event sources.
- 94. don’t span over non-AWS services
- 96. static view
- 97. our tools need to do more to help us with understanding & debugging our distributed system, not just what happens inside one function
- 99. “one user action/vertical slice through the system”
- 100. microservices death stars circa 2015
- 101. microservices death stars circa 2015 HELP…
- 103. WARNING: this is part fiction, part inspired by new tools
- 104. DASHBOARDS
- 108. different dimensions of X splattered across the screen
- 111. + cold starts + throttled invocations + concurrent executions + estimated cost ($)
- 112. SubscriberGetAccount 200,545 0 19 94 0 0 % 0 % Est Cost: Req Rate: $54.0/s 20,056.0/s Concurrency Median Mean 99.5th 99th 90th370 1ms 4ms 61ms 44ms 10ms circle colour and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic no. of concurrent executions of this function Request rate Estimated cost Error percentage of last 10 seconds Cold start percentage last 10 seconds last minute latency percentiles 200,545 0 19 94 0 Rolling 10 second counters with 1 second granularity Successes Cold starts Timeouts Throttled Invocations Errors
- 113. SubscriberGetAccount 200,545 0 19 94 0 0 % 0 % Est Cost: Req Rate: $54.0/s 20,056.0/s Concurrency Median Mean 99.5th 99th 90th370 1ms 4ms 61ms 44ms 10ms circle colour and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic no. of concurrent executions of this function Request rate Estimated cost Error percentage of last 10 seconds Cold start percentage last 10 seconds last minute latency percentiles 200,545 0 19 94 0 Rolling 10 second counters with 1 second granularity Successes Cold starts Timeouts Throttled Invocations Errors
- 114. SubscriberGetAccount 200,545 0 19 94 0 0 % 0 % Est Cost: Req Rate: $54.0/s 20,056.0/s Concurrency Median Mean 99.5th 99th 90th370 1ms 4ms 61ms 44ms 10ms circle colour and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic no. of concurrent executions of this function Request rate Estimated cost Error percentage of last 10 seconds Cold start percentage last 10 seconds last minute latency percentiles 200,545 0 19 94 0 Rolling 10 second counters with 1 second granularity Successes Cold starts Timeouts Throttled Invocations Errors
- 115. SubscriberGetAccount 200,545 0 19 94 0 0 % 0 % Est Cost: Req Rate: $54.0/s 20,056.0/s Concurrency Median Mean 99.5th 99th 90th370 1ms 4ms 61ms 44ms 10ms circle colour and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic no. of concurrent executions of this function Request rate Estimated cost Error percentage of last 10 seconds Cold start percentage last 10 seconds last minute latency percentiles 200,545 0 19 94 0 Rolling 10 second counters with 1 second granularity Successes Cold starts Timeouts Throttled Invocations Errors
- 116. SubscriberGetAccount 200,545 0 19 94 0 0 % 0 % Est Cost: Req Rate: $54.0/s 20,056.0/s Concurrency Median Mean 99.5th 99th 90th370 1ms 4ms 61ms 44ms 10ms circle colour and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic no. of concurrent executions of this function Request rate Estimated cost Error percentage of last 10 seconds Cold start percentage last 10 seconds last minute latency percentiles 200,545 0 19 94 0 Rolling 10 second counters with 1 second granularity Successes Cold starts Timeouts Throttled Invocations Errors
- 122. birds-eye view of our system as it lives and breathes
- 125. user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API trace async invocations create-auth0-user
- 126. user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API trace non-AWS resources create-auth0-user
- 127. user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API Logs timestamp component message POST /user2018/01/25 20:51:23.188 2018/01/25 20:51:23.201 create-user 2018/01/25 20:51:23.215 create-user 2018/01/25 20:51:23.521 tag-user incoming request… saving user [theburningmonk] in the [user] table… saved user [theburningmonk] in the [user] table level debug debug debug debug tagging user [theburningmonk] with Azure Face API… create-auth0-user
- 128. user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API Logs timestamp component message POST /user2018/01/25 20:51:23.188 2018/01/25 20:51:23.201 create-user 2018/01/25 20:51:23.215 create-user 2018/01/25 20:51:23.521 tag-user incoming request… saving user [theburningmonk] in the [user] table… saved user [theburningmonk] in the [user] table level debug debug debug debug tagging user [theburningmonk] with Azure Face API… create-auth0-user
- 129. user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API Logs timestamp component message POST /user2018/01/25 20:51:23.188 incoming request… level debug request-id start-time 0ae4ba5d-dab1-4f9e-9de7-eace27ebfbc2 2018/01/25 20:51:23.188 method POST create-auth0-user
- 130. user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API Logs timestamp component message 2018/01/25 20:51:23.201 create-user 2018/01/25 20:51:23.215 create-user 2018/01/25 20:51:23.585 saving user [theburningmonk] in the [user] table… saved user [theburningmonk] in the [user] table level debug debug debug uploading profile image… create-user debug tagged user [theburningmonk] with Azure Face API… create-user2018/01/25 20:51:23.587 create-auth0-user
- 131. user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API Logs timestamp component message 2018/01/25 20:51:23.201 create-user 2018/01/25 20:51:23.215 create-user 2018/01/25 20:51:23.585 saving user [theburningmonk] in the [user] table… saved user [theburningmonk] in the [user] table level debug debug debug uploading profile image… create-user debug tagged user [theburningmonk] with Azure Face API… create-user2018/01/25 20:51:23.587 click here to go to code create-auth0-user
- 133. Logs Input/Output user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API input output { "body": "{ "username":"theburningmonk"}", "resource": "/user", "requestContext": { "resourceId": "123456", "apiId": “1234567890", "resourcePath": "/user", { "statusCode": 200 } create-auth0-user
- 134. Logs Input/Output user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API input output { "Records": [ { "Sns": { "Type": "Notification", "MessageId": "…", "TopicArn": "…", "Message": "…", "Timestamp": "2018/01/25 20:51:24.215", { "error": null, "result": "OK" } create-auth0-user
- 135. Logs Input/Output user profile-images POST /user process-images resize-images image-tasks Auth0 create-user reformat-imagestag-user Face API input error { "Records": [ { "Sns": { "Type": "Notification", "MessageId": "…", "TopicArn": "…", "Message": "…", "Timestamp": "2018/01/25 20:51:24.215", [com.spaceape.dragon.handler.ReformatProfileImageHandle r] Null reference exception *java.lang.NullReferenceException: … * at … * at … * at … create-auth0-user
- 136. Logs Input/Output user profile-images POST /user process-images resize-images image-tasks Auth0 create-user create-auth0-user reformat-imagestag-user Face API input error { "Records": [ { "Sns": { "Type": "Notification", "MessageId": "…", "TopicArn": "…", "Message": "…", "Timestamp": "2018/01/25 20:51:24.215", [com.spaceape.dragon.handler.ReformatProfileImageHandle r] Null reference exception *java.lang.NullReferenceException: … * at … * at … * at … !
- 138. All 0 200 400 600 800 create-user …user.insert_user …user.upload_img tag-user create-auto0-user process-images resize-images reformat-images! 837ms 406ms 66ms 114ms 122ms 82ms 240ms 157ms 35ms
- 139. All 0 200 400 600 800 create-user …user.insert_user …user.upload_img tag-user create-auto0-user process-images resize-images reformat-images! 837ms 406ms 66ms 114ms 122ms 82ms 240ms 157ms 35ms
- 140. Input/Output user profile-images POST /user process-images resize-images image-tasks Auth0 create-user create-auth0-user reformat-imagestag-user Face API Logs ! All 0 200 400 600 800 create-user …user.insert_user …user.upload_img tag-user create-auto0-user process-images resize-images reformat-images! 837ms 406ms 66ms 114ms 122ms 82ms 240ms 157ms 35ms
- 141. Input/Output user profile-images POST /user process-images resize-images image-tasks Auth0 create-user create-auth0-user reformat-imagestag-user Face API Logs ! All 0 200 400 600 800 create-user …user.insert_user …user.upload_img tag-user create-auto0-user process-images resize-images reformat-images! 837ms 406ms 66ms 114ms 122ms 82ms 240ms 157ms 35ms
- 142. all your needs in one placeTRACING
- 144. mmm… it’s a graph
- 145. what if we can query it like a graph?
- 148. ability to query based on the relationship between observed components (as well as the components themselves)
- 149. root cause analysis
- 150. the elevated error rate in service X was caused by DynamoDB table throttling.“ ”
- 151. payment was slow last night around 10PM. investigate.
- 152. time 95-percentile latency service A service B 10PM
- 153. time 95-percentile latency service A service B 10PM causality? or correlation?
- 154. user-service USESUSES DEPENDS_ON auth-serviceUSES payment-service DEPENDS_ON “payment was slow last night around 10PM” user-table
- 156. user-table user-stream DEPENDS_ON DEPENDS_ON USES USES USES USES USES DEPENDS_ON D EPEN D S_O N DEPENDS_ON PUBLISHES_TO “what else is impacted by the throttled exceptions on user-table?”
- 157. user-table user-stream DEPENDS_ON DEPENDS_ON USES USES USES USES USES DEPENDS_ON D EPEN D S_O N DEPENDS_ON PUBLISHES_TO “what else is impacted by the throttled exceptions on user-table?”
- 158. wouldn’t that be nice?
- 160. MACHINE LEARNING
- 161. use ML to auto-detect erroneous or suspicious behaviours, or to suggest possible improvements
- 163. ! Function [X] just performed an unexpected write against DynamoDB table [Y]. Should I… ignore it from now on shut it down!!
- 165. Observability Bot <bot@bestobservability.com> don’t bother me about this again
- 166. Observability Bot <bot@bestobservability.com> auto-modify IAM role with DENY rule
- 167. Function [X]’s performance has degraded since yesterday - 99% latency has gone up by 47% from 100ms to 147ms. !
- 168. ! Function [X] can run faster & cheaper if you increase its memory allocation. Should I… ignore it from now on update setting
- 169. zzz… the future of… zzz … serverless observability… zzz
- 171. Simon Wardley
- 173. However, I would argue that the health of the system no longer matters. We've entered an era where what matters is the health of each individual event, or each individual user's experience, or each shopping cart's experience (or other high cardinality dimensions). With distributed systems you don't care about the health of the system, you care about the health of the event or the slice. ”http://bit.ly/2E2QngU- Charity Majors “
- 174. “one user action/vertical slice through the system”
- 176. The best way to predict the future is to invent it. - Alan Kay
- 180. Charity Majors @mipsytipsy Cindy Sridharan @copyconstruct Erica Windisch @ewindisch Liz Fong-Jones @lizthegrey JBD @rakyll
- 181. API Gateway and Kinesis Authentication & authorisation (IAM, Cognito) Testing Running & Debugging functions locally Log aggregation Monitoring & Alerting X-Ray Correlation IDs CI/CD Performance and Cost optimisation Error Handling Configuration management VPC Security Leading practices (API Gateway, Kinesis, Lambda) Canary deployments http://bit.ly/production-ready-serverless get 40% off with code: ytcui