Distributed Tracing at UBER Scale: Creating a treasure map for your monitoring data

Editor's Notes

• #2: Thank you all for staying till the very last time segment of the conference. I am sure everyone is tired, so I promise there will be no math or cat pictures in this talk.
• #3: I am an engineer on the Observability team in NYC, here is a picture of me. It’s actually most of the NY office, both engineers and operations, on the High Line next to Hudson Yards. The Uber office is just two streets to the right. And in the direction we’re all facing, there are no buildings, so I leave it to your imagination how the picture was taken. I work on the distributed tracing system Jaeger that we recently open sourced. I was one of the co-founders of the OpenTracing project, which I will talk about later. And before joining Uber I spent way more time than any engineer should working in finance technology.
• #4: Many speakers in the last three days mentioned how valuable tracing can be.
• #9: The poll is biased. Show of hands: - How many people are using some metrics system in their organization? - And how many people have a fully functional distributed tracing solution? - That works across multiple programming languages
• #11: These adjectives actually refer to the same thing – engineers must spend time adding this instrumentation. You could say, well it’s the same thing with other monitoring, like metrics and logging. But
• #12: Engineers are used to metrics and logging. Tracing instrumentation is new, and harder to get right (or more boring) Why is that?
• #13: Tracing instrumentation requires context propagation. It means the architecture must be capable of passing around certain metadata about the distributed transaction. What does it look like in a single node?
• #14: Here we have an application or microservice, that has a handler for inbound HTTP requests, and it makes some outbound calls to other services. The instrumentation in purpose is responsible for extracting the distributed context from the request, and converting it to some in-memory representation. Then the service itself must make sure that the context object is available to instrumentation on the right to be encoded into the outbound request. In case of HTTP, the context is typically passed as one or more HTTP headers. Obvious: start with infrastructure libraries The difficult part here is actually the dotted line in the middle, passing the context inside the application – in-process propagation.
• #15: Some languages like Java and Python support thread-local storage. We can use that to pass the context implicitly. Happy story for instrumentation, no changes to the application’s code. Other languages do not have thread locals. For example, Go has no way of even identifying a go-routine. Node.js has CLS, but it’s not performant enough. So we need to explicitly pass the context, which means changes to the application code. Google / twitter experience - localized
• #16: When I said earlier “start with the infra libraries” – that’s also not so simple in a polyglot environment. You are starting to see the problem? This is why we stood behind OpenTracing from the early days
• #17: OpenTracing is an open, vendor neutral standard. Frameworks can instrument themselves! Or at minimum, it can be done via open source plugins that everybody can reuse.
• #18: But OpenTracing does not help with the need for explicit in-process context propagation in languages like Go and Node.js It has to be done manually, there is no other way. Now consider that Uber has somewhere between 2 and 3.000 microservices, and only 5 people on the tracing team. It doesn’t scale, we cannot go and change the source code of half of those services. Inevitable conclusion – application owners must do that themselves.
• #19: But they are busy shipping features. How do we incentivize them?
• #20: I call it a traveling salesman problem, 2017 edition. Find the optimal way to spend your tracing team’s time to achieve maximum level of adoption. It’s not a pure technology problem, there’s a salesmanship involved.
• #21: How do we make developers use ANY given tool? You need to give them something they want. A sticks and carrots approach can also work, because engineering organizations are not democracies.
• #22: Remember we talked about these features? It turns out the last two