Distributed tracing with service meshes and tracing spans across polyglot Microservices

Distributed tracing
Unified microservices:
The service mesh story
—
Denilson Nastacio
Senior Software Eng.

@dnastacio
sourcepatch.blogspot.com
github.com/nastacio/
About me
husband
father of 3 (boys)
educator
(little league) soccer coach
backyard farmer
master inventor
cancer fighter (former)
anime and sci-fi fan
avid reader
aspiring writer
IBM Cloud / March 12, 2020 / © 2020 IBM Corporation 2
http://sourcepatch.blogspot.com/search/label/clinical_trials
http://www.rtpscrolls.com/

Contents
Introduction
Why microservices? 04
Landscape 06
Observability 07
Instrumentation
Example microservices architecture 12
Istio injection 11
Distributed Tracing Model 12
Code modifications 13
Visualization
Grafana (monitoring externals) 19
Kiali (monitoring internals) 20
Jaeger (tracing) 21

Introduction
Microservices
Cloud Native Landscape
Service mesh
Distributed Tracing Data Model

Time
Why microservices?
IBM Cloud / March 12, 2020 / © 2020 IBM Corporation
5
“Fridayafternoon”

Time
Why microservices?
“When you hit a wall,
focus on one brick.”
Tyler May
“Fridayafternoon”
“And don’t forget the
manuals and brick
adapters.”
Me

Observability and
Analysis
Monitoring
Tracing
Logging
What…
• …is working?
• …is not working
When…
• …did it start?
• …does it happen? Fixed, cyclic,
random?
• …does it happen? Deploy, start,
runtime?
• …worsening or stable?
Where…
• …which instances?
• …which functions?
• …which regions?
• …in combination with others?
Why…
• What is connected to what?
• What is not connected to what?
• What is taking too long?
• Why is taking too long?
• How different are those two
transactions?
• Where is this request going
through?
• “Unable to connect to SSL services due to ‘PKIX Path
Building Failed’
• “Printing is not supported on this printer”
• “Initiating shutdown of Southeast region”
• “Installation succeeded but failed with errors.”
• “Failed to startup SSH session: Socket error: success

Distributed Tracing
Data Model
https://github.com/opentracing/specification/blob/master/specification.md
• Traces
• Spans
• References
Collection of spans (directed acyclic graph)
Typically associated with a business transaction
Microservice “C”
Microservice “A”
Microservice “B”
• Operation name
• Start and finish timestamp
• Tags
• Logs
• Baggage Items
• References to causally-related spans
Trace id: 124861aedfa quoteItem
Span “quoteItem”
Span “B-phase 1” Span “B-phase 2”
ChildOfChildOf
Span “quoteItem”
“B-phase 1” “B-phase 2”
Time

Istio Injection
Istio.io
• Connect
• Secure
• Control
• Observe
Microservice A
Microservice C
Microservice B
3rd party service
Observe

Instrumentation with
Node.js, Java Spring
Boot, JEE (Open Liberty)

Microservice “C”
Microservice “A”
Microservice “B”
Business transaction
OpenTracing traffic
Jaeger Backend Jaeger UI
Kiali UI
Prometheus
Istio injection traffic
Our program(s) for
today Grafana

Istio injection
$ kubectl label namespace tracing istio-injection=enabled
$ kubectl get deployment nodejs-tracing -o yaml |
istioctl kube-inject -f - |
kubectl apply -f -
or
+
metadata:
annotations:
...
sidecar.istio.io/inject: "true"
...
labels:
app: nodejs-tracing
version: v1
...
deployment.yaml
Automatic injection on
all deployments
across entire
namespace
Injection to specific
deployment

nodejs-tracing
├ app.js
└ package.json
Distributed Tracing
Microservice “A” (Node.js)
const opentracing = require('opentracing')
var initTracerFromEnv = require('jaeger-client').initTracerFromEnv
var config = { serviceName: 'app-a-nodejs' }
var options = {}
var tracer = initTracerFromEnv(config, options)
...
app.get('/node-springboot', (req, res) => {
const wireCtx = tracer.extract(opentracing.FORMAT_HTTP_HEADERS, req.headers)
const span = tracer.startSpan(spanName, { childOf: wireCtx })
span.log({ event: 'request_received' })
...
span.finish()
}
app.js
{
"name": "app-a-nodejs",
...,
"dependencies": {
"jaeger-client": ...,
"opentracing": ...,
...
},
}
package.json

springboot-tracing
├ pom.xml
└ src/main/java/application
├ Main.java
└ ServiceResource.java
Distributed Tracing
Microservice “B” (Spring Boot)
import io.jaegertracing.Configuration;
import io.opentracing.Tracer;
@SpringBootApplication
public class Main {
@Bean
public Tracer initTracer() {
return Configuration.fromEnv("app-b-springboot").getTracer();
}
Main.java
<dependencies>
...
<groupId>io.opentracing.contrib</groupId>
<artifactId>opentracing-spring-cloud-starter</artifactId>
...
<groupId>io.opentracing</groupId>
<artifactId>opentracing-api</artifactId>
...
<groupId>io.jaegertracing</groupId>
<artifactId>jaeger-client</artifactId>
...
</dependencies>
pom.xml
import io.opentracing.Span;
import io.opentracing.tag.Tags;
@RestController
public class ServiceResource {
@Autowired
private Tracer tracer;
@GetMapping(value = "/resource", produces = "application/json")
public ResponseEntity<Object> quoteItem(...) {
...
Span parentSpan = tracer.scopeManager().activeSpan();
Span spanPhase1 = tracer.buildSpan("phase_1")
.asChildOf(parentSpan).start();
...
spanPhase1.setTag(Tags.ERROR.getKey(), true);
...
spanPhase1.finish();
}
ServiceResource.java

jee-tracing
├ pom.xml
├ src/main
├ java/application
│ └ ServiceResource.java
└ main/liberty
└ server.xml
Distributed Tracing
Microservice “C” (Open Liberty)
<server description="Liberty server">
<featureManager>
<feature>microProfile-3.2</feature>
<feature>mpOpenTracing-1.3</feature>
...
<webApplication location="app-c-jee.war" contextRoot="/" >
<classloader apiTypeVisibility="+third-party" />
</webApplication>
</server>
server.xml
<dependencies>
...
<groupId>io.jaegertracing</groupId>
<artifactId>jaeger-client</artifactId>
pom.xml
import io.opentracing.Scope;
import io.opentracing.tag.Tags;
@Path("/resource”)
public class ServiceResource {
@Inject
private Tracer tracer;
@GET
public ResponseEntity<Object> completeOrder(...) {
...
try (Scope childScope = tracer.buildSpan("phase_1")
.startActive(true)) {
...
childScope.span().setTag(Tags.ERROR.getKey(), true);
...
}
...
}
ServiceResource.
java
https://openliberty.io/blog/2019/12/06/microprofile-32-health-metrics-190012.html#jmo

Visualizations with
Grafana (monitoring)
Kiali (deeper monitoring)
Jaeger (tracing)

Grafana
• Charts and dashboards shareable
and composable through JSON
files
• Explore Data with PromQL, the
Prometheus Query Language
• PromQL = Prometheus Query
Language
• Dashboards contain multiple
charts
count by (destination_service_name,response_code)
(istio_requests_total{destination_service_namespace="tracing",response_code
!="200"})

Kiali
metadata:
...
labels:
app: nodejs-tracing
version: v1
...
deployment.yaml

Kiali
virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: jee-tracing
spec:
hosts:
- jee-tracing
http:
- fault:
abort:
httpStatus: 502
percentage:
value: 30
route:
- destination:
host: jee-tracing
- route:
- destination:
host: jee-tracing

Kiali (workloads)

Jaeger
C
A
B

Jaeger – Troubleshooting
virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: spring-tracing
spec:
hosts:
- spring-tracing
http:
- fault:
delay:
fixedDelay: 1s
percentage:
value: 90
route:
- destination:
host: spring-tracing
- route:
- destination:
host: spring-tracing

Closing thoughts

Best Practices for
Distributed Tracing
• Service Names
• Span names
• Tags
• Baggage
• Timestamps
• Extract span context when receiving a call
• Inject span context before making a call
https://opentracing.io/docs/best-practices/instrumenting-frameworks/
If no timestamp provided, “wall time” is used. How much precision do you need?
Service meshes like Istio will inject spans if none is found, but often you want to
propagate the context across many of the nodes servicing a higher-level transaction
There are some standard tags, such as “error”, “log”, HTTP request information, remote
peer information.
Baggage text

Takeaways
1. A basic understanding of how distributed tracing
relates to a product architecture
2. Concrete examples of how to instrument
microservices in a Kubernetes environment
3. Concrete examples of how to visualize and act upon
distribute tracing information

Next steps
Dive deeper into OpenTracing
https://opentracing.io/docs/overview
Complete source code and instructions for the technical bits
https://github.com/IBM/icpa-opentracing
(Yet another) great “101” guide from my colleague Carlos Santana:
https://ibm-cloud-architecture.github.io/learning-distributed-tracing-
101/docs/index.html
Try Istio locally
https://istio.io
Guided Istio workshop
https://learn.openshift.com/servicemesh/

Distributed tracing with service meshes and tracing spans across polyglot Microservices

More Related Content

Similar to Distributed tracing with service meshes and tracing spans across polyglot Microservices (20)

More from Nebulaworks (19)

Recently uploaded (20)

Distributed tracing with service meshes and tracing spans across polyglot Microservices