Datadog and Elixir with Erlang Solutions

www.erlang-solutions.com
Elixir with
Datadog
Ludwik Bukowski
ludwik.bukowski@erlang-solutions.com
EMPEX
Los Angeles 2019

graphQL

Absinthe

Need of Observability

Need of Observability
Metrics!

Datadog
- Metrics
- Tracing
- Logs
- Analysis

Metrics - method
Agent

Metrics - method
Agent
- Statix library

Statix library
Statix.increment("hits", 1, [])
-----------> counter
Statix.gauge("cpu_usage", 0.83, [])
-----------> gauge
Statix.histogram("online_users", 123, [])
-----------> histogram
https://github.com/lexmag/statix

Metrics - instrumentation

Measure two times; in the beginning of
plugs pipeline and just before sending
response back to the user
Use:
Plug.Conn.register_before_send/2
Phoenix

request
response
Plug A
Plug B
Plug C
Plug D
...
Phoenix

request
response
Plug A
Plug B
Plug C
Plug D
...
Start Time←
← Calculate diff← End time ←

For Controllers and Views’
instrumentation use built-in Phoenix
instrumentation
Use:
config :realworld, RealWorld.Endpoint,
instrumenters: [RealWorld.Phoenix.Instrumenter]

Register callback for LogEntry structure. The
structure has fields: query_time, queue_time,
queue_time
Use:
config :real_world, RealWorld.Repo,
loggers: [
{RealWorld.Metrics, :record_metric,[]} ]
Ecto

config :real_world, RealWorld.Repo,
loggers: [
{RealWorld.Metrics, :record_metric,[]} ]

Flame
graph!
time

Spandex library
- Each query is a new Trace
- Trace have spans. Spans have metadata
- Spans storage in process dictionary or in ETS
- start_span/1, finish_span/1
https://github.com/spandex-project/spandex
Tracing - method

Tracing - instrumentation
request
response
Spandex.Plug.StartTrace
Plug B
Plug C
Spandex.Plug.EndTrace
...
Phoenix/Plug

request
response
Spandex.Plug.StartTrace
Plug B
Plug C
Spandex.Plug.EndTrace
...
Use spandex_phoenix library
Phoenix/Plug

Absinthe

start span

end span

Create span before field is resolved and
finish it after the resolution.
Use:
Implement Middleware behaviour
Absinthe

Register callback for LogEntry structure. The
structure has fields: query_time, queue_time,
queue_time. Artificially create a span, i.e
update its start end finish times.
Use:
Spandex.update_span/1
Ecto

Ecto

Ecto
Use spandex_ecto library

Simply wrap function calls with
start_span/1 and finish_span/1 using
decoration.
Use:
@decorate annotation
https://github.com/arjan/decorator
Custom code

...
@decorate tracer(:cache)
def get_from_cache(key) do
val = Cachex.get(:my_cache, key)
{:ok, val}
end
Custom code

Custom code

- Number of traces/number of spans sent should be controlled. Configurable in
spandex (batch_size and sync_threshold options)
- Metadata size consumes network between node and agent and agent and
datadog
- How track asynchronously spawned processes
- For query that involves more than one node distributed tracing in spandex can
be used.
- Tracing overhead. Few hundreds spans per trace still consumes few
milliseconds per query, measured with xprof
https://github.com/Appliscale/xprof
- Telemetry integration ?
Tracing - issues

others
- Great Datadog support team
- Events, monitors, logs and analysis

Thank you

DON’T BE JELLO.
ACCELERATE WITH OUR
ELIXIR ARCHITECTURE
SESSIONS
www.erlang-solutions.com/solutions/
elixir-development
Thank you

Datadog and Elixir with Erlang Solutions

More Related Content

What's hot (20)

Similar to Datadog and Elixir with Erlang Solutions (20)

More from Erlang Solutions (8)

Recently uploaded (20)

Datadog and Elixir with Erlang Solutions