Easy Cloud Native Transformation with Nomad

Easy Cloud Native Transformation
with HashiCorp Nomad
Bram Vogelaar
@attachmentgenie

• Used to be a Molecular Biologist
• Then became a Dev, now an Ops
• Currently Cloud Engineer @ The Factory
• Amsterdam HUG organizer
• Almost author => Workload Orchestration Made Easy With Nomad
$ whoami

And than stuff got complicated….

l Open-source tool for dynamic workload scheduling
l Batch, containerized, and non-containerized applications.
l Has native Consul and Vault integrations.
l Has token based access setup.
l Jobs written in (H)ashiCorp (C)onfiguration (L)anguage
https://www.nomadproject.io/
Nomad

Deploy the blog
job "blog" {
group ”hugo" {
network {
port "http" { to = ”80” }
}
service {
name = ”attachmentgeniedotcom"
port. = ”http"
}
task ”nginx" {
driver = "docker"
config {
image = "${PRIVATE}.dkr.ecr.us-east-1.amazonaws.com/blog:19”
ports = ["http"]
}
}
}

1 == None
job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 2

Force onto different hardware
job "blog" {
type = "service"
group "hugo" {
count = 2
constraint {
operator = "distinct_hosts"
value = "true"
}

Suggest onto different hardware
job "blog" {
type = "service"
group "hugo" {
count = 2
Spread {
attribute = "${node.datacenter}"
}

Based on custom meta-data
/etc/nomad.d/config.hcl
Client {
Enabled = true
Meta {
"rack" = "his"
}
}
blog.nomad
job "blog" {
group "hugo" {
Spread {
attribute = "${meta.rack}"
target "his" {
percent = 50
}
target "her" {
percent = 50
}
}

service {
name = "grafana"
provider = "nomad"
port = "grafana_ui"
check {
type = "http"
path = "/api/health"
interval = "10s"
timeout. = "2s"
}
tags = [
"traefik.enable=true",
]
}
Native Service Discovery ( > 1.3)

template {
data = <<EOH
server {
listen *:80;
server_name grafana.teambla.com;
location / {
{{ range nomadService ”grafana" }}
proxy_pass http://{{ .Address }}:{{ .Port }};
{{ end }}
….
}
EOH
destination = "local/nginx.conf"
}
Native Service Discovery ( > 1.3)

$ nomad alloc checks f6683eb8
Status of 1 Nomad Service Checks
ID = e47351cef1c81ca56bf70be6b48ae44a
Name = service: "grafana" check
Group = grafana.grafana[0]
Task = grafana
Service = grafana
Status = success
StatusCode = 200
Mode = healthiness
Timestamp = 2022-10-03T12:43:48Z
Output = nomad: http ok
Service Checks

$ cat grafana.nomad
service {
name = "grafana"
provider = "nomad"
port = "grafana_ui"
tags = [
"traefik.enable=true",
]
}
Ingress / Traefik
$ cat traefik.yaml
providers:
nomad:
address: http://${attr.unique.network.ip-address}:4646”
exposedByDefault: false
https://traefik.io <- 2.8.0

Metrics / Prometheus
https://prometheus.io <- 2.37.0
$ cat grafana.nomad
service {
name = "grafana"
provider = "nomad"
port = "grafana_ui"
tags = [
"metrics",
]
}
$ cat prometheus.yaml
scrape_configs:
- job_name: 'nomad_sd'
nomad_sd_configs:
- server: 'http://{{ env "attr.unique.network.ip-address" }}:4646'
relabel_configs:
- source_labels: ['__meta_nomad_tags']
regex: '(.*),metrics,(.*)'
action: keep
- source_labels: [__meta_nomad_service]
target_label: job

l Open-Source Service Discovery Tool
l Build-in KV store
l Service Mesh tool
l Uses watchers to have near instant feedback loops
https://www.consul.io/
Consul

• Levant
• Templating and packaging tool
• Easily deploy popular applications to Nomad
• Re-use common patterns across internal applications
• Find and share job definitions with the Nomad community
• Jobs written in (H)ashiCorp (C)onfiguration (L)anguage
• Templates are written using Go Template Syntax.
• Nightlies only right now!
Nomad Pack
https://github.com/hashicorp/nomad-pack

variable "datacenters" {
description = "A list of datacenters in the region which are eligible for task placement"
type = list(string)
default = [“dc1”]
}
Variable “resources” {
description = “The resource to assign to the blog service task”
type = object({
cpu = number
memory = number
})
default = {
cpu = 200,
memory = 256
}
}
variables.hcl

$ cat packs/blog/templates/blog.nomad.tpl
….
datacenters = [[ .my.datacenters | toStringList ]]
…
resources {
cpu = [[ .my.blog_resources.cpu ]]
memory = [[ .my.blog_resources.memory ]]
}
…
Pack Templates

• nomad-pack registry list
• nomad-pack registry add https://github.com/attachmentgenie/hashitalks-registry
• nomad-pack run blog --var namespace=dashboard
• nomad-pack run packs/blog -f vars/blog.hcl
Pack Registries
https://github.com/hashicorp/nomad-pack-community-registry

Secrets
task "grafana" {
driver = "docker"
config {
image = "grafana/grafana:9.2.3"
ports = ["grafana_ui"]
}
env {
{{- with nomadVar "nomad/jobs/grafana" -}}
GF_SECURITY_ADMIN_USER = {{. GF_SECURITY_ADMIN_USER}}
GF_SECURITY_ADMIN_PASSWORD = {{. GF_SECURITY_ADMIN_PASSWORD}}
{{- end -}}
}

l Open-Source tool to do secrets management
l Secure, store and tightly control access to tokens, passwords,
encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.
l Certificate management
l Password rotation (e.g. SSH, MySQL)
https://www.vaultproject.io/
Vault

Nomad autoscaler
Introduced in/with Nomad 0.11
(Currently) independently release cycle
Gaining new functionality every release
Build in Functionality for horizontal and vertical scaling
But extendable by your own (community) plugins

● Makes decisions based on a checks
● Checks are a combination of
• Data queried from an APM
• Defined STRATEGY
• Attempt to approach TARGET value
● Multiple Checks can be combined
• Answer with the most resources will win!
• ScaleOut and ScaleIn => ScaleOut
• ScaleOut and ScaleNone => ScaleOut
• ScaleIn and ScaleNone => ScaleNone
• ScaleOut(10) and ScaleOut(9) => ScaleOut(10)
• ScaleIn(3) and ScaleIn(4) => ScaleIn(4)
Auto-scaling TLDR

/etc/nomad.d/config.hcl
nomad {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
strategy "target-value" {
driver = "target-value"
}
Config for the autoscaler

job "blog" {
type = "service"
group "hugo" {
count = 3
scaling {
enabled = true
min = 1
max = 20
policy {
cooldown = “20s”
check “avg_instance_sessions” {
source = “prometheus”
query = “scalar(avg(traefik_service_open_connections{service=”blog@consulcatalog”}))”
strategy “target-value” {
target = 5
}
Enable autoscaling for the blog

Moving it all to the cloud - QED

Questions ?
The Floor is yours…
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie

Easy Cloud Native Transformation with Nomad

More Related Content

What's hot (20)

Similar to Easy Cloud Native Transformation with Nomad (20)

More from Bram Vogelaar (20)

Recently uploaded (20)

Easy Cloud Native Transformation with Nomad