Self scaling Multi cloud nomad workloads

Self scaling
multi cloud
nomad workloads

Cloud Engineer @ Seaplane.io
@attachmentgenie
Bram
Vogelaar

What would it take
to keep the
platform running
(no matter what)?
And more importantly….

Vertical vs Horizontal Scaling

Absorb additional pressure and pay for
what you need

The aim becomes 100% 99.999% uptime

Task Scheduler
Spec Sheet
Open-source tool for dynamic workload
scheduling
Batch, containerized, and non-containerized
applications.
Jobs written in (H)ashiCorp (C)onﬁguration
(L)anguage
https://www.nomadproject.io/

CODE EDITOR
Lets build awesome product
job "lorem-ipsum" {
group ”frontend" {
network {
port "http" { to = ”3000” }
}
service {
name = ”lorem"
port = ”http"
}
task "server" {
driver = "docker"
config {
image = ”cicero/lorem-ipsum:v1.0.0"
ports = ["http"]
}
}

CODE EDITOR
Expose it to the world
## traefik.yml
entryPoints:
web:
address: :80
providers:
file:
directory: /etc/traefik.d/dynamic.yml
watch: true

CODE EDITOR
## dynamic.yml
http:
routers:
lorem-ipsum:
rule: "Host(`lorem-ipsum.io`)"
service: lorem-ipsum
services:
lorem-ipsum:
loadBalancer:
servers:
- url: http://localhost:26489

CODE EDITOR
Sleep++ step1
job "lorem-ipsum" {
count = 2
constraint {
operator = "distinct_hosts"
value = "true"
}

Your scheduler
should not be an
expensive systemd
Service Discovery + Service Mesh + Scheduler

Service Discovery + Service Mesh
Spec Sheet
Open-Source Service Discovery Tool
Build-in KV store
Service Mesh tool
Uses watchers to have near instant feedback
loops
https://www.consul.io/

CODE EDITOR
Exposing as a service
job "lorem-ipsum" {
service {
name = "www"
tags = ["metrics",”traefik.enable=true”]
port = "http"
check {
name = "alive"
type = "http"
path = "/health"
interval = "10s"
timeout = "2s"
}
}

CODE EDITOR
## dynamic.yml
http:
routers:
lorem-ipsum:
rule: "Host(`lorem-ipsum.io`)"
service: lorem-ipsum
services:
lorem-ipsum:
loadBalancer:
servers:
- url: http://lorem-ipsum.service.consul:26489

CODE EDITOR
Sleep++ lazy++
## traefik.yml
entryPoints:
web:
address: :80
providers:
consulCatalog:
defaultRule: "Host(`{{ .Name }}.lorem-ipsum.io`)"
endpoint:
address: 'http://localhost:8500'
exposedByDefault: false

CODE EDITOR
Dynamic Metric Scraping
# prometheus.yml
- job_name: nomad_workload
scrape_interval: 60s
consul_sd_configs:
- server: localhost:8500
tags:
- metrics

CODE EDITOR
Sleep++ Autoscaling
job "lorem-ipsum" {
scaling {
enabled = false
min = 1
max = 20
policy {
cooldown = "20s"
check "avg_sessions" {
source = "prometheus"
query = "sum(traefik_entrypoint_open_connections{entrypoint="lorem-ipsum"}"
strategy "target-value" {
target = 5
}
}
}
}

CODE EDITOR
Nomad Federation
server {
enabled = true
bootstrap_expect = 3
server_join {
authoritative_region = “mgmt”
retry_join = [
"1.1.1.1", “1.1.1.2”, “1.1.1.3”,
“2.2.2.1”,”2.2.2.2”,”2.2.2.3”
]
retry_max = 3
retry_interval = "15s"
}
}

CODE EDITOR
Sleep++ Separate mgmt from workload
server {
enabled = true
bootstrap_expect = 3
server_join {
authoritative_region = “mgmt”
retry_join = [ "1.1.1.1", “1.1.1.2”, “1.1.1.3”,
“2.2.2.1”,”2.2.2.2”,”2.2.2.3” ]
retry_max = 3
retry_interval = "15s"
}
}

CODE EDITOR
Sleep++ Ingress
resource "ns1_record" "www" {
zone = “lorem-ipsum.io”
domain = "www.lorem-ipsum.io"
type = "CNAME"
ttl = 60
regions {
name = "usa"
meta = {
country = "US"
}
}
answers {
answer = "usa.lorem-ipsum.io"
region = "usa"
}
}

CODE EDITOR
Identify workload region
external_labels:
datacenter: "%{::trusted.extensions.pp_datacenter}"
region: "%{::trusted.extensions.pp_region}" # ${IATA_CODES}
env: "%{::trusted.extensions.pp_environment}"

What would it take
to keep it running
while losing an
entire region
Limit the blast radius

Spread your risks!
US
AWS
GCP
DO
AZ
EU
AWS
GCP
DO
AZ

Not all vendors are
created equally
VPCs vs Network mesh, Base Images etc,etc

CODE EDITOR
Lazy++ Redirect to other regions
## traefik.yml
entryPoints:
web:
address: :80
providers:
consul:
endpoints:
address: 'http://localhost:8500'
consul kv put traefik/http/services/www/loadbalancer/servers/0/url "-X PUT
-d 'http://usa.lorem-ipsum.io'"

CODE EDITOR
Sleep++ Redirect to other regions
curl http://127.0.0.1:8500/v1/query
--request POST
--data @- << EOF
{
"Name": "www",
"Service": {
"Service": "www",
"Failover": {
"Datacenters": ["us", "eu"]
}
}
}
EOF

CODE EDITOR
Sleep++ Redirect to other regions
curl http://127.0.0.1:8500/v1/query
--request POST
--data @- << EOF
{
"Name": "www",
"Service": {
"Service": "www",
"Failover": {
"NearestN": 2,
"Datacenters": ["us", "eu"]
}
}
}
EOF

Follow the sun?
Follow the moon?
Discover nearest?
Sleep++ Consul watchers

Thank You
bram@attachmentgenie.com | @attachmentgenie | https://www.slideshare.net/attachmentgenie

Self scaling Multi cloud nomad workloads

More Related Content

Similar to Self scaling Multi cloud nomad workloads (20)

More from Bram Vogelaar (20)

Recently uploaded (20)

Self scaling Multi cloud nomad workloads