Tatu: ssh as a service

Tatu
SSH as a Service
for OpenStack
February 2018

Some problems with SSH in OpenStack
1. MITM vulnerability on first connection.
2. No automated way to revoke user access.
3. Need a FloatingIP per instance (or manage bastions).

Problem 1: MITM vulnerability
ECDSA key fingerprint is….
Are you sure you want to continue connecting (yes/no)?
Does anyone check the fingerprint?
How would we anyway? We don’t have password access.

Problem 2: Public key management
OpenStack writes the user public key to the instance’s
authorized_keys file.
It’s definitely better than password authentication.
But how do multiple users access the instance?
● Share the private key?
● Add more keys to authorized_keys file… but who cleans
up?

Problem 3: FloatingIP per instance
- Wastes FloatingIP addresses.
- Avoid waste by using some
instances as bastions.
● Who manages the bastions and how?
- Or you could SSH over IPv6.
● IPv6 readiness on client and server side?

Solution: SSH Certificates
● Available since OpenSSH 5.4
● For both Users and Hosts
● SSH client trusts the Certificate Authority, not the host.
● Similarly, SSH host trusts the CA, not the user.

SSH HostSSH User
SSH with Certificates
Host CA (Green) vouches for hosts.
User CA (Red) vouches for users.
/home/user/.ssh/key.pub
/home/user/.ssh/key
/home/user/.ssh/known_hosts
/home/user/.ssh/key-cert.pub
/etc/ssh/rsa_key.pub
/etc/ssh/rsa_key
/etc/ssh/ca_user.pub
/etc/ssh/rsa_key-cert.pub
I trust hosts that the
Green CA vouches for.
sshd_config TrustedUserCAKeys
sshd_config HostCertificate
I trust users that the
Red CA vouces for.

Ok, so what’s Tatu?
● SSH as a Service, named after author of SSH
● Introduces SSH Certificates to OpenStack
● OpenStack CLI and Horizon panels for users to:
○ Copy the Host CA public key to known_hosts file
○ Generate new certificates
○ Revoke old certificates
● Automates the SSH setup on instances
○ Using Nova (dynamic and static) vendor data + cloud-init.
● Manages SSH bastions and DNS for ease of use
● Integrates with Barbican, Designate, Keystone, Nova...

User certificate generation
Tatu
Barbican
OpenStack
DB (MySQL)
Keystone
1) Please sign my public key?
2) Fetch the project’s
User CA private key.
3) Fetch the user’s
roles in the project.
4) Store the user and
certificate metadata.
6) Customer
served!
5. Run
ssh-keygen
in a
temporary
directory

User certificate details
# ssh-keygen -Lf ~/.ssh/key22-cert.pub
/root/.ssh/key22-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:ZDsaPxjKHlBo6BWf3R0OKokrNU+T3TDy8MU5v1YOJAY
Signing CA: RSA SHA256:LQ5ikXe8LybhCFiuWGuiVqagSIyy2eiYpRhhu9lWnfw
Key ID: "testID"
Serial: 22
Valid: from 2018-02-08T18:01:56 to 2019-02-09T18:01:56
Principals:
admin
root
Critical Options: (none)
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc
In-progress: set principals by querying Keystone for user’s roles in project.
Done: use serial numbers for certificate revocation.
In-progress: set the Key ID to the user’s name.

New instance (host) certificate generation
Tatu
Barbican
OpenStack
DB (MySQL)
Keystone
8) Fetch the project’s
Host CA private key,
then sign.
2) Fetch all
the roles in
the project.
5/9) Store
host/cert
metadata.
Nova
1) Dynamic vendor data, please?
ConfigDrive:
- SSH setup scripts
- One-time Password
- User CA public key
- User account names
Dragonflow
3) Reserve
ports for
PAT bastion
Designate
4) Create
type SRV
DNS records
6) Served!
7) Please
sign my
public key?
MetaData
Proxy
10) Served!

Instance periodically fetches CRL (cron job)
Tatu
Barbican
OpenStack
DB (MySQL)
Keystone
2) Store revoked
serial numbers
per project.
Dragonflow
Designate
3) I need
the latest
CRL!
MetaData
Proxy
5) Served!
1) Please revoke certificates
with serial numbers X, Y...
4) Fetch all the
revoked user
serial numbers
for the project.

PAT (port address translation) bastions
● Real bastion VMs are better (planned for a future Tatu release).
● But PAT already allows us to avoid FloatingIP-per-instance.
● Like load-balancing: each VM is the single backend/target for a single port on
a few (for redundancy) load-balancers.
● Currently only works with Dragonflow (TODO: use LBaaS API).
robin
goat
hamster
User A
User B
User C
PAT 1
172.24.4.11
PAT 2
172.24.4.28
Port 50
Port 51
Port 52
Port 42
Port 43
Port 44

Integration with Designate
● Users shouldn’t have to remember bastion or instance IP addresses.
● They should just SSH to a URL composed of instance and project names.
● In the case of PAT bastions, we look up SRV records rather than A records.
### tatu/scripts/srvssh -i ~/.ssh/key root@goat.666a812a.tatuPAT.com.
srv: '_ssh._tcp.goat.666a812a.tatuPAT.com has SRV record 10 50 52
bastion-172-24-4-11.tatuPAT.com.'
srv: 'bastion-172-24-4-11.tatuPAT.com has address 172.24.4.11'
After SRV lookup -- HOST: 172.24.4.11 PORT: 52
/usr/bin/ssh -v -i /root/.ssh/key22 -p 52 root@172.24.4.11
In the example, the instance name is ‘goat’; project ID prefix is ‘666a812a’;
srvssh is a wrapper script that resolves the SRV record and then calls ssh.

Successful use of certificates looks like...
/usr/bin/ssh -v -i /root/.ssh/key22 -p 52 root@172.24.4.11
OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g 1 Mar 2016
...
debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com
SHA256:tobg2ZPvX45OpqQh9dHlRiZxkg9Vmv6gUQaV1T9dyss, serial 0 ID "testID" CA
ssh-rsa SHA256:qS95+uo3KEzwxyrihHfe4NWqKTUX64bRSfCWuG5i5RM valid from
2018-02-07T18:26:19 to 2019-02-08T18:26:19
debug1: Host '[172.24.4.11]:52' is known and matches the RSA-CERT host
certificate.
debug1: Found CA key in /root/.ssh/known_hosts:1
...
debug1: Offering RSA public key: /root/.ssh/key22
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering RSA-CERT public key: /root/.ssh/key22-cert
debug1: Server accepts key: pkalg ssh-rsa-cert-v01@openssh.com blen 1095
Client accepted host cert
Server rejected user’s public key since
it’s not in the authorized_keys file.
Host accepted user cert

In the background...
A Tatu daemon reacts to these oslo event notifications:
● Project creation: create new CA key pairs.
● User deletion: revoke all the user’s certificates.
● Host deletion: clean up PAT port entries in Dragonflow
and DNS entries in Designate.

Recap of some details...
● Each OpenStack project has separate user and host CA.
● CA private keys are stored in Barbican.
● Tatu does not see user/host private keys.
○ Later we’ll consider generating high-quality private keys as an RFE.
● Puts a one-time password in the instance’s ConfigDrive;
the instance uses it in its certificate generation request.
● Instance runs a cron job to periodically fetch the CRL.
● User accounts on instance correspond to Keystone roles.

Summary: OpenStack’s current user experience
● Generate SSH key pair (locally with ssh-keygen)
● Upload public key to OpenStack
● Launch VM - include key pair and assign Floating IP (or manage bastion)
● SSH into VM
○ ECDSA key fingerprint is…. Are you sure you want to continue connecting (yes/no)?
○ How do I verify the fingerprint? I’ll just cross my fingers.
● When multiple colleagues want access to same VMs, share private keys or
add public keys to authorized_keys file.
● When colleagues leave wait weeks or months before getting around to
remove they public keys or re-generate/rotate shared keys.

Summary: SSH user experience with Tatu
● Generate SSH key pair (locally with ssh-keygen)
● Get your key signed by your project’s CA
○ /home/pino/.ssh/key-cert.pub
● Put the CA’s host public key in the known_hosts file
○ @cert-authority <domain> ssh-rsa AAAAB3NzaC…
● Launch VM without key pair and without Floating IP.
● SSH into VM via automatically assigned PAT IP+port and without MITM risk.
● Colleagues automatically have access without sharing keys.
● Access can be revoked with a click and is automatically revoked if the user is
deleted.

Reference commands
● host -t SRV _ssh._tcp.<hostname>.<project_id>.tatuPAT.com. localhost
● dig @localhost -t SRV _ssh._tcp.fox.666a812a.tatuPAT.com.
● /opt/stack/tatu/scripts/srvssh -i ~/.ssh/key root@fox.666a812a.tatuPAT.com.
● ssh -i ~/.ssh/key -p 40 root@172.24.4.11
● openstack sshca show -f value -c 'Host Public Key' <project_id>
● echo '@cert-authority *' `openstack sshca show -f value -c 'Host Public Key'
666a812a-3dd4-4a64-9855-2b8617813001` > /root/.ssh/known_hosts
● openstack usercert create -f value -c Certificate "`cat .ssh/key14.pub`" >
.ssh/key14-cert.pub
● openstack usercert revoke <serial number>

Project Status
● Tatu code in OpenStack Github/Gerrit since early February.
● Tatu-dashboard in OpenStack Gerrit since Feb. 20.
● python-tatuclient not imported yet (as of Feb. 23)
● Demo-quality code - needs cleanup and hardening!
● Needs automated testing, installers, and more.
● In progress:
○ Devstack scripts
○ User accounts created for Keystone roles
● Outreach started in February. Looking for contributors and users!!!

Future work
● Bastion VM management
● Centralized SSH audit logs
● Use Uber’s PAM module to validate existing SSH sessions
○ https://github.com/uber/pam-ussh
● Rotation of CA keys
● Rotation of host certificates
● Ansible installer (and Kolla!)
Ideas are welcome!

Resources
● https://github.com/openstack/tatu
● https://github.com/openstack/tatu-dashboard
● https://github.com/pinodeca/python-tatuclient
● https://launchpad.net/tatu
Demo Video: https://youtu.be/y6ICCPO08d8
IRC: #openstack-tatu on freenode.net

Thanks!
pino.decandia at huawei dot com
giuseppe.decandia at gmail dot com
@pino_deca on Twitter

Tatu: ssh as a service

More Related Content

What's hot (20)

Similar to Tatu: ssh as a service (20)

Recently uploaded (20)

Tatu: ssh as a service