SlideShare a Scribd company logo

Deep Dive into Keystone Tokens and Lessons Learned

Download as PPTX, PDF
Priti Desai, profile picture
Priti Desai

This document provides an in-depth exploration of keystone token formats in OpenStack, detailing UUID, PKI, PKIZ, and Fernet formats, including their generation and validation workflows. Each section outlines the configuration settings, pros and cons, and specific examples of token usage and attributes. The document aims to guide users in selecting the appropriate token format based on their deployment requirements.

Engineering
1 of 62
Downloaded 202 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
Deep Dive into Keystone Tokens and Lessons Learned Priti Desai & Brad Pokorny
Who are we? Priti Desai Advisory Software Engineer, IBM Brad Pokorny Principal Software Engineer, Symantec Deep Dive into Keystone Tokens and Lessons Learned
What token format should we configure in our OpenStack Deployment?
Token Formats Deep Dive into Keystone Tokens and Lessons Learned UUID PKI PKIZ Fernet
UUID Deep Dive into Keystone Tokens and Lessons Learned
UUID • Simplest and Most Light Weight • Version 4 UUID • Configuration in keystone.conf : [token] provider = keystone.token.providers.uuid.Provider Deep Dive into Keystone Tokens and Lessons Learned
UUID – Token Generation Workflow Keystone KVS Request Token with: • User Name • Password • Project Name Identity Resources Assignment Catalog User Validation Retrieves User ID Token Project Validation Retrieves Project ID and Domain ID Retrieves Roles for this User on the Project or Domain Returns Failure if the User does not have any Role Retrieves Services and Endpoints for all the services Bundles Identity, Resource, Assignment, and Catalog information into Token Payload Creates Token ID : uuid.uuid4().hex Store them in SQL/KVS: • Token ID • Expiration • Valid • User ID • Extra Token Generation Workflow
Sample UUID Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: f10700e71ff045cbb850072a0bd6a4e6 expires: 2015-10-08 21:18:43 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10- 08T21:18:43.995255Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["bI1EMzqUQM2sqFimOtIPpQ"], "issued_at": "2015-10-08T20:18:43.995284Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "f10700e71ff045cbb850072a0bd6a4e6", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
UUID – Keystone Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token payload from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow
UUID – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
UUID Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS UUID Tokens Tokens KVS UUID Tokens Keystone UUID - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token UUID Token Token Validation VM Instance Token Validation Token Not Found Token Found Token Not Found
Pros and cons • Pros – Simplest and Smallest Token Format – Recommended for Simple OpenStack Deployment • Cons – Persistent Token Format – Token validation can only be done by Identity service – Not feasible for multiple OpenStack deployments Deep Dive into Keystone Tokens and Lessons Learned
PKI/PKIZ Deep Dive into Keystone Tokens and Lessons Learned
PKI • Cryptographically Encrypted Signed Document using X509 Standards • CMS • Converted to custom URL- Safe format • Compressed PKI • Prefixed with “PKIZ” Deep Dive into Keystone Tokens and Lessons Learned PKIZ
PKI/PKIZ Configuration - Certificates • Signing Key (signing_key.pem) : • Generate private key in PEM format • Signing Certificate (signing_cert.pem) : • Generate CSR using Signing Key • Submit CSR to CA • Receive Certificate from CA • Certificate Authority Certificate (ca.pem) Deep Dive into Keystone Tokens and Lessons Learned
PKI/PKIZ Configuration • Configuration in keystone.conf : [token] provider = keystone.token.providers.[pki|pkiz].Provider [signing] certfile = /etc/keystone/ssl/certs/signing_cert.pem keyfile = /etc/keystone/ssl/private/signing_key.pem ca_certs = /etc/keystone/ssl/certs/ca.pem Deep Dive into Keystone Tokens and Lessons Learned
PKI/PKIZ – Token Generation Workflow Validate Identity, Resource, and Assignment Request Token with: • User Name • Password • Project Name Token Generation Workflow Create JSON Token Payload Sign JSON Payload with Signing Key and Signing Certificate openssl cms –sign –outform PEM Convert it to UTF-8 Convert CMS Signed Token in PEM format to custom URL Safe format: • “/” replaced with “-” • Deleted: “n”, “----BEGIN CMS----”,“----END CMS---- ” Compress using zlib Convert it to UTF-8 Base64 URL Safe Append Prefix PKIZ PKI PKIZ Store Token into SQL/KVS
Sample PKI Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: b460fec2efcd0d803e2baf48d3bcd72b expires: 2015-10-09 20:07:36 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-09T20:07:36.656431Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["8dh07HudSh6rHoU1G9bs-Q"], "issued_at": "2015-10-09T19:07:36.656460Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "MIIDiwYJKoZIhvcNAQcCoIIDfDCCA3gCAQExDTALBglghkgBZQMEAgEwggHZBgkqhkiG9w0BBwGgggHKBIIBxnsidG9rZW4iOnsib WV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiIxNjg4NDQ5Y2YxZGY0NDgzOWIxMGE0MWUzZDliMDlkZCIsIm5hb WUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTAtMDlUMjA6MDc6MzYuNjU2NDMxWiIsInByb2plY3QiOnsiZG9tYWluIjp7I mlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiNDIzZDQ1Y2RkZWM4NDE3MGJlMzY1ZTBiMzFhMWIxNWYiLCJuY W1lIjo…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
Sample PKIZ Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: c48321ac51a903b07c264ac3e80809c6 expires: 2015-10-12 18:45:23 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-12T18:45:23.806229Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["kKmQzTuxSnCN9vo3bzxErw"], "issued_at": "2015-10-12T17:45:23.806257Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "PKIZ_eJxtlMtyqzgQhvc8xexTqcPFdsLiLCQEWCSCgAGBdgZscbVxDOHy9CMnc6mpGlWpSmqpW39_Uuv5WTRo2tj9wy CHxiN35dqjqybi9eb6DuE7ZLd7_WxtAd6MtR1wP7PT5PxJE2F7U53WYH5D5qZbc53OSkeWPoo3hdrU7VQwhe5JBReo 71GWv72WT2vLPRk62_XuDmt_T9sZku-veT-xPfUaEk…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
PKI/PKIZ – Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token reference from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Unique ID of X-Subject-Token Hash PKI Token with the pre- configured hashing algorithm
PKI/PKIZ – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
PKI/PKIZ - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS PKI/PKIZ Tokens Tokens KVS PKI/PKIZ Tokens Keystone PKI/PKIZ - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token PKI/PKIZ Token Token Validation VM Instance Token Validation VM Instance
Pros and Cons PKI • Pros – Token validation without a request to Keystone • Cons – Larger than standard HTTP Header Size – Complex configuration – base64 –d <pki_token – Not truly feasible for multiple OpenStack Deployments PKIZ • Pros – Token validation without a request to Keystone • Cons – Still Larger than standard HTTP Header Size – Similar to PKI Deep Dive into Keystone Tokens and Lessons Learned
FERNET Deep Dive into Keystone Tokens and Lessons Learned
Fernet • Cryptographic Authentication Method – Fernet • Symmetric Key Encryption • Fernet Keys stored in /etc/keystone/fernet-keys/ – Encrypted with Primary Fernet Key – Decrypted with a list of Fernet Keys Deep Dive into Keystone Tokens and Lessons Learned
• Configuration in keystone.conf : [token] provider = keystone.token.providers.fernet.Provider [fernet_tokens] key_repository = /etc/keystone/fernet-keys/ max_active_keys = <number of keys> # default is 3 Deep Dive into Keystone Tokens and Lessons Learned Fernet Configuration
Fernet Keys • Fernet Key File - 256 bits 83b4sCF0Q4pb3aNWJYtSdtdaH8PMA_5dlN7OswXKbvE= xf3vxf8xb0!tCx8a[xddxa3V%x8bRvxd7Zx1fxc3xccx03xfe]x94xdexcexb3x05xcanxf1 Deep Dive into Keystone Tokens and Lessons Learned SHA256 HMAC Signing Key (128 bits) AES Encrypting Key (128 bits)
Fernet Keys • Fernet Key File Name - Integers starting from 0 • ls /etc/keystone/fernet-keys => 0 1 2 3 4 • Type 1: Primary Key – Encrypt and Decrypt – Key file named with the highest index • Type 2: Secondary Key – Only Decrypt – Lowest Index < Secondary Key File Name < Highest Index • Type 3: Staged Key – Decrypt and Next In Line to become Primary Key – Key file named with lowest index (of 0) Deep Dive into Keystone Tokens and Lessons Learned
Fernet Key Rotation 0 1 Primary KeyStaged Key No Secondary Key 2 Primary Key0Staged Key 1 Secondary Key 3 Primary Key21 Secondary Key 0Staged Key Secondary Key Rotate Rotate
Fernet – Token Generation Workflow Token Generation Workflow HMACFernet Token Version Current Timestamp IV Cipher Text Token Payload: Version User ID Methods Project ID Expiry Time Audit ID Padding Encrypted using Encrypting Key Signed using Signing Key
Sample Fernet Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned gAAAAABWLUzy0dxSNo2--K- 3trDutnX7LpUpv3us0crQIl8BDHLLd3lR3F243VwnYpNJHIaUiPEE2roYJJNA- SwBe1swDcr6MYaFR1t9ZYcYF4GRqDm3N9_1EGgXgICbzE_GuUVidG4gky0Cv8 f1nwD7XM26NRh59VEnt2iVTAxlnvAICJDeK5k
Fernet – Keystone Token Validation Workflow Determine the Version from the Token Payload Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Re-inflate token with “=” and return token with correct padding Version: Fixed Versioning by Keystone: • Unscoped Payload : 0 • Domain Scoped Payload : 1 • Project Scoped Payload : 2 Restore Padding Is Revoked? Token Not Found HTTP/1.1 200 OK No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Decrypt using Fernet Keys to retrieve Token Payload Disassemble payload to determine validation fields For Project Scoped Token: • User ID Project ID • Methods Token Expiry • Audit ID Current Time < Expiry Time Token Not Found No
Fernet – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
Fernet - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Keystone Fernet - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token Fernet Token Token Validation VM Instance Token Validation VM Instance Validate Fernet Token Validate Fernet Token
Pros and cons • Pros – No persistence – Reasonable Token Size – Multiple Data Center • Cons – Token validation impacted by the number of revocation events Deep Dive into Keystone Tokens and Lessons Learned
Fernet Token Validation 11.17 46.406 83.654 124.974 163.529 234.398 376.604 510.058 0 100 200 300 400 500 600 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 2000 3000 4000 Time(ms) Revocation Events Revocation Events Impact on Validation Time Response Time (ms) Deep Dive into Keystone Tokens and Lessons Learned
Fernet Token Validation 89.46 21.55 11.95 8 5.77 1.96 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 2000 3000 4000 ValidationsPerSecond Revocation Events Revocation Events Impact on Validation Requests Token Validation Requests Deep Dive into Keystone Tokens and Lessons Learned
What token format should we configure in our OpenStack Deployment? Fernet for Multiple OpenStack Deployments with minimal Revocation Events Deep Dive into Keystone Tokens and Lessons Learned
HORIZON AND TOKENS Deep Dive into Keystone Tokens and Lessons Learned
How horizon uses tokens Deep Dive into Keystone Tokens and Lessons Learned • Tokens for each logged in user • Unscoped token and project scoped token • Token reuse • Reduced transaction load on Keystone • Stored in the session • Configurable token storage methods • Local memory cache • Cookie backend • Memcache • Database • Cached Database
Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • Currently the devstack default • Token stored in browser cookie • Secure cookies in production, use https • CSRF_COOKIE_SECURE = True • SESSION_COOKIE_SECURE = True • http://docs.openstack.org/developer/horizon/topics/settings.html • http://docs.openstack.org/security-guide/dashboard/cookies.html • Highly scalable • The dreaded boot back to login
Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • The dreaded boot back to login • Now sign in…
Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • And you see… • Cookie overflow!
memcache backend Deep Dive into Keystone Tokens and Lessons Learned • Allows storage of larger token sizes • Tokens stored on server side • Requires memcached • Can be used with backing DB • http://docs.openstack.org/developer/horizon/topics/deployment.html
Token hashing Deep Dive into Keystone Tokens and Lessons Learned • Hashed in Django OpenStack Auth (DOA) • Keeps stored token data small • Currently not working for PKI tokens • New config in Liberty to disable • OPENSTACK_TOKEN_HASH_ENABLED • PKI - Will increase memcache storage requirement
Multiregion and tokens Deep Dive into Keystone Tokens and Lessons Learned • Service regions vs. Authentication regions • Service regions in Keystone catalog • Auth regions specified in AVAILABLE_REGIONS • UUID, PKI, and PKIZ Tokens don’t work across auth regions • Token replication is infeasible • But Fernet tokens work between Authentication regions! Service Region Authentication Region
Horizon and Fernet Deep Dive into Keystone Tokens and Lessons Learned • Yes, Fernet tokens work with Horizon • Liberty and beyond – No patches necessary • Kilo – Needs a patch for DOA • https://review.openstack.org/#/c/169994/
V3 domains Domain Scoped Token Project Scoped Token Deep Dive into Keystone Tokens and Lessons Learned "auth": { "identity": { }, “scope”: { ”domain": { “name”: “Default” } } } "auth": { "identity": { }, “scope”: { “project”: { ”domain": { “name”: “Default” }, “name”: “ProjectA” } } } • Extra token for Horizon
V3 domains Deep Dive into Keystone Tokens and Lessons Learned • Requires changes in Django OpenStack Auth and Horizon • Planned for Mitaka • Info on usage (a bit out of date): • http://www.symantec.com/connect/blogs/how-use-horizon-keystone-v3 • Domains patches: • https://review.openstack.org/#/c/148082/ • https://review.openstack.org/#/c/141153/ • https://review.openstack.org/#/c/196328/
Will fernet tokens solve all our problems? Deep Dive into Keystone Tokens and Lessons Learned • Smaller token size • No persistence for tokens • Seamless authentication across regions • Performance issues with token revocation
Thank You !!! Questions ?
References • Token: https://clubpenguincheatscitya4.files.wordpress.com/2011/08/1_token.jpg • Key to Cloud: https://www.hc1.com/wp- content/uploads/2013/10/14916002_cloud_computing_and_storage_security_concept_blue_glossy_cloud_icon_with_with_golden_key_in_keyhole _.jpg • User Icon: http://findicons.com/files/icons/1075/scrap/300/user_2.png • Password: http://icons.iconarchive.com/icons/sbstnblnd/plateau/512/Apps-password-icon.png • UUID: https://plugins.qgis.org/static/cache/21/c0/21c0d3fedb5bf42ff8a6a11712595124.png • PKI: http://www.zaheerspeaks.com/wp-content/uploads/2009/10/PKI-Certificate.gif • PKIZ: http://i571.photobucket.com/albums/ss153/rijal_abror/pun170-winzip-file-compress-icon59.gif • Identity: https://www.innopay.com/assets/Uploads/icon-digitalidentity-232x232.png Deep Dive into Keystone Tokens and Lessons Learned
EXTRA Deep Dive into Keystone Tokens and Lessons Learned
What is an Openstack Token? Deep Dive into Keystone Tokens and Lessons Learned Key to OpenStack Cloud
How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”MyDomain.com” }, ”name": ”PritiDesai", "password": "secretsecret” }
How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ ”token" ], ”token": { ”id": ”e8079ab…” } }
Token generated using password Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: a740dcd6f3fc404aaaf556b9cbd2f994 Body: { "token": { "methods": [ "password" ], "expires_at": "2015-10-05T20:25:03.180741Z", "extras": {}, "user": { "domain": { "id": "default", "name": "Default" }, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": ”smith" }, "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”Default” }, ”name": ”Smith", "password": "secretsecret” }
Token generated using token Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: 3fb7b3b0a0a8489882f07fdb9cd2a990 Body: { "token": { "issued_at": "2015-10-05T19:40:38.943250Z", "audit_ids": [ "4vNgmP5cQk6sMpPiw7EnCg", "HFwMKdDrSCOq-MAtkXKTlw" ], "user": { "name": ”smith", "id": "1334f3ed7eb2483b91b8192ba043b580", "domain": { "name": "Default", "id": "default" } }, "extras": {}, "auth": { "identity": { "methods": [ ”token" ], ”token": { “id”: “a7409b” } } }
Identity Token Path User OpenStack Service Step 1: Obtain unscoped token with credentials POST v3/auth/tokens Keystone Step 2: Discover projects you have access to GET v3/users/<user_id>/projects Step 3: Obtain project scoped token either with your credentials or unscoped token from step 1. Step 4: Invoke the target service by sending requests to endpoints in token from step 3 Step 5: Validate roles and access metadata in token with Keystone service or Keystone Middleware Step 6: Serve API request Step 7: Return response
Token Creation 0 10 20 30 40 50 60 70 80 90 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 5 10 15 20 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec
Token Validation 0 2 4 6 8 10 12 14 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 20 40 60 80 100 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec
Configurable token hashing https://github.com/openstack/django_openstack_auth/blob/master/openstack_auth/user.py Deep Dive into Keystone Tokens and Lessons Learned

More Related Content

PPTX
Building IAM for OpenStack
Steve Martinelli
 
PPTX
OpenStack Architecture and Use Cases
Jalal Mostafa
 
PDF
DevOps Meetup ansible
sriram_rajan
 
PPTX
Docker Networking Overview
Sreenivas Makam
 
PDF
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
Ji-Woong Choi
 
PDF
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
smalltown
 
PDF
쿠키런: 킹덤 대규모 인프라 및 서버 운영 사례 공유 [데브시스터즈 - 레벨 200] - 발표자: 용찬호, R&D 엔지니어, 데브시스터즈 ...
Amazon Web Services Korea
 
PDF
[MeetUp][1st] 오리뎅이의_쿠버네티스_네트워킹
InfraEngineer
 
Building IAM for OpenStack
Steve Martinelli
 
OpenStack Architecture and Use Cases
Jalal Mostafa
 
DevOps Meetup ansible
sriram_rajan
 
Docker Networking Overview
Sreenivas Makam
 
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
Ji-Woong Choi
 
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
smalltown
 
쿠키런: 킹덤 대규모 인프라 및 서버 운영 사례 공유 [데브시스터즈 - 레벨 200] - 발표자: 용찬호, R&D 엔지니어, 데브시스터즈 ...
Amazon Web Services Korea
 
[MeetUp][1st] 오리뎅이의_쿠버네티스_네트워킹
InfraEngineer
 

What's hot (20)

PPTX
Kubernetes CI/CD with Helm
Adnan Abdulhussein
 
PDF
[오픈소스컨설팅]쿠버네티스를 활용한 개발환경 구축
Ji-Woong Choi
 
PDF
Kubernetes Architecture and Introduction
Stefan Schimanski
 
PDF
OpenStack Architecture
Mirantis
 
PDF
Zero-Downtime Deployment with Kubernetes, SpringBoot & Flyway
Nicolas Fränkel
 
PPTX
Kubernetes PPT.pptx
ssuser0cc9131
 
PDF
Storage Capacity Management on Multi-tenant Kafka Cluster with Nurettin Omeroglu
HostedbyConfluent
 
PPTX
OVN - Basics and deep dive
Trinath Somanchi
 
PDF
[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법
Open Source Consulting
 
PDF
[OpenInfra Days Korea 2018] (Track 2) Neutron LBaaS 어디까지 왔니? - Octavia 소개
OpenStack Korea Community
 
PPTX
AWS로 게임 기반 다지기 - 김병수, 박진성 :: AWS Game Master 온라인 세미나 #3
Amazon Web Services Korea
 
PPTX
Introduction to Docker - 2017
Docker, Inc.
 
PPTX
Hashicorp Vault ppt
Shrey Agarwal
 
PDF
The basics of fluentd
Treasure Data, Inc.
 
PDF
An Introduction to Kubernetes
Imesh Gunaratne
 
PDF
[2018] 오픈스택 5년 운영의 경험
NHN FORWARD
 
ODP
Monitoring With Prometheus
Knoldus Inc.
 
PPTX
OpenTelemetry For Developers
Kevin Brockhoff
 
PDF
Red Hat OpenStack 17 저자직강+스터디그룹_1주차
Nalee Jang
 
PPTX
OpenStack Keystone
Deepti Ramakrishna
 
Kubernetes CI/CD with Helm
Adnan Abdulhussein
 
[오픈소스컨설팅]쿠버네티스를 활용한 개발환경 구축
Ji-Woong Choi
 
Kubernetes Architecture and Introduction
Stefan Schimanski
 
OpenStack Architecture
Mirantis
 
Zero-Downtime Deployment with Kubernetes, SpringBoot & Flyway
Nicolas Fränkel
 
Kubernetes PPT.pptx
ssuser0cc9131
 
Storage Capacity Management on Multi-tenant Kafka Cluster with Nurettin Omeroglu
HostedbyConfluent
 
OVN - Basics and deep dive
Trinath Somanchi
 
[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법
Open Source Consulting
 
[OpenInfra Days Korea 2018] (Track 2) Neutron LBaaS 어디까지 왔니? - Octavia 소개
OpenStack Korea Community
 
AWS로 게임 기반 다지기 - 김병수, 박진성 :: AWS Game Master 온라인 세미나 #3
Amazon Web Services Korea
 
Introduction to Docker - 2017
Docker, Inc.
 
Hashicorp Vault ppt
Shrey Agarwal
 
The basics of fluentd
Treasure Data, Inc.
 
An Introduction to Kubernetes
Imesh Gunaratne
 
[2018] 오픈스택 5년 운영의 경험
NHN FORWARD
 
Monitoring With Prometheus
Knoldus Inc.
 
OpenTelemetry For Developers
Kevin Brockhoff
 
Red Hat OpenStack 17 저자직강+스터디그룹_1주차
Nalee Jang
 
OpenStack Keystone
Deepti Ramakrishna
 
Ad

Viewers also liked (20)

PDF
OpenStack keystone identity service
openstackindia
 
PDF
Keystone fernet token
Yuki Nishiwaki
 
PDF
OpenStack Identity - Keystone (liberty) by Lorenzo Carnevale and Silvio Tavilla
Lorenzo Carnevale
 
PDF
Openstack Keystone
Kamesh Pemmaraju
 
PPTX
Keystone - Openstack Identity Service
Prasad Mukhedkar
 
PDF
Keystone: Federated
jamielennox
 
ODP
OpenStack keystone identity service
openstackindia
 
PDF
Making Glance tasks work for you - OpenStack Summit May 2015 Vancouver
Brian Rosmaita
 
PPTX
Couch to open_stack_keystone
ProfessionalVMware
 
PDF
8 Key Facts about the Keystone Pipeline
U.S. Chamber of Commerce
 
PDF
Open stack networking_101_part-1
yfauser
 
PPTX
OpenStack Storage Overview
Bharat Kumar Kobagana
 
PDF
Identity as a Service: a missing gap for moving enterprise applications in In...
Hoang Tri Vo
 
PDF
Stateless authentication with OAuth 2 and JWT - JavaZone 2015
Alvaro Sanchez-Mariscal
 
PPTX
IDaaS. The Now Big Thing
Nishant Kaushik
 
PDF
Diffie-Hellman key exchange
hughpearse
 
PPTX
IdM vs. IDaaS
Drew Koenig
 
PPTX
AWS IAM and security
Erik Paulsson
 
PDF
Marriott management philosophy
Fawad Akhtar
 
PDF
Solinea Lazuli Tower Project Brief
sevenseaspropertycorp
 
OpenStack keystone identity service
openstackindia
 
Keystone fernet token
Yuki Nishiwaki
 
OpenStack Identity - Keystone (liberty) by Lorenzo Carnevale and Silvio Tavilla
Lorenzo Carnevale
 
Openstack Keystone
Kamesh Pemmaraju
 
Keystone - Openstack Identity Service
Prasad Mukhedkar
 
Keystone: Federated
jamielennox
 
OpenStack keystone identity service
openstackindia
 
Making Glance tasks work for you - OpenStack Summit May 2015 Vancouver
Brian Rosmaita
 
Couch to open_stack_keystone
ProfessionalVMware
 
8 Key Facts about the Keystone Pipeline
U.S. Chamber of Commerce
 
Open stack networking_101_part-1
yfauser
 
OpenStack Storage Overview
Bharat Kumar Kobagana
 
Identity as a Service: a missing gap for moving enterprise applications in In...
Hoang Tri Vo
 
Stateless authentication with OAuth 2 and JWT - JavaZone 2015
Alvaro Sanchez-Mariscal
 
IDaaS. The Now Big Thing
Nishant Kaushik
 
Diffie-Hellman key exchange
hughpearse
 
IdM vs. IDaaS
Drew Koenig
 
AWS IAM and security
Erik Paulsson
 
Marriott management philosophy
Fawad Akhtar
 
Solinea Lazuli Tower Project Brief
sevenseaspropertycorp
 
Ad

Similar to Deep Dive into Keystone Tokens and Lessons Learned (6)

PPTX
OpenStack Toronto Meetup - Keystone 101
Steve Martinelli
 
PDF
CIS 2015- Building IAM for OpenStack- Steve Martinelli
CloudIDSummit
 
PDF
Keystone deep dive 1
Jsonr4
 
PPTX
Secure Keystone Deployment
Priti Desai
 
PPTX
PKI token as a secure mechanism of Keystone authentication system for OpenStack
shaerraezzaty
 
PPTX
Identity service keystone ppt
university of Gujrat, pakistan
 
OpenStack Toronto Meetup - Keystone 101
Steve Martinelli
 
CIS 2015- Building IAM for OpenStack- Steve Martinelli
CloudIDSummit
 
Keystone deep dive 1
Jsonr4
 
Secure Keystone Deployment
Priti Desai
 
PKI token as a secure mechanism of Keystone authentication system for OpenStack
shaerraezzaty
 
Identity service keystone ppt
university of Gujrat, pakistan
 

Recently uploaded (20)

PDF
composite construction of structures.pdf
AinieButt1
 
PPTX
Construction Project Organization Group 2.pptx
vj5agdales
 
PPT
CRASH COURSE IN ALTERNATIVE PLUMBING CLASS
rene militante
 
PPTX
Lecture Notes Electrical Wiring System Components
eedgecbhojpur
 
PPTX
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
ShriNRPrasad
 
PDF
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 
PDF
PPT on Performance Review to get promotions
prashant593122
 
PPTX
additive manufacturing of ss316l using mig welding
premcdr7799
 
PPTX
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
PDF
Enhancing Cyber Defense Against Zero-Day Attacks using Ensemble Neural Networks
IJCNCJournal
 
PDF
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
PDF
PRIZ Academy - 9 Windows Thinking Where to Invest Today to Win Tomorrow.pdf
PRIZ Guru
 
PPTX
bas. eng. economics group 4 presentation 1.pptx
kiribakimoses1993
 
PPTX
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
PPTX
Internet of Things (IOT) - A guide to understanding
Davies Chacko
 
PDF
keyrequirementskkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk
mojeeburahmanwardak
 
DOCX
ASol_English-Language-Literature-Set-1-27-02-2023-converted.docx
AYUSHMANAV
 
PDF
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
PPTX
OOP with Java - Java Introduction (Basics)
Rashmi T V
 
PDF
R24 SURVEYING LAB MANUAL for civil enggi
MNANDITHACIVILSTAFF
 
composite construction of structures.pdf
AinieButt1
 
Construction Project Organization Group 2.pptx
vj5agdales
 
CRASH COURSE IN ALTERNATIVE PLUMBING CLASS
rene militante
 
Lecture Notes Electrical Wiring System Components
eedgecbhojpur
 
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
ShriNRPrasad
 
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 
PPT on Performance Review to get promotions
prashant593122
 
additive manufacturing of ss316l using mig welding
premcdr7799
 
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
Enhancing Cyber Defense Against Zero-Day Attacks using Ensemble Neural Networks
IJCNCJournal
 
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
PRIZ Academy - 9 Windows Thinking Where to Invest Today to Win Tomorrow.pdf
PRIZ Guru
 
bas. eng. economics group 4 presentation 1.pptx
kiribakimoses1993
 
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
Internet of Things (IOT) - A guide to understanding
Davies Chacko
 
keyrequirementskkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk
mojeeburahmanwardak
 
ASol_English-Language-Literature-Set-1-27-02-2023-converted.docx
AYUSHMANAV
 
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
OOP with Java - Java Introduction (Basics)
Rashmi T V
 
R24 SURVEYING LAB MANUAL for civil enggi
MNANDITHACIVILSTAFF
 

Deep Dive into Keystone Tokens and Lessons Learned

  • 1. Deep Dive into Keystone Tokens and Lessons Learned Priti Desai & Brad Pokorny
  • 2. Who are we? Priti Desai Advisory Software Engineer, IBM Brad Pokorny Principal Software Engineer, Symantec Deep Dive into Keystone Tokens and Lessons Learned
  • 3. What token format should we configure in our OpenStack Deployment?
  • 4. Token Formats Deep Dive into Keystone Tokens and Lessons Learned UUID PKI PKIZ Fernet
  • 5. UUID Deep Dive into Keystone Tokens and Lessons Learned
  • 6. UUID • Simplest and Most Light Weight • Version 4 UUID • Configuration in keystone.conf : [token] provider = keystone.token.providers.uuid.Provider Deep Dive into Keystone Tokens and Lessons Learned
  • 7. UUID – Token Generation Workflow Keystone KVS Request Token with: • User Name • Password • Project Name Identity Resources Assignment Catalog User Validation Retrieves User ID Token Project Validation Retrieves Project ID and Domain ID Retrieves Roles for this User on the Project or Domain Returns Failure if the User does not have any Role Retrieves Services and Endpoints for all the services Bundles Identity, Resource, Assignment, and Catalog information into Token Payload Creates Token ID : uuid.uuid4().hex Store them in SQL/KVS: • Token ID • Expiration • Valid • User ID • Extra Token Generation Workflow
  • 8. Sample UUID Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: f10700e71ff045cbb850072a0bd6a4e6 expires: 2015-10-08 21:18:43 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10- 08T21:18:43.995255Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["bI1EMzqUQM2sqFimOtIPpQ"], "issued_at": "2015-10-08T20:18:43.995284Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "f10700e71ff045cbb850072a0bd6a4e6", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
  • 9. UUID – Keystone Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token payload from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow
  • 10. UUID – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
  • 11. UUID Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS UUID Tokens Tokens KVS UUID Tokens Keystone UUID - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token UUID Token Token Validation VM Instance Token Validation Token Not Found Token Found Token Not Found
  • 12. Pros and cons • Pros – Simplest and Smallest Token Format – Recommended for Simple OpenStack Deployment • Cons – Persistent Token Format – Token validation can only be done by Identity service – Not feasible for multiple OpenStack deployments Deep Dive into Keystone Tokens and Lessons Learned
  • 13. PKI/PKIZ Deep Dive into Keystone Tokens and Lessons Learned
  • 14. PKI • Cryptographically Encrypted Signed Document using X509 Standards • CMS • Converted to custom URL- Safe format • Compressed PKI • Prefixed with “PKIZ” Deep Dive into Keystone Tokens and Lessons Learned PKIZ
  • 15. PKI/PKIZ Configuration - Certificates • Signing Key (signing_key.pem) : • Generate private key in PEM format • Signing Certificate (signing_cert.pem) : • Generate CSR using Signing Key • Submit CSR to CA • Receive Certificate from CA • Certificate Authority Certificate (ca.pem) Deep Dive into Keystone Tokens and Lessons Learned
  • 16. PKI/PKIZ Configuration • Configuration in keystone.conf : [token] provider = keystone.token.providers.[pki|pkiz].Provider [signing] certfile = /etc/keystone/ssl/certs/signing_cert.pem keyfile = /etc/keystone/ssl/private/signing_key.pem ca_certs = /etc/keystone/ssl/certs/ca.pem Deep Dive into Keystone Tokens and Lessons Learned
  • 17. PKI/PKIZ – Token Generation Workflow Validate Identity, Resource, and Assignment Request Token with: • User Name • Password • Project Name Token Generation Workflow Create JSON Token Payload Sign JSON Payload with Signing Key and Signing Certificate openssl cms –sign –outform PEM Convert it to UTF-8 Convert CMS Signed Token in PEM format to custom URL Safe format: • “/” replaced with “-” • Deleted: “n”, “----BEGIN CMS----”,“----END CMS---- ” Compress using zlib Convert it to UTF-8 Base64 URL Safe Append Prefix PKIZ PKI PKIZ Store Token into SQL/KVS
  • 18. Sample PKI Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: b460fec2efcd0d803e2baf48d3bcd72b expires: 2015-10-09 20:07:36 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-09T20:07:36.656431Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["8dh07HudSh6rHoU1G9bs-Q"], "issued_at": "2015-10-09T19:07:36.656460Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "MIIDiwYJKoZIhvcNAQcCoIIDfDCCA3gCAQExDTALBglghkgBZQMEAgEwggHZBgkqhkiG9w0BBwGgggHKBIIBxnsidG9rZW4iOnsib WV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiIxNjg4NDQ5Y2YxZGY0NDgzOWIxMGE0MWUzZDliMDlkZCIsIm5hb WUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTAtMDlUMjA6MDc6MzYuNjU2NDMxWiIsInByb2plY3QiOnsiZG9tYWluIjp7I mlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiNDIzZDQ1Y2RkZWM4NDE3MGJlMzY1ZTBiMzFhMWIxNWYiLCJuY W1lIjo…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
  • 19. Sample PKIZ Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: c48321ac51a903b07c264ac3e80809c6 expires: 2015-10-12 18:45:23 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-12T18:45:23.806229Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["kKmQzTuxSnCN9vo3bzxErw"], "issued_at": "2015-10-12T17:45:23.806257Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "PKIZ_eJxtlMtyqzgQhvc8xexTqcPFdsLiLCQEWCSCgAGBdgZscbVxDOHy9CMnc6mpGlWpSmqpW39_Uuv5WTRo2tj9wy CHxiN35dqjqybi9eb6DuE7ZLd7_WxtAd6MtR1wP7PT5PxJE2F7U53WYH5D5qZbc53OSkeWPoo3hdrU7VQwhe5JBReo 71GWv72WT2vLPRk62_XuDmt_T9sZku-veT-xPfUaEk…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
  • 20. PKI/PKIZ – Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token reference from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Unique ID of X-Subject-Token Hash PKI Token with the pre- configured hashing algorithm
  • 21. PKI/PKIZ – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
  • 22. PKI/PKIZ - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS PKI/PKIZ Tokens Tokens KVS PKI/PKIZ Tokens Keystone PKI/PKIZ - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token PKI/PKIZ Token Token Validation VM Instance Token Validation VM Instance
  • 23. Pros and Cons PKI • Pros – Token validation without a request to Keystone • Cons – Larger than standard HTTP Header Size – Complex configuration – base64 –d <pki_token – Not truly feasible for multiple OpenStack Deployments PKIZ • Pros – Token validation without a request to Keystone • Cons – Still Larger than standard HTTP Header Size – Similar to PKI Deep Dive into Keystone Tokens and Lessons Learned
  • 24. FERNET Deep Dive into Keystone Tokens and Lessons Learned
  • 25. Fernet • Cryptographic Authentication Method – Fernet • Symmetric Key Encryption • Fernet Keys stored in /etc/keystone/fernet-keys/ – Encrypted with Primary Fernet Key – Decrypted with a list of Fernet Keys Deep Dive into Keystone Tokens and Lessons Learned
  • 26. • Configuration in keystone.conf : [token] provider = keystone.token.providers.fernet.Provider [fernet_tokens] key_repository = /etc/keystone/fernet-keys/ max_active_keys = <number of keys> # default is 3 Deep Dive into Keystone Tokens and Lessons Learned Fernet Configuration
  • 27. Fernet Keys • Fernet Key File - 256 bits 83b4sCF0Q4pb3aNWJYtSdtdaH8PMA_5dlN7OswXKbvE= xf3vxf8xb0!tCx8a[xddxa3V%x8bRvxd7Zx1fxc3xccx03xfe]x94xdexcexb3x05xcanxf1 Deep Dive into Keystone Tokens and Lessons Learned SHA256 HMAC Signing Key (128 bits) AES Encrypting Key (128 bits)
  • 28. Fernet Keys • Fernet Key File Name - Integers starting from 0 • ls /etc/keystone/fernet-keys => 0 1 2 3 4 • Type 1: Primary Key – Encrypt and Decrypt – Key file named with the highest index • Type 2: Secondary Key – Only Decrypt – Lowest Index < Secondary Key File Name < Highest Index • Type 3: Staged Key – Decrypt and Next In Line to become Primary Key – Key file named with lowest index (of 0) Deep Dive into Keystone Tokens and Lessons Learned
  • 29. Fernet Key Rotation 0 1 Primary KeyStaged Key No Secondary Key 2 Primary Key0Staged Key 1 Secondary Key 3 Primary Key21 Secondary Key 0Staged Key Secondary Key Rotate Rotate
  • 30. Fernet – Token Generation Workflow Token Generation Workflow HMACFernet Token Version Current Timestamp IV Cipher Text Token Payload: Version User ID Methods Project ID Expiry Time Audit ID Padding Encrypted using Encrypting Key Signed using Signing Key
  • 31. Sample Fernet Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned gAAAAABWLUzy0dxSNo2--K- 3trDutnX7LpUpv3us0crQIl8BDHLLd3lR3F243VwnYpNJHIaUiPEE2roYJJNA- SwBe1swDcr6MYaFR1t9ZYcYF4GRqDm3N9_1EGgXgICbzE_GuUVidG4gky0Cv8 f1nwD7XM26NRh59VEnt2iVTAxlnvAICJDeK5k
  • 32. Fernet – Keystone Token Validation Workflow Determine the Version from the Token Payload Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Re-inflate token with “=” and return token with correct padding Version: Fixed Versioning by Keystone: • Unscoped Payload : 0 • Domain Scoped Payload : 1 • Project Scoped Payload : 2 Restore Padding Is Revoked? Token Not Found HTTP/1.1 200 OK No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Decrypt using Fernet Keys to retrieve Token Payload Disassemble payload to determine validation fields For Project Scoped Token: • User ID Project ID • Methods Token Expiry • Audit ID Current Time < Expiry Time Token Not Found No
  • 33. Fernet – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
  • 34. Fernet - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Keystone Fernet - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token Fernet Token Token Validation VM Instance Token Validation VM Instance Validate Fernet Token Validate Fernet Token
  • 35. Pros and cons • Pros – No persistence – Reasonable Token Size – Multiple Data Center • Cons – Token validation impacted by the number of revocation events Deep Dive into Keystone Tokens and Lessons Learned
  • 37. Fernet Token Validation 89.46 21.55 11.95 8 5.77 1.96 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 2000 3000 4000 ValidationsPerSecond Revocation Events Revocation Events Impact on Validation Requests Token Validation Requests Deep Dive into Keystone Tokens and Lessons Learned
  • 38. What token format should we configure in our OpenStack Deployment? Fernet for Multiple OpenStack Deployments with minimal Revocation Events Deep Dive into Keystone Tokens and Lessons Learned
  • 39. HORIZON AND TOKENS Deep Dive into Keystone Tokens and Lessons Learned
  • 40. How horizon uses tokens Deep Dive into Keystone Tokens and Lessons Learned • Tokens for each logged in user • Unscoped token and project scoped token • Token reuse • Reduced transaction load on Keystone • Stored in the session • Configurable token storage methods • Local memory cache • Cookie backend • Memcache • Database • Cached Database
  • 41. Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • Currently the devstack default • Token stored in browser cookie • Secure cookies in production, use https • CSRF_COOKIE_SECURE = True • SESSION_COOKIE_SECURE = True • http://docs.openstack.org/developer/horizon/topics/settings.html • http://docs.openstack.org/security-guide/dashboard/cookies.html • Highly scalable • The dreaded boot back to login
  • 42. Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • The dreaded boot back to login • Now sign in…
  • 43. Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • And you see… • Cookie overflow!
  • 44. memcache backend Deep Dive into Keystone Tokens and Lessons Learned • Allows storage of larger token sizes • Tokens stored on server side • Requires memcached • Can be used with backing DB • http://docs.openstack.org/developer/horizon/topics/deployment.html
  • 45. Token hashing Deep Dive into Keystone Tokens and Lessons Learned • Hashed in Django OpenStack Auth (DOA) • Keeps stored token data small • Currently not working for PKI tokens • New config in Liberty to disable • OPENSTACK_TOKEN_HASH_ENABLED • PKI - Will increase memcache storage requirement
  • 46. Multiregion and tokens Deep Dive into Keystone Tokens and Lessons Learned • Service regions vs. Authentication regions • Service regions in Keystone catalog • Auth regions specified in AVAILABLE_REGIONS • UUID, PKI, and PKIZ Tokens don’t work across auth regions • Token replication is infeasible • But Fernet tokens work between Authentication regions! Service Region Authentication Region
  • 47. Horizon and Fernet Deep Dive into Keystone Tokens and Lessons Learned • Yes, Fernet tokens work with Horizon • Liberty and beyond – No patches necessary • Kilo – Needs a patch for DOA • https://review.openstack.org/#/c/169994/
  • 48. V3 domains Domain Scoped Token Project Scoped Token Deep Dive into Keystone Tokens and Lessons Learned "auth": { "identity": { }, “scope”: { ”domain": { “name”: “Default” } } } "auth": { "identity": { }, “scope”: { “project”: { ”domain": { “name”: “Default” }, “name”: “ProjectA” } } } • Extra token for Horizon
  • 49. V3 domains Deep Dive into Keystone Tokens and Lessons Learned • Requires changes in Django OpenStack Auth and Horizon • Planned for Mitaka • Info on usage (a bit out of date): • http://www.symantec.com/connect/blogs/how-use-horizon-keystone-v3 • Domains patches: • https://review.openstack.org/#/c/148082/ • https://review.openstack.org/#/c/141153/ • https://review.openstack.org/#/c/196328/
  • 50. Will fernet tokens solve all our problems? Deep Dive into Keystone Tokens and Lessons Learned • Smaller token size • No persistence for tokens • Seamless authentication across regions • Performance issues with token revocation
  • 52. References • Token: https://clubpenguincheatscitya4.files.wordpress.com/2011/08/1_token.jpg • Key to Cloud: https://www.hc1.com/wp- content/uploads/2013/10/14916002_cloud_computing_and_storage_security_concept_blue_glossy_cloud_icon_with_with_golden_key_in_keyhole _.jpg • User Icon: http://findicons.com/files/icons/1075/scrap/300/user_2.png • Password: http://icons.iconarchive.com/icons/sbstnblnd/plateau/512/Apps-password-icon.png • UUID: https://plugins.qgis.org/static/cache/21/c0/21c0d3fedb5bf42ff8a6a11712595124.png • PKI: http://www.zaheerspeaks.com/wp-content/uploads/2009/10/PKI-Certificate.gif • PKIZ: http://i571.photobucket.com/albums/ss153/rijal_abror/pun170-winzip-file-compress-icon59.gif • Identity: https://www.innopay.com/assets/Uploads/icon-digitalidentity-232x232.png Deep Dive into Keystone Tokens and Lessons Learned
  • 53. EXTRA Deep Dive into Keystone Tokens and Lessons Learned
  • 54. What is an Openstack Token? Deep Dive into Keystone Tokens and Lessons Learned Key to OpenStack Cloud
  • 55. How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”MyDomain.com” }, ”name": ”PritiDesai", "password": "secretsecret” }
  • 56. How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ ”token" ], ”token": { ”id": ”e8079ab…” } }
  • 57. Token generated using password Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: a740dcd6f3fc404aaaf556b9cbd2f994 Body: { "token": { "methods": [ "password" ], "expires_at": "2015-10-05T20:25:03.180741Z", "extras": {}, "user": { "domain": { "id": "default", "name": "Default" }, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": ”smith" }, "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”Default” }, ”name": ”Smith", "password": "secretsecret” }
  • 58. Token generated using token Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: 3fb7b3b0a0a8489882f07fdb9cd2a990 Body: { "token": { "issued_at": "2015-10-05T19:40:38.943250Z", "audit_ids": [ "4vNgmP5cQk6sMpPiw7EnCg", "HFwMKdDrSCOq-MAtkXKTlw" ], "user": { "name": ”smith", "id": "1334f3ed7eb2483b91b8192ba043b580", "domain": { "name": "Default", "id": "default" } }, "extras": {}, "auth": { "identity": { "methods": [ ”token" ], ”token": { “id”: “a7409b” } } }
  • 59. Identity Token Path User OpenStack Service Step 1: Obtain unscoped token with credentials POST v3/auth/tokens Keystone Step 2: Discover projects you have access to GET v3/users/<user_id>/projects Step 3: Obtain project scoped token either with your credentials or unscoped token from step 1. Step 4: Invoke the target service by sending requests to endpoints in token from step 3 Step 5: Validate roles and access metadata in token with Keystone service or Keystone Middleware Step 6: Serve API request Step 7: Return response
  • 60. Token Creation 0 10 20 30 40 50 60 70 80 90 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 5 10 15 20 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec
  • 61. Token Validation 0 2 4 6 8 10 12 14 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 20 40 60 80 100 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec

Editor's Notes

  • #8: Identity: Checks if User exist in User Domain Check if User is enabled Retrieves User ID Matches Password Resource: Checks if Domain or Project exist Check if Domain or Project is enabled Retrieves Project ID and Domain ID Catalog: Retrieves Services associated with User’s Project Retrieves the list of endpoints for all the services
  • #18: Identity: Checks if User exist in User Domain Check if User is enabled Retrieves User ID Matches Password Resource: Checks if Domain or Project exist Check if Domain or Project is enabled Retrieves Project ID and Domain ID Catalog: Retrieves Services associated with User’s Project Retrieves the list of endpoints for all the services
  • #31: Identity: Checks if User exist in User Domain Check if User is enabled Retrieves User ID Matches Password Resource: Checks if Domain or Project exist Check if Domain or Project is enabled Retrieves Project ID and Domain ID Catalog: Retrieves Services associated with User’s Project Retrieves the list of endpoints for all the services
  • #40: I’ll next take us through how Horizon uses tokens. We’ve used Horizon for most of the time we’ve been using OpenStack, and we’ve found some interesting behaviors with the way Horizon manages them.
  • #41: When logging in, Horizon gets a keystone token using the user’s credentials. This is an important aspect of security in Horizon, as Horizon doesn’t require a service credential, minimizing the impact of an attacker compromising Horizon itself. First get an unscoped token, which is used to get scoped tokens to projects when the user switches projects. The storage method for tokens is configurable and can have a large impact depending on the complexity of your cloud. The configurable methods are: We are using the memcache backend. I’ll talk more just about the cookie backend and Memcache backend.
  • #42: The cookie backend has some very strong advantages, and it’s currently the devstack default. In this case, tokens are stored in a browser cookie on the client side. If using the cookie backend in production, it’s important to configure https connections to Horizon and also configure security for the tokens. Otherwise, someone could recover a token while sniffing the network. The cookie backend is highly scalable, as token storage is all done on the client side. However, the cookie backend can’t be used if you have many endpoints in the keystone catalog.
  • #43: Cookie sizes for most browsers are about 4KB. When the token takes up a lot of that space, you’ll log into Horizon, sign in…
  • #44: ..And see this!
  • #45: Using the memcache backend resolves cookie overflow issues. We currently use it at Symantec.
  • #46: Token Hashing has been used in the past to reduce the impact of large token sizes. We currently use token hashing with Kilo Horizon, which works fine with Hashing, but you could have issues if using the master branch.
  • #47: For UUID, PKI, and PKIZ, Tokens won’t work across auth regions. This will be a benefit of Fernet tokens, as they will allow authentication between multiple keystone instances.
  • #48: Fernet tokens work with Horizon
  • #49: This is somewhat out of scope for the different token types.
  • #54: I’ll next take us through how Horizon uses tokens. We’ve used Horizon for most of the time we’ve been using OpenStack, and we’ve found some interesting behaviors with the way Horizon manages them.
  • #63: This is just a code example for the Liberty code that disables token hashing.