Experiences in Providing Secure Mult-Tenant Lustre Access to OpenStack
1 like841 views
The document discusses the implementation of secure multi-tenant Lustre access in an OpenStack environment, highlighting performance and security considerations. It outlines the critical configuration steps for integrating Lustre filesystems with OpenStack while addressing challenges such as legacy pipelines and the need for high-performance shared storage. Key findings indicate that both physical and virtual Lustre routers can deliver acceptable performance while providing enhanced security and fault isolation for tenants.
![Lustre server:
Map nodemap to network
Add the tenant network range to the Lustre nodemap
lctl nodemap_add_range --name ${TENANT_NAME} --range
[0-255].[0-255].[0-255].[0-255]@tcp${TENANT_UID}
And this command adds a route via a Lustre network router. This is run on all
MDS and OSS (or the route added to
/etc/modprobe.d/lustre.conf)
lnetctl route add --net tcp${TENANT_UID} --gateway ${LUSTRE_ROUTER_IP}@tcp
In the same way a similar command is needed on each client using TCP](https://image.slidesharecdn.com/wellcome-trust-sanger-sc17-user-group-171213163551/85/Experiences-in-Providing-Secure-Mult-Tenant-Lustre-Access-to-OpenStack-11-320.jpg)
Experiences in Providing Secure Mult-Tenant Lustre Access to OpenStack
- 1. Experiences in providing secure multi-tenant Lustre access to OpenStack Peter Clapham <pc7@sanger.ac.uk> Wellcome Trust Sanger Institute
- 2. Sanger Science Scientific Research Programmes Core Facilities
- 3. HPC and Cloud computing are complementary Traditional HPC ● Highest possible performance ● A mature and centrally managed compute platform ● High performance Lustre filesystems for data intensive analysis Flexible Compute ● Full segregation of projects ensures data security ● Developers no longer tied to a single stack ● Reproducibility through containers / images and infrastructure-as-code
- 4. But there’s a catch or two... • Large number of traditional/legacy pipelines • They require a performant shared POSIX filesystem, while cloud workloads support object stores • We do not always have the source code or expertise to migrate • We need multi-gigabyte per second performance • The tenant will have root • and could impersonate any user, but Lustre trusts the client’s identity assertions, just like NFSv3 • The solution must be simple for the tenant and administrator
- 5. Lustre hardware 6+ year old hardware • 4x Lustre object storage servers • Dual Intel E5620 @ 2.40GHz • 256GB RAM • Dual 10G network • lustre: 2.9.0.ddnsec2 • https://jira.hpdd.intel.com/browse/LU-9289 (landed in 2.10) • OSTs from DDN SFA-10k • 300x SATA, 7200rpm , 1TB spindles We have seen this system reach 6 GByte/second in production
- 7. Lustre 2.9 features • Each tenant’s I/O can be squashed to their own unique UID/GID • Each tenant is restricted to their own subdirectory of the Lustre filesystem It might be possible to treat general access outside of OpenStack as a separate tenant with: • a UID space reserved for a number of OpenStack tenants • only a subdirectory exported for standard usage
- 8. Public network tcp32769 Provider Network Lustre router Lustre serverstcp0 Tenant network tcp32770 Provider Network Logical layout Tenant network
- 9. Lustre server Per-tenant UID mapping Allows UIDs from a set of NIDs to be mapped to another set of UIDs These commands are run on the MGS: lctl nodemap_add ${TENANT_NAME} lctl nodemap_modify --name ${TENANT_NAME} --property trusted --value 0 lctl nodemap_modify --name ${TENANT_NAME} --property admin --value 0 lctl nodemap_modify --name ${TENANT_NAME} --property squash_uid --value ${TENANT_UID} lctl nodemap_modify --name ${TENANT_NAME} --property squash_gid --value ${TENANT_UID} lctl nodemap_add_idmap --name ${TENANT_NAME} --idtype uid --idmap 1000:${TENANT_UID}
- 10. Lustre server: Per-tenant subtree restriction Constrains client access to a subdirectory of a filesystem. mkdir /lustre/secure/${TENANT_NAME} chown ${TENANT_NAME} /lustre/secure/${TENANT_NAME} Set the subtree root directory for the tenant: lctl set_param -P nodemap.${TENANT_NAME}.fileset=/${TENANT_NAME}
- 11. Lustre server: Map nodemap to network Add the tenant network range to the Lustre nodemap lctl nodemap_add_range --name ${TENANT_NAME} --range [0-255].[0-255].[0-255].[0-255]@tcp${TENANT_UID} And this command adds a route via a Lustre network router. This is run on all MDS and OSS (or the route added to /etc/modprobe.d/lustre.conf) lnetctl route add --net tcp${TENANT_UID} --gateway ${LUSTRE_ROUTER_IP}@tcp In the same way a similar command is needed on each client using TCP
- 12. Openstack: Network configuration neutron net-create LNet-1 --shared --provider:network_type vlan --provider:physical_network datacentre --provider:segmentation_id ${TENANT_PROVIDER_VLAN_ID} neutron subnet-create --enable-dhcp --dns-nameserver 172.18.255.1 --no-gateway --name LNet-subnet-1 --allocation-pool start=172.27.202.17,end=172.27.203.240 172.27.202.0/23 ${NETWORK_UUID} openstack role create LNet-1_ok For each tenant user that needs to create instances attached to this Lustre network: openstack role add --project ${TENANT_UUID} --user ${USER_UUID} ${ROLE_ID}
- 13. Openstack policy Simplify automation by minimal change to /etc/neutron/policy.json "get_network": "rule:get_network_local" /etc/neutron/policy.d/get_networks_local.json then defines the new rule: { "get_network_local": "rule:admin_or_owner or rule:external or rule:context_is_advsvc or rule:show_providers or ( not rule:provider_networks and rule:shared )" }
- 14. Openstack policy /etc/neutron/policy.d/provider.json is used to define networks and their mapping to roles, and allow access to the provider network. { "net_LNet-1": "field:networks:id=d18f2aca-163b-4fc7-a493-237e383c1aa9", "show_LNet-1": "rule:net_LNet-1 and role:LNet-1_ok", "net_LNet-2": "field:networks:id=169b54c9-4292-478b-ac72-272725a26263", "show_LNet-2": "rule:net_LNet-2 and role:LNet-2_ok", "provider_networks": "rule:net_LNet-1 or rule:net_LNet-2", "show_providers": "rule:show_LNet-1 or rule:show_LNet-2" } Restart Neutron - can be disruptive!
- 15. Physical router configuration • Repurposed Nova compute node • RedHat 7.3 • Lustre 2.9.0.ddnsec2 • Mellanox ConnectX-4 (2*25GbE) • Dual Intel E5-2690 v4 @ 2.60GHz • 512 GB RAM Connected in a single rack so packets from other racks will have to transverse the spine. No changes from default settings.
- 16. Client virtual machines • 2 CPU • 4 GB RAM • CentOS Linux release 7.3.1611 (Core) • Lustre: 2.9.0.ddnsec2 • Two NICs • Tenant network • Tenant-specific Lustre provider network
- 17. Filesets and uid mapping have no effect Instance size has little effect Single client read performance
- 18. Single client write performance
- 19. Multiple VMs, aggregate write performance, metal LNet routers
- 20. Multiple VMs, aggregate read performance, metal LNet routers
- 21. Virtualised Lustre routers • We could see that bare metal Lustre routers gave acceptable performance • We wanted to know if we could virtualise these routers • Each tenant could have their own set of virtual routers • Fault isolation • Ease of provisioning routers • No additional cost • Increases east-west traffic, but that’s OK.
- 22. Public network Tenant network tcp32769 Provider Network Lustre router Lustre servers tcp0 Provider Network Tenant network Lustre router tcp32770 Provider Network Logical layout
- 23. Improved security As each tenant has its own set of Lustre routers: • The traffic to a different tenant does not go to a shared router • A Lustre router could be compromised without directly compromising another tenant’s data - the filesystem servers will not route data for @tcp1 to the router @tcp2 • Either a second Lustre router or the Lustre servers would need to be compromised to intercept or reroute the data
- 24. Port security... The routed Lustre provider network ( tcp32769 etc) required that port security was disabled on the virtual Lustre router ports. neutron port-list | grep 172.27.70.36 | awk '{print $2}' 08a1808a-fe4a-463c-b755-397aedd0b36c neutron port-update --no-security-groups 08a1808a-fe4a-463c-b755-397aedd0b36c neutron port-update 08a1808a-fe4a-463c-b755-397aedd0b36c --port-security-enabled=False http://kimizhang.com/neutron-ml2-port-security/ We would need to have iptables inside the instance rather than rely on iptables in the ovs/hypervisor. The tests do not include this.
- 25. Virtual Lnet routers: Sequential performance
- 26. Virtual Lnet routers: Random Performance
- 27. Asymmetric routing? http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html Public network Lustre routers Lustre servers tcp0 Tenant network tcp32770 Provider Network Hostile system (e.g. laptop) tcp0 tcp0
- 28. Conclusions • Follow our activities on http://hpc-news.sanger.ac.uk • Isolated POSIX islands can be deployed to OpenStack with Lustre 2.9+ • Performance is acceptable • Lustre routers require little CPU and memory • Physical routers work and can give good locality for network usage • Virtual routers work, can scale and give additional security benefits • Next steps: • Improve configuration automation • Understand the neutron port security issue • Improve network performance (MTU, OpenVSwitch etc).
- 29. Acknowledgements DDN: Sébastien Buisson, Thomas Favre-Bulle, Richard Mansfield, James Coomer Sanger Informatics Systems Group: Pete Clapham, James Beal, John Constable, Helen Cousins, Brett Hartley, Dave Holland, Jon Nicholson, Matthew Vernon