Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pulsar Summit NA 2021
1 like442 views
This document summarizes a presentation about Apache Pulsar multi-tenancy and security features at Verizon Media. It discusses how Pulsar implements tenant and namespace isolation through storage quotas, throttling policies, broker and bookie isolation. It also covers authentication, authorization, encryption in transit and at rest, and how Pulsar proxy supports SNI routing for hybrid cloud deployments and cross-organization replication. Future plans include tenant-based broker virtualization and hybrid cloud deployments with geo-replication.
![Broker Isolation
● Regex of Namespaces to
Regex of Brokers/IP Range
● Primary and Secondary broker
Regexes
13
Broker Isolation
Why
● High Profile/Reserved capacity
● Misbehaving tenants
● Debugging
bin/pulsar-admin ns-isolation-policy set
--auto-failover-policy-type min_available
--auto-failover-policy-params min_limit=5,usage_threshold=80
--namespaces ‘my-tenant/.*’
--primary ‘broker-mytenant[0-9]+.mydomain’ --secondary
‘spare[0-9]+.mydomain’ my-cluster policy-name](https://image.slidesharecdn.com/pulsarsummit2021-multitenancyandsecurity-210623162509/85/Security-and-Multi-Tenancy-with-Apache-Pulsar-in-Yahoo-Verizon-Media-Pulsar-Summit-NA-2021-13-320.jpg)
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pulsar Summit NA 2021
- 1. Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement. Apache Pulsar Multi-tenancy and Security June 17, 2021 Rajan Dhabalia rdhabalia@verizonmedia.com Ludwig Pummer ludwig@verizonmedia.com 1
- 2. Speakers 2 Rajan Dhabalia Principal Software Engineer, Verizon Media Ludwig Pummer Principal Production Engineer, Verizon Media
- 3. Agenda ● Pulsar in Yahoo/Verizon Media ● Multi tenancy ● Security ● SNI routing and proxy support ● Future ● QA 3
- 4. Pulsar journey in Yahoo ● Developed as a hosted pub-sub service within Yahoo/VMG ○ open-sourced in 2016 ● Global deployment ○ 6 DC (Asia, Europe, US) ○ full mesh replication ● Mission critical use cases ○ Serving applications ○ Lower latency bus for use by other low latency services ○ Write availability 4
- 5. ● Pulsar scale and storage evolution talk https://pulsar-summit.org/en/event/virtual-conference-2020/sessions/pulsar-storage-on-bookkeepe r-seamless-evolution ● Pulsar growth since 2015 ○ 120+ tenants and 15M rps ○ Storage evolution : HDD, SDD, NVMe, PMEM ○ On-prem, public-cloud and cross org integration ● Scale but what about multi-tenancy? Scale & Multi-Tenancy 5
- 6. 6 Secured multi-tenant system with Apache Pulsar
- 7. Multi-tenancy & Security Requirement 7 Multi-tenancy Tenant and Namespace IO isolation Quota and Throttling Broker and Bookie isolation Anti-affinity group Security Authentication & Authorization Encryption in transit Encryption at rest Pulsar proxy Support ATS, HAProxy, Nginx
- 9. Tenant ● Highest level of provisioning ● Unit of administration ● Managed by Pulsar administrators ● Usually one team 9 Tenant and Namespace Namespace ● Middle level of provisioning ● Unit of data policy ● Managed by Pulsar administrators and/or Tenants ● Usually one application/use case persistent://tenant/namespace/topic
- 10. 1. Portal find User to Team mapping 2. User creates or modifies tenant ○ Tenant name, Admin Authorization Principals ○ Clusters, WPS & RPS Estimates ○ Jira project, Contact Info, Documentation Link 3. Portal reviews capacity & calls Admin API to manage tenant ○ Jira ticket for Pulsar operator if needed 10 Self-Service Tenant Management
- 11. 11 IO Isolation Writer Reader Journal Data File Data Device Journal Device Write Reads (cold)
- 12. Storage Quota ● Tenant-controlled ● Namespace-level and Topic-level ● Storage Limit ● Policy Throttling ● Pulsar Administrator-controlled ● Namespace-level ● Publish Rate (broker) ● Dispatch Rate ● Replicator Dispatch Rate ● Max ○ Producers ○ Subscriptions ○ Consumers ○ Unacked Messages 12 Quota & Throttling
- 13. Broker Isolation ● Regex of Namespaces to Regex of Brokers/IP Range ● Primary and Secondary broker Regexes 13 Broker Isolation Why ● High Profile/Reserved capacity ● Misbehaving tenants ● Debugging bin/pulsar-admin ns-isolation-policy set --auto-failover-policy-type min_available --auto-failover-policy-params min_limit=5,usage_threshold=80 --namespaces ‘my-tenant/.*’ --primary ‘broker-mytenant[0-9]+.mydomain’ --secondary ‘spare[0-9]+.mydomain’ my-cluster policy-name
- 14. Bookie Isolation ● Bookies to “Affinity Group” ● Namespace(s) to Primary/Secondary Affinity Group ● Rack-Aware within group 14 Bookie Isolation Why ● SLA ● High Profile/Reserved capacity bin/pulsar-admin bookies set-bookie-rack -b 1.1.1.1:3181 -g group-bookie1 --hostname bookie1.mydomain -r /default-rack ... bin/pulsar-admin namespaces set-bookie-affinity-group my-tenant/my-namespace1 --primary-group group-bookie1
- 15. ● Common unit of failure for multiple brokers 15 Failure Domain bin/pulsar-admin clusters create-failure-domain cluster-name --domain-name domain-1 --broker-list broker-1,broker-2 Broker-1 Broker-2 Domain-1 Broker-3 Broker-4 Domain-2 Namespace-1 Namespace-2 Namespace-3 Namespace-4 1 2 3 4 Loadbalancer: Namespace assignment sequence Anti-affinity-namespaces: “Namespace-X”
- 16. ● Assign Namespaces to Anti-Affinity Group ● Changes Load Balancer Behavior 16 Anti-affinity group bin/pulsar-admin namespaces set-anti-affinity-group tenant/namespace1 --group tenant-aag-a bin/pulsar-admin namespaces set-anti-affinity-group tenant/namespace2 --group tenant-aag-a Broker-1 Broker-2 Domain-1 Broker-3 Broker-4 Domain-2 Namespace-1 Namespace-2 Namespace-3 Namespace-4 1 2 3 4 Loadbalancer: Namespace assignment sequence Anti-affinity-namespaces: “Namespace-X”
- 17. Security 17
- 18. ● Authentication ○ TLS Authentication ○ Athenz ○ Kerberos ○ JSON Web Token Authentication ○ Pluggable authentication provider ● Authorization ○ Pluggable authorization provider ○ Default authorization provider on metadata service 18 Authentication & Authorization
- 19. 19 Encryption over the wire PulsarClient client = PulsarClient.builder() .serviceUrl("pulsar+ssl://pulsar-broler:6651/") .tlsTrustCertsFilePath("/ca.cert.pem") .authentication(AUTH, "tlsCertFile:/cert.pem,"+"tlsKeyFile:/key.pem") .enableTlsHostnameVerification(true) .build();
- 20. Producer creation Producer producer = pulsarClient.newProducer() .topic( "persistent://my-tenant/my-ns/my-topic" ) .addEncryptionKey("myappkey") .cryptoKeyReader(new MyCryptoKeyReader()) .create(); 20 Encryption at rest Consumer creation Consumer consumer = pulsarClient.newConsumer() .topic( "persistent://my-tenant/my-ns/my-topic" ) .subscriptionName( "my-subscriber-name" ) .cryptoKeyReader(new MyCryptoKeyReader()) .subscribe();
- 21. ● Proxy for hybrid could application ● Gateway in a cloud environment or on Kubernetes 21 Pulsar Proxy: Public cloud access Proxy Configuration brokerServiceURLTls=pulsar+ssl://brokers.example.com:6651 brokerWebServiceURLTls=https://brokers.example.com:8443
- 22. ● Proxy server creates a TLS tunnel between remote client and server ● The goal is to enable external clients to connect to internal services and do their own client certificate verification, possibly because distribution of private keys to the edge Traffic Server instances is too difficult or too risky. 22 Support Layer-4 SNI Routing
- 23. 23 Pulsar client: SNI Routing PulsarClient client = PulsarClient.builder() .serviceUrl("pulsar+ssl://pulsar-broker:6651/") .enableTls(true).tlsTrustCertsFilePath("/ca.cert.pem") .proxyServiceUrl(proxyUrl, ProxyProtocol.SNI) .authentication(AUTH, "tlsCertFile:/cert.pem,"+"tlsKeyFile:/key.pem") .build();
- 24. 24 Cross Organization geo-replication pulsar-admin clusters create orgB-cluster --broker-url-secure pulsar+ssl:// orgB-broker-vip:6651 --proxy-protocol SNI --proxy-url pulsar+ssl:// orgA-proxy:443 pulsar-admin clusters create orgA-cluster --broker-url-secure pulsar+ssl:// orgA-broker-vip:6651 --proxy-protocol SNI --proxy-url pulsar+ssl:// orgB-proxy:443 For more info: PIP-60: https://github.com/apache/pulsar/wiki/PIP-60%3A-Support-Proxy-server-with-SNI-routing
- 25. Future Roadmap ● Tenant based broker virtualization ○ Container based brokers on BookKeeper service ● Hybrid cloud deployment with geo-replication 25
- 26. Questions? 26
- 27. Thank you Rajan Dhabalia rdhabalia@verizonmedia.com Ludwig Pummer ludwig@verizonmedia.com