Private Apps in the Public Cloud - DevConTLV March 2016
Download as PPTX, PDF1 like660 views
The document introduces AppCloud, a platform designed for application discovery and user engagement within cloud environments, emphasizing zero-configuration support and security. It discusses architectural design considerations for single and multi-tenant environments, utilizing AWS features like VPCs and security groups. The document also outlines a provisioning service that securely manages access and storage for applications across multiple instances, leveraging technologies like Consul and Vault.
Private Apps in the Public Cloud - DevConTLV March 2016
- 1. Principal Architect Private Apps in the Public Cloud Issac Goldstand
- 3. Introducing AppCloud Out-of-the-Box Experience Users discover applications as they set up a new PC or smartphone App Discovery Engine Users browse for software in a curated catalog
- 4. Introducing AppCloud Dynamic Notifications Re-engaging users when it makes sense Analytics Understanding users.
- 5. Behind the scenes... Introducing AppCloud Sponsored AppsPopular Apps AppCloud Catalog * Popular Apps - Free apps that users are likely to install on their device * Sponsored Apps - Apps with campaigns that can generate revenue Mix of popular and sponsored apps App Personalization Engine Sponsored App
- 10. 2 Major Concerns
- 11. “The more possessions one owns, the more worries one needs to deal with” “ מרבהנכסים , מרבהדאגות ”
- 14. What happens if an attacker breaches one of the servers?
- 17. We need to partition our environments properly
- 18. Two types of environments
- 19. Two types of environments Single Tenant Multiple Tenant
- 21. In a modern cloud, where most hardware is multi-tenant by definition, how can we accomplish single tenant partitioning?
- 23. Hardware Layer
- 24. TL; DR We use dedicated compute instance per component/environment (customer)
- 25. Networking Layer
- 26. Network How we used to do it? https://commons.wikimedia.org/wiki/File:3_men_working_on_a_portable_phone_switchboard.jpg
- 27. Each customer gets their own (set of) VLAN(s)
- 28. No interconnectivity between customer VLANs
- 29. Additional VLAN(s) for shared components
- 30. AWS
- 31. AWS VLAN == VPC
- 32. AWS VLAN == VPC (loosely speaking)
- 33. Network Layer Split each customer to their own VPC
- 35. Network Layer AWS Jump Rules Target is another (or the same) security group
- 36. Network Layer Each customer/component pair has a security group (at least one) Allows fine-grained control of which services can access different sets of data Note the separation of S3 buckets + use of IAM roles to access the S3 data
- 37. Each customer/component pair has a security group (at least one)
- 38. Allows fine-grained control of which services can access different sets of data
- 39. In addition to security, S3 replication allows for cross-region deployments https://github.com/issacg/s3sync
- 41. Application Layer Example Workflow 1) Back-end sends app to “Publish App” microservice 1) “Publish” microservice stores data in S3 storage 1) “Publish” microservice calls “Parse App Metadata” and “Sign App” microservices 1) “Publish” microservice saves metadata + signature to database Future slides will use PoV of “Publish App” microservice
- 42. “Elastic applications in a public cloud should support zero- configuration”
- 43. “Elastic applications in a public cloud should support zero- configuration” - Me
- 44. Zero configuration allows us to support both auto-scaling groups and auto-healing in case of (many, but not all) problems
- 45. Zero-Configuration Networking Service Discovery Credentials/Identity Management Application + Config
- 46. Networking
- 48. Options
- 49. Option 1 AWS Internal ELB Route53 Private Hosted Zones
- 50. Service Discovery Route53 Private Zone points to Internal ELB ELB load balances traffic between Publish workers If a publish worker fails the ELB health check, it is removed from the pool of healthy workers
- 51. Option 2 Standalone service discovery ZooKeeper, Consul, etc
- 52. Consul
- 53. Rich feature-set built-in Service discovery KV storage Global mutex/semaphores Leader election High availability (active/active) Encryption (Gossip + HTTP/RPC) Health checks
- 54. Incredibly elastic Fits the cloud well
- 55. Consul Register current instance as “publish- i1234567890abcdefa.node. customerA” and “publish.service.customerA” + healthchecks Addresses of well-known “parseapp” and “signapp” services via service discovery Name of S3 bucket + path via KV storage S3 access via IAM role Database (host, user, password) via KV storage
- 56. Zero-Configuration via Consul Application Layer Server comes up via configuration management scripts Server joins consul cluster Server fetches application configuration based on well-known locations in the consul KV store Server fetches application bits and boots Service registers with consul (including healthcheck)
- 57. Instead of looking up an ELB with a well-known hostname, we can use a well-known service name and connect to any machine inside that service group
- 58. Service Discovery Consul DNS lookup for “publish” service Consul randomly picks a healthy instance and returns the address of the worker If a publish worker fails the consul health check, it is removed from the pool of healthy workers If we have a leader/follower app, we can use consul “tags” to get a specific instance (eg. master.publish.service…..consul)
- 59. Application Layer Service Discovery redis.service.nyc3.consul / redis.service.consul
- 60. Application Layer https://www.flickr.com/photos/cogdog/566323330 Service Discovery
- 63. Two gossip pools - WAN & LAN
- 64. LAN pools encapsulate a single (virtual) datacenter Divided into server and client agents
- 65. In each DC, a single server is elected as “Leader”
- 66. Transactions are forwarded and committed to all servers
- 67. Leader is responsible for maintaining consistency in its DC
- 68. WAN pool spans all datacenters (servers only)
- 69. Cross-datacenter requests use RPC-forwarding (between server nodes) to query the remote DC
- 70. No DC stores information about other DCs
- 71. Rich ACL system Who can access what?
- 73. We already split everything into VPCs
- 74. Each VPC becomes a DC in consul
- 75. Each environment (customer) automatically gets a private KV store and private service registry
- 76. Shared services live in their own well-known dedicated DC with their own “shared” KV store & service registry
- 78. It’s possible to perform cross- datacenter queries Controllable via ACLs
- 79. Application Layer Publish asks Consul Blue - who is local signapp? signapp.service.consul Consul Blue answers with gossipped address of random signapp instance in healthy state
- 80. Application Layer Publish asks Consul Blue - who is tagged as leader node of parse in green DC? leader.parse.service.green.consul Consul Blue checks the WAN gossipped peers for a server address of Consul Green cluster Consul Blue forwards the query (via HTTP/S) to Consul Green over the WAN Consul Green answers with gossipped address of a parse node with the tag “leader” and in healthy state
- 81. Is that good enough? Probably
- 85. Secure storage and audit control of private data
- 87. Growing ecosystem of backends supporting one-time-passwords AWS-STS, MySQL, PostgreSQL, SSH, PKI, Consul
- 89. Consul + Vault Consul + Vault access via Vault (via Provisioning Service) Addresses of well-known “parseapp” and “signapp” services via consul service discovery Name of S3 bucket + path via KV storage (access via IAM Role*) * Could also use Vault AWS backend Database host via consul KV storage Database user, password via Vault Register current instance for consul service discovery
- 90. Zero-Configuration via Consul & Vault Networking Service Discovery Credentials/Identity Management Application + Config Application Layer Server comes up via configuration management scripts Server identifies itself to Vault-backed provisioning service and gets consul SSL keypair + Consul access token + Vault access token for future queries Server joins encrypted Consul cluster Server fetches application bits and boots Service fetches application configuration from Vault secret backend and Consul KV store Service registers with consul (including healthcheck)
- 91. The Challenge
- 93. How do you bootstrap access for a single image running in multiple instances (eg, an AMI in an auto- scaling group)?
- 94. We want to audit each machine’s access individually - no shared authentication
- 95. We don’t want to allow multiple machines (or “anything”s) to authenticate the same token twice
- 96. We don’t want to store secrets in a non-secret place
- 97. Many suggestions for inclusion as an authentication plugin for Vault
- 98. I haven’t seen any I like
- 99. Let Vault focus on protecting the data, extend it with external tooling to fit your needs
- 100. Provisioning Service
- 101. Amazon EC2 Instance Identity Document http://169.254.169.254/latest/dynamic/instance-identity/
- 102. Includes embedded cryptographic signature to authenticate the document
- 103. Signed by AWS
- 104. Provisioning Service AWS EC2 Identity Document Application Layer Subset of fields in the Identity Document ● AWS Account Number ● Instance ID ● Instance Primary Private IP address ● AMI + Kernel ID ● Launch request time
- 107. We currently store those in the EC2 user-data to be processed by our configuration management system
- 108. After authenticating the instance the provisioning service queries EC2 to obtain the user-data It’s flexible to harden this later
- 109. Provisioning Service Instance sends its identity document to Provisioning Service (PrvSrv) PrvSrv authenticates the AWS signature PrvSrv verifies that IP making the request matches the IP in the doc PrvSrv verifies that the AWS account, AMI are whitelisted PrvSrv uses the instance id to query the EC2 API to fetch additional metadata Using this metadata, PrvSrv requests/generates credentials for Vault + Consul and returns this info to the instance PrvSrv sends additional Vault token to bootstrap Consul
- 110. EC2 Instance IDs are globally unique across accounts and are never recycled
- 111. Provisioning service will only provide a single token for an instance
- 112. Instances are guaranteed to be coming from inside our AWS accounts, and from a verified IP address
- 114. TLS keys for the node (PKI backend) https://github.com/issacg/vault-pki-client
- 115. Current gossip shared-key (Generic backend)
- 116. Token for consul (Consul backend)
- 117. Provides vault token for application (no backend)
- 118. No built-in Vault backend for Vault
- 119. No built-in Vault backend for Vault Not an unsolvable problem - provisioning service can take care of this
- 120. Future Plans & Challenges
- 121. Separate Vault per environment (eg, Vault per Consul DC)
- 122. How do we manage unsealing with so many Vault clusters?
- 123. How do we pass the secrets from the provisioning service client to the application service in a secure manner?
- 124. How do we need to change the provisioning service to run with containers?
- 125. Summing Things Up
- 126. In a modern cloud, where most hardware is multi-tenant by definition, how can we accomplish single tenant partitioning?
- 127. Partitioning Hardware (Logical) Layer Networking (Logical) Layer Application Layer
- 128. Hardware (Logical) Layer Instance per-service per- environment
- 129. Networking (Logical) Layer VPC per environment Security Group per component/environment compute-instance
- 130. Application Layer Separation of shared / private microservices Consul + Vault + Provisioning service to provide partitioned zero-configuration
- 131. Questions?
- 132. Thank you Principal Architect Issac Goldstand issac@ironsrc.com
Editor's Notes
- #2: Sample Text
- #3: When you get a new phone, you probably get 4 navigation apps. You get Orange GPS, Google Maps, Glympse (whatever that is) but all you really wanted is Waze. The apps are burned 6 months before you ever get your phone, so they’re all way out of date too. AppCloud aims to solve this by allowing users to choose what they want to install when they set up their device for the first time.
- #4: Benefits to user: ability to choose what they want to install Benefits to advertiser: more targeted users Benefits to OEM/Mobile Network: higher fees from advertiser + trust relationship with the user
- #11: We’ll focus on 2 concerns that came up
- #16: Even if we’ve segmented environments to dedicated instances, if they all live in a single security group, then an attacker who breaches the backend server belonging to Customer A could use that to hijack the database of Customer B.
- #17: Our enterprise customers were specifically worried about a scenario where they could be attacked based on a flaw present in a different customer’s environment (possibly due to special requirements by another customer)
- #25: This can be a presentation unto itself. For brevity, let’s summarize the conclusion
- #27: Let’s try to model the networking considerations on a classic networking design pattern before deciding how to do it in the cloud
- #36: Extensive use of jump rules. Instead of specifying a IP or CIDR block, we can specify another security group as the rule target
- #45: Design pattern called immutable servers
- #47: Our cloud provider will deal with this
- #50: I didn’t know about private hosted zones at the time...
- #56: What information does the Publish App microservice need from Consul?
- #59: How does a service discovery request for the publish microservice happen with Consul?
- #60: This is what it looks like inside Consul’s UI. This example is from the Demo UI, and not from the AppCloud consul UI.
- #62: (If we have time)
- #63: Two gossip pools - WAN & LAN LAN pools encapsulate a single (virtual) datacenter Divided into server and client agents In each DC, a single server is elected as “Leader” Transactions are forwarded and committed to all servers Leader is responsible for maintaining consistency in its DC WAN pool spans all datacenters (servers only) Cross-datacenter requests use RPC-forwarding (between server nodes) to query the remote DC No DC stores information about other DCs Rich ACL system Who can access what?
- #71: Except for server information which is gossipped in the WAN pool
- #73: How do we use consul?
- #80: How do local and cross datacenter queries work?
- #83: But my perfectionistic nature got the best out of me...
- #84: ...and a great looking tool existed to make things better - (unusually) I let myself fall for shiny object syndrome - and decided to use it
- #97: Git, S3, chef (excepting encrypted data bags), EC2 user-data, etc are considered non-secret
- #105: VPC ID would be nice...
- #106: Could be tightly coupled to the AMI, but then you’d need to secure the AMI registry and coupling to vault roles
- #107: Reference is to the AppCloud environment/customer identifier; not AWS. If we partition by AWS accounts, then we can get this in the instance-data, but there’s no real advantage to us in doing so; breach of the AWS account will always be an attack vector to fool the provisioning service
- #123: Possibly use semi-automated unsealing process using phone push
- #124: We could use envconsul, but the same question would apply: how to get the VAULT_TOKEN in the proper environment when sent to envconsul - also envconsul, although marginally better than consul-template, still won’t let us rotate OTPs inside the application
- #125: The provisioning service currently uses the instance ID as a unique index - how will that apply to multiple containers running on a single instance?