Orchestrated - multi tenant architecture at scale with serverless
0 likes288 views
The document discusses key challenges in organizational change, specifically related to managing projects, people, and budgets, which can hinder agility initiatives. It presents a case study on building a multi-tenant SaaS application using AWS tools and serverless technologies, highlighting architectural considerations and security measures for ensuring tenant isolation and efficient resource management. The need for a balance between simplicity and cost-effective operational strategies is emphasized alongside various implementation details, including event sourcing patterns and AWS-specific infrastructure setups.
Orchestrated - multi tenant architecture at scale with serverless
- 4. ● Based on recently published research and our observations in our work, the 3 factors that undo or stall large org change/transformation efforts is how projects (or products), people and budgets are handled. ● Existing ways of managing projects, people & budgets can be called the organisation’s antibodies to change. they’re established and known processes that tend to “snap back” on agility initiatives if they’re not also included ● Our tools help make these factors more visible and comprehensible at staggering scales, particularly when responsibility for these 3 pillars usually rests under different executives
- 6. - there are a lot of concerns to be congnisant of when working with an enterprises’ key data - only ask for the minimum needed information to provide value - sensibly balance agility and innovation speed, with safeguards
- 7. - we like single tenant because it’s simple to build, easy to reason about, strong isolation - we like multi-tenant infrastructures because they’re a lot less expensive per customer to operate. Can be significantly more complex to implement tenant isolation safeguards when building, testing & operating
- 8. - is there a way to get the best of both worlds between single-tenant architecture’s isolation and simplicity, and multi-tenant’s operational cost benefits?
- 9. - AWS account-level isolation: Need to know which AWS account used for which tenant. But awesome in cost insights, Blast radius of security is minimum.
- 10. AWS VPC network-level isolation: Need to know which VPC and CIDR block used per tenant.Cost insights depends how you tag, Blast radius is how you manage Routing. Limitation of 5 per Region.
- 11. - AWS subnet-level isolation: Need to know which Subnet and CIDR block used per tenant.Cost insights depends how you tag, Blast radius is how you manage Routing and access control within subnet. - have to be very careful of tracking subnets & IP address allocation; mistakes in automation or infra changes can affect multiple customers
- 12. - AWS multi-tenant isolation using containers: Beautiful in terms of separation and cluster management. Get better insights and control through ALB
- 13. - we found that deploying an application using serverless technologies allowed us to have the single-tenant isolation benefits, while gaining most of the multi- tenant cost benefits - in practice: serverless + containers for services that can’t be deployed serverless + AWS services (eg DynamoDB, Kinesis, API Gateway, … a long list, read on)
- 14. - Building a Saas application that could be trusted by customers isn’t an easy task, and it often underestimated by product development organisations - A lot of learning and experiments had been done so far as we couldn’t find any magic solutions / framework out there that could be applied to our application straight away - working our way iteratively. “Crossing the river by feeling the stones” (Deng Xiaoping) describes our incremental journey of discovery
- 15. 1. Our product development started from a prototype that we could use to verify our idea to potential customers. 2. diagram: our prototype contains a react client app and a graphql backend services built based around event-sourcing and CQRS patterns (both event sourcing & CQRS are significant topics in themselves. We won’t be covering details here, please ask us if you’d like to know more 3. We start looking for solution to turn this prototype and evolved it to a production ready Saas application after getting stakeholder buy-in
- 16. - This is the high level architecture diagram - Components in the box are deployed per tenant, and the items on the left hand side are for sharing - The following slides will break this down into parts
- 17. Out first target is the graphQL service and the query resolver. - Migrate from APOLLO graphql server (https://www.apollographql.com/) to a lambda function and api gateway - We separate the graphql and query resolver to an individual function for single responsibility concern, and we placed an instance of the back-end neo4j database in a ecs cluster for each tenant. why do we deploy the same lambda functions to each tenant? These are couple of main reasons: 1. allows us to assign fine-grained AWS IAM access control policies to each lambda execution role 2. allows rolling upgrades of tenants, which is more straightforward to implement and less impact to customers than making a large scale multi- tenant upgrade
- 18. This implementation is a relatively complex compare to others, and relies on knowledge of event sourcing which isn’t the focus of this talk. Please get in touch if you’d like to know more. For the event sourcing mutation resolver, we found a more efficient implementation than replacing all prototype components with their equivalent AWS provided product With this approach, we can guarantee that events get persisted before being dispatched: - A command send from graphql will be handled by the command handler function, which will append the output events into a dynamodb table. - The change will be published to dynamoDB stream once the records persisted. - The viewBuilder lambda function that consumes the event stream will be triggered to process the event and eventually update the graph database
- 19. - with the backend changes nearly complete, worked on the front end and client - we used AWS Cognito to take care our authentication and authorization; it has good extensibility to delegate these tasks to a client’s own systems (eg their Active Directory or other SAML provider)
- 20. This setup assigns a given user -> to a Cognito group -> which is tied to a tenant and that tenant’s environment - animation 1: We create a cognito user group for each tenant and map it to an IAM role that allow user access to their tenant resources only - animation 2: When user login to our app, Cognito user pool returns an auth token after a successful authentication - animation 3: This token use to exchange a temporary AWS credentials from cognito identity pool, and this to sign the request sending to AWS. - animation 4: AWS only accept request signed with a valid credential, and - animation 5: request will be rejected if the target resource not specified in the IAM role policy.
- 21. the client front-end single page application’s artifact is deployed to an S3 origin, which then uses AWS Cloudfront CDN for distribution
- 22. For deployment automation, we used Terraform and the serverless framework. - performance: it currently takes 10-15 minutes to provision a new tenant environment - because the cost of a dormant environment is minimal, we can reduce this time by having a sensibly sized pool of pre-provisioned environments ready to service new sign-ups
- 23. Now, the security discussion - a first class citizen in our thoughts
- 24. Find a sensible solution to balance: - Confidentiality, Keep it secret - Integrity, ensure authenticity, avoid tampering - Availability, make it accessible
- 27. Example: command handler Lambda function - IAM Policy: start with the Allow Action, resource any, then see how we lock down - The command handler’s job is to write events to a DynamoDB table as part of event sourcing. Based on this definition, if you have a look at the policy it only allow to PutItem to DynamoDB which implements the principle of the least privilege. - This policy itself will be attached to IAM role which will be assumed by the Lambda function Again, automation is our friend here - allows us to minimise human error in our pursuit of better security. We used the serverless.com framework’s automation capabilities
- 28. Multiple tenants, each with their own copy of the stack. Provides strong data separation. - access is controlled between tenants at the front end, consistent use of IAM rules + least privilege to constrain access down to the specific table resource - each tenant has access toa specific table in their own space or stack - example: with these policy based protections, we get a second layer of protection inhibiting a command handler in tenant 1 from accessing a dynamoDB table that belongs to tenant 2
- 29. - wend end up with each tenant having it’s own set of lambda functions - each function has its own role - apart from the lambda roles, we can then continue to apply the principle of least privilege to: users, groups, firewall rules, security groups, and network access control lists (NACLs)
- 30. - defence in depth: never just rely on one layer of security - these days we’d extend this saying to “implement security beyond the perimeter”, i.e. don’t just trust the outer firewall(s) - AWS products and their consistent feature set make it straightforward to achieve this
- 31. We use neo4j as our back-end graph database - when prototyping we used the graphene.com hosted service. This won’t suit our target customers, completely outside our control + observability, and Graphene makes no representations that their product is fit for our data use case - per tenant: we deployed neo4j to a container, hosted in the Elastic Container Service (ECS), inside a VPC - we can apply a suite of controls to VPCs: security groups, private subnets, and Network access control lists (NACLs)
- 33. - Minimise the attack surface: the fewer interfaces or exposed systems you have, the better - when using lambda functions, the traffic is routed over the Internet within AWS - in order to minimise the attack surface, we work to avoid unnecessary internet traffic Specific examples follow
- 34. scenario 1: Query resolver tries to access the graph database in the VPC - the function’s traffic is routed through the internet to access the private VPC - how can this be improved?
- 35. Improve by using a Lambda VPC - setup a virtual private interface (ENI) when provisioning the Lambda VPC - this ENI is attached to the Lambda function, and connects it to a private subnet - traffic can then be routed through AWS’ internal network instead of the Internet - traffic in/out is then controlled by the Security Group
- 36. We can extend this lambda-VPC-ENI-SecurityGroup access control pattern to each tenant
- 37. scenario 2: By default, traffic from the lambda function to the DynamoDB table is also routed to the Internet - again, we solve this problem by routing the traffic through a VPC details follow.
- 38. 1. setup a VPC endpoint 2. setup the Lambda VPC 3. route traffic through VPCs, which is within AWS’ network end to end The result is a great reduction of the attack surface by having no open interfaces and traffic routed privately.
- 39. scenario 3: Apply the same lambda-VPC-ENI-SecurityGroup pattern to other resources, eg - S3 - Kinesis streams - ...
- 41. Encryption: - we split this challenge to encryption at rest, and encryption in transit - for encryption at rest: we use a Master key generated by KMS to encrypt all storage
- 42. Encryption at rest in a multi-tenant environment: - Each tenant has its own master key, and uses its master key for data encryption & decryption - we do this for all components except DynamoDB, which doesn’t support customer master keys; DynamoDB encryption is still managed by AWS - there’s a bonus slide at the end of this deck which adds more detail to the CMK support landscape
- 43. Encryption in Transit: - We use transport layer security, with certificates provided by AWS’ Certificate Manager (ACM) - Each Lambda function uses AWS’ SDK, so the request will be automatically signed by AWS