Scaling the Cloud - Cloud Security

Editor's Notes

• #2: Why did Netflix migrate to the public Cloud?\nWhich InfoSec controls were harder or easier in the Cloud?\nWhat’s left to solve?\n\nRunning in a public cloud is less about virtualization and more about disrupting how you currently deliver services. Here’s the infosec lens on how Netflix is migrating to the Cloud.\n\nAugments Jason Chan’s “Practical Cloud Security” presentations.\n
• #3: I won’t spend a lot of time on background, but it’s important to cover the context so that you understand why we’re doing this.\n\nIn two years we went from “traditional IT” to running one of the largest public cloud infrastructures on Amazon.\n\nWhen I briefed the DoD CyberSecurity Task Force, they were shocked at the rate of our innovation. I thought 2 years was a long time; but they helped put things into perspective.\n\nThese ideas may seem strange to you. But you probably have teams doing this already, or are trying to achieve this, or you will acquire a company that does this now. I assert that many of these design and operations ideas will be the norm for new companies in less than 5 years.\n\n
• #4: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #5: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #6: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #7: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #8: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #9: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #10: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #11: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #12: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #13: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #14: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #15: (doubled subscribers in 2010, moved to cloud)\n3+ billion rev in 2011, S&P 500\n\nQ: Soon every TV sold anywhere in the world will have WiFi and Netflix built in \n\n
• #16: We’re dev-focused so it was OK for us to build our own.\nDidn’t need to wait for industry to build shims and orchestration tools.\nAlso weren’t multi-CSP concerned at this point, YMMV. We’re also not in a regulated industry, so again YMMV.\n
• #17: In other words: the Cloud is not a technology, it’s more than virtualization. It’s a fundamentally different way of thinking about writing applications, providing computing services, and running your business.\n
• #18: In other words: the Cloud is not a technology, it’s more than virtualization. It’s a fundamentally different way of thinking about writing applications, providing computing services, and running your business.\n
• #19: (No Central architecture review boards, etc)\n(eliminated unnecessary complexity)\nLoosely-coupled, highly-aligned teams\nResponsible people thrive on, are worthy of freedom\nIncrease freedom as we grow, rather than limit it\nNetflix loves killing unnecessary processes\n
• #20: (No Central architecture review boards, etc)\n(eliminated unnecessary complexity)\nLoosely-coupled, highly-aligned teams\nResponsible people thrive on, are worthy of freedom\nIncrease freedom as we grow, rather than limit it\nNetflix loves killing unnecessary processes\n
• #21: (No Central architecture review boards, etc)\n(eliminated unnecessary complexity)\nLoosely-coupled, highly-aligned teams\nResponsible people thrive on, are worthy of freedom\nIncrease freedom as we grow, rather than limit it\nNetflix loves killing unnecessary processes\n
• #22: \n
• #23: \n
• #24: \n
• #25: \n
• #26: (doubled subscribers in 2010, moved to cloud)\nExample: Superbowl, Christmas scaling\n\nScale up early, scale down slowly\nprovision for AZ capacity\n\nWe now kill and respawn more Cloud servers every week than we have in our datacenter. It’s approaching a daily rate.\n
• #27: (doubled subscribers in 2010, moved to cloud)\nExample: Superbowl, Christmas scaling\n\nScale up early, scale down slowly\nprovision for AZ capacity\n\nWe now kill and respawn more Cloud servers every week than we have in our datacenter. It’s approaching a daily rate.\n
• #28: (doubled subscribers in 2010, moved to cloud)\nExample: Superbowl, Christmas scaling\n\nScale up early, scale down slowly\nprovision for AZ capacity\n\nWe now kill and respawn more Cloud servers every week than we have in our datacenter. It’s approaching a daily rate.\n
• #29: (doubled subscribers in 2010, moved to cloud)\nExample: Superbowl, Christmas scaling\n\nScale up early, scale down slowly\nprovision for AZ capacity\n\nWe now kill and respawn more Cloud servers every week than we have in our datacenter. It’s approaching a daily rate.\n
• #30: (doubled subscribers in 2010, moved to cloud)\nExample: Superbowl, Christmas scaling\n\nScale up early, scale down slowly\nprovision for AZ capacity\n\nWe now kill and respawn more Cloud servers every week than we have in our datacenter. It’s approaching a daily rate.\n
• #31: (doubled subscribers in 2010, moved to cloud)\nExample: Superbowl, Christmas scaling\n\nScale up early, scale down slowly\nprovision for AZ capacity\n\nWe now kill and respawn more Cloud servers every week than we have in our datacenter. It’s approaching a daily rate.\n
• #32: \n
• #33: Goal: Assume Man In The Middle\nCountermeasures / Mindset:\nEnd-to-end encryption\nMutual authentication\nEncrypt storage\nFBI warning\n
• #34: Countermeasures / Mindset:\nSegment key management from data usage\nSegment build / run environment\nTest for conformance, integrity\n
• #35: ASG for everything\nAWS fleet-wide patch\nApril 2011 outage of a single AZ in US-EAST\n\nCountermeasures / Mindset:\nNever depend on “one” of anything (host, AZ, etc)\nStateless design in running instances\nTest for conformity, alert on non-conformity\n
• #36: You can’t protect Software with More Software\nCountermeasures / Mindset:\nStrong key management\nSeparation of keys, data\nHardware key management\n
• #37: Hard\nCountermeasures / Mindset:\nSLA, CSP in your Incident Response plan, TEST!\nRely on your other CIAp controls\n
• #38: Hard\nCountermeasures / Mindset:\nSLA, CSP in your Incident Response plan, TEST!\nRely on your other CIAp controls\n
• #39: Hard\nCountermeasures / Mindset:\nSLA, CSP in your Incident Response plan, TEST!\nRely on your other CIAp controls\n
• #40: Hard\nCountermeasures / Mindset:\nSLA, CSP in your Incident Response plan, TEST!\nRely on your other CIAp controls\n
• #41: Some lessons learned, some aspirational and in-motion\nIt’s hard work to move your systems, processes, and staff into this new environment\nAt times, it’ll feel chaotic..like you’re herding sheep and they’re running every which way\nBut once you learn the vocabulary and understand this technology, you’ll come to appreciate it. It’s actually very enabling and refreshing.\n
• #42: Some lessons learned, some aspirational and in-motion\nIt’s hard work to move your systems, processes, and staff into this new environment\nAt times, it’ll feel chaotic..like you’re herding sheep and they’re running every which way\nBut once you learn the vocabulary and understand this technology, you’ll come to appreciate it. It’s actually very enabling and refreshing.\n
• #43: Some lessons learned, some aspirational and in-motion\nIt’s hard work to move your systems, processes, and staff into this new environment\nAt times, it’ll feel chaotic..like you’re herding sheep and they’re running every which way\nBut once you learn the vocabulary and understand this technology, you’ll come to appreciate it. It’s actually very enabling and refreshing.\n
• #44: Just like learning a new skill, it’s hard at first. Some things are still hard, but we’re working to make them easier. We’ll have some announcements in this space for everyone’s benefit, very exciting.\n
• #45: Just like learning a new skill, it’s hard at first. Some things are still hard, but we’re working to make them easier. We’ll have some announcements in this space for everyone’s benefit, very exciting.\n
• #46: Just like learning a new skill, it’s hard at first. Some things are still hard, but we’re working to make them easier. We’ll have some announcements in this space for everyone’s benefit, very exciting.\n
• #47: Just like learning a new skill, it’s hard at first. Some things are still hard, but we’re working to make them easier. We’ll have some announcements in this space for everyone’s benefit, very exciting.\n
• #48: Just like learning a new skill, it’s hard at first. Some things are still hard, but we’re working to make them easier. We’ll have some announcements in this space for everyone’s benefit, very exciting.\n
• #49: Just like learning a new skill, it’s hard at first. Some things are still hard, but we’re working to make them easier. We’ll have some announcements in this space for everyone’s benefit, very exciting.\n
• #50: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #51: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #52: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #53: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #54: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #55: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #56: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #57: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #58: Here’s a sample of what we’ve found to be easier, in our environment.\nWe’ll discuss some of these in more detail.\n
• #59: Classic IT: uptime was paramount. Rebooting was something you snickered at the Windows guys about.\n\nYou patched, and tweaked, and documented all your changes.\n\nAnd prayed to God that all those fixes and tweaks worked, and the system actually came back up the next time you restarted it.\n\nEvery instance was unique, a special snowflake.\n
• #60: Classic IT: uptime was paramount. Rebooting was something you snickered at the Windows guys about.\n\nYou patched, and tweaked, and documented all your changes.\n\nAnd prayed to God that all those fixes and tweaks worked, and the system actually came back up the next time you restarted it.\n\nEvery instance was unique, a special snowflake.\n
• #61: Classic IT: uptime was paramount. Rebooting was something you snickered at the Windows guys about.\n\nYou patched, and tweaked, and documented all your changes.\n\nAnd prayed to God that all those fixes and tweaks worked, and the system actually came back up the next time you restarted it.\n\nEvery instance was unique, a special snowflake.\n
• #62: We’ve been moving towards automation for a while now. The paradigm was to make adjustments to instances already running. The best models create “gold standard” images and deploy those.\n\nThe goal is to have every instance look exactly the same, run the same, and behave the same.\n\nWe’re taking a hard stance on this. We got here because of agility, but we have many security wins as a result.\n\n
• #63: It sounds heretical. Why patch when you can throw it away and start over?\n
• #64: Why bother fixing the configuration when you can deploy the “right” configuration from the start?\n\nThe same behavior for deploying new, for patching, and for upgrades means operations becomes simpler, easy to do/train/monitor.\n
• #65: Look at problems as opportunities. Rather than mandate 100% uptime from our CSP, we assumed the environment would be unpredictable. This forced us to build reliability into our applications and infrastructure.\n\nIn other words: the Cloud is not a technology, it’s more than virtualization. It’s a fundamentally different way of thinking about writing applications, providing computing services, and running your business.\n
• #66: Look at problems as opportunities. Rather than mandate 100% uptime from our CSP, we assumed the environment would be unpredictable. This forced us to build reliability into our applications and infrastructure.\n\nIn other words: the Cloud is not a technology, it’s more than virtualization. It’s a fundamentally different way of thinking about writing applications, providing computing services, and running your business.\n
• #67: Look at problems as opportunities. Rather than mandate 100% uptime from our CSP, we assumed the environment would be unpredictable. This forced us to build reliability into our applications and infrastructure.\n\nIn other words: the Cloud is not a technology, it’s more than virtualization. It’s a fundamentally different way of thinking about writing applications, providing computing services, and running your business.\n
• #68: Look at problems as opportunities. Rather than mandate 100% uptime from our CSP, we assumed the environment would be unpredictable. This forced us to build reliability into our applications and infrastructure.\n\nIn other words: the Cloud is not a technology, it’s more than virtualization. It’s a fundamentally different way of thinking about writing applications, providing computing services, and running your business.\n
• #69: Look at problems as opportunities. Rather than mandate 100% uptime from our CSP, we assumed the environment would be unpredictable. This forced us to build reliability into our applications and infrastructure.\n\nIn other words: the Cloud is not a technology, it’s more than virtualization. It’s a fundamentally different way of thinking about writing applications, providing computing services, and running your business.\n
• #70: \n
• #71: \n
• #72: Conformity:\nProvisioning: Can easily list every application running, all attributes including owner\nConformity Monkey checks for consistency , detects out-of-spec instances\nInconsistencies create runtime problems, outages, troubleshooting nightmares.\n* Lesson Learned: Identify failure modes, bake these test controls into your infrastructure.\n\nConsistency:\nAutomated software packaging, host build processes\nHands-off launch process; spans hosts, load balancers, security groups, etc.\nInstances are formed into “application groups”\n\n
• #73: Conformity:\nProvisioning: Can easily list every application running, all attributes including owner\nConformity Monkey checks for consistency , detects out-of-spec instances\nInconsistencies create runtime problems, outages, troubleshooting nightmares.\n* Lesson Learned: Identify failure modes, bake these test controls into your infrastructure.\n\nConsistency:\nAutomated software packaging, host build processes\nHands-off launch process; spans hosts, load balancers, security groups, etc.\nInstances are formed into “application groups”\n\n
• #74: Conformity:\nProvisioning: Can easily list every application running, all attributes including owner\nConformity Monkey checks for consistency , detects out-of-spec instances\nInconsistencies create runtime problems, outages, troubleshooting nightmares.\n* Lesson Learned: Identify failure modes, bake these test controls into your infrastructure.\n\nConsistency:\nAutomated software packaging, host build processes\nHands-off launch process; spans hosts, load balancers, security groups, etc.\nInstances are formed into “application groups”\n\n
• #75: All instances have:\n- ASG, SecGrp, ELBs, owners, description, email addr -- almost like a CMDB.\n- everything that doesn’t gets killed by janitor monkey\n-control over my env is straightforward\n\nA few clicks on a web page and about an hour to go from nothing to a very large Cassandra cluster consisting of 288 medium sized instances, with 96 instances in each of three EC2 availability zones in the US-East region.\n\n15 minutes to boot EC2, out of our total of 66 minutes. The rest of the time was taken to boot Linux, start the Apache Tomcat JVM that runs our automation tooling, start the Cassandra JVM and join the "ring" that makes up the Cassandra data store.\n\n For a more typical 12 instance Cassandra cluster the same sequence takes 8 minutes.\n\n
• #76: All instances have:\n- ASG, SecGrp, ELBs, owners, description, email addr -- almost like a CMDB.\n- everything that doesn’t gets killed by janitor monkey\n-control over my env is straightforward\n\nA few clicks on a web page and about an hour to go from nothing to a very large Cassandra cluster consisting of 288 medium sized instances, with 96 instances in each of three EC2 availability zones in the US-East region.\n\n15 minutes to boot EC2, out of our total of 66 minutes. The rest of the time was taken to boot Linux, start the Apache Tomcat JVM that runs our automation tooling, start the Cassandra JVM and join the "ring" that makes up the Cassandra data store.\n\n For a more typical 12 instance Cassandra cluster the same sequence takes 8 minutes.\n\n
• #77: All instances have:\n- ASG, SecGrp, ELBs, owners, description, email addr -- almost like a CMDB.\n- everything that doesn’t gets killed by janitor monkey\n-control over my env is straightforward\n\nA few clicks on a web page and about an hour to go from nothing to a very large Cassandra cluster consisting of 288 medium sized instances, with 96 instances in each of three EC2 availability zones in the US-East region.\n\n15 minutes to boot EC2, out of our total of 66 minutes. The rest of the time was taken to boot Linux, start the Apache Tomcat JVM that runs our automation tooling, start the Cassandra JVM and join the "ring" that makes up the Cassandra data store.\n\n For a more typical 12 instance Cassandra cluster the same sequence takes 8 minutes.\n\n
• #78: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #79: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #80: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #81: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #82: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #83: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #84: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #85: some items are aspirational, but we’re working on it.\n\nThese are similar to NIST’s “continuous monitoring” movement.\n
• #86: \n
• #87: \n
• #88: It’s a completely different way to provide your services. More disruptive than just a new technology.\n Bring your InfoSec expertise and foresight to secure your data in the cloud migration.\n You can embed security controls in your development, infrastructure, business operations.\n The old ways won’t work; embrace the new ones and have better control.\n
• #89: It’s a completely different way to provide your services. More disruptive than just a new technology.\n Bring your InfoSec expertise and foresight to secure your data in the cloud migration.\n You can embed security controls in your development, infrastructure, business operations.\n The old ways won’t work; embrace the new ones and have better control.\n
• #90: It’s a completely different way to provide your services. More disruptive than just a new technology.\n Bring your InfoSec expertise and foresight to secure your data in the cloud migration.\n You can embed security controls in your development, infrastructure, business operations.\n The old ways won’t work; embrace the new ones and have better control.\n
• #91: It’s a completely different way to provide your services. More disruptive than just a new technology.\n Bring your InfoSec expertise and foresight to secure your data in the cloud migration.\n You can embed security controls in your development, infrastructure, business operations.\n The old ways won’t work; embrace the new ones and have better control.\n
• #92: It’s a completely different way to provide your services. More disruptive than just a new technology.\n Bring your InfoSec expertise and foresight to secure your data in the cloud migration.\n You can embed security controls in your development, infrastructure, business operations.\n The old ways won’t work; embrace the new ones and have better control.\n
• #93: It’s a completely different way to provide your services. More disruptive than just a new technology.\n Bring your InfoSec expertise and foresight to secure your data in the cloud migration.\n You can embed security controls in your development, infrastructure, business operations.\n The old ways won’t work; embrace the new ones and have better control.\n
• #94: It’s a completely different way to provide your services. More disruptive than just a new technology.\n Bring your InfoSec expertise and foresight to secure your data in the cloud migration.\n You can embed security controls in your development, infrastructure, business operations.\n The old ways won’t work; embrace the new ones and have better control.\n
• #95: \n
• #96: \n