Real Security in a Virtual Environment

Editor's Notes

• #3: Who am IMe: name, Age, ExperienceNucleus: short recap
• #4: Linux Shell.
• #5: Not in front of large crowds. No where I’m standing now.
• #6: For the reason you’re all here; what is this talk actually about?
• #7: There is no “onedefinition” of cloudcomputing. It’s a hype term, thateveryoneuses to suite hispurpose. Cloudcomputingcan stand for a lot of things.
• #8: Cloudthat provides “infrastructure”. These companiescan provide youwith a virtual machine foryou to use.Soyouget RDP accessforwindows, shellaccessfor Linux. Youwon’t (orhardly) notice the differencewith a physical server, but have the advantage of easy back-ups, scalability (memory/cpuupgrades), remote console, …
• #9: SaaS is the “nohassle” solution of “the cloud”, youpay a monthly fee for the service you want (your email, your CRM or ERP package), and itwillwork. Youdon’t manage any of it, youpayyoursaas-provider to manage thatforyou. They do backups, updates, maintenance, bugfixes, …
• #10: These providers offer you the platform to buildyourown services, a large set of API’s to use to buildyourapplications. They keep the platform online, you provide yourapplications.
• #11: I know,you want hearaboutsecurity. Butit’s important to know the different kinds of “service”. This talk mostlymentionsIaaS.
• #12: In short: it’s about preventing this cloud ...
• #13: In turning into this cloud. Preventing a possible catastrophe. Protecting your (virtual) environment to be as secure as possible.Not just from possible intruders or hackers, but from data loss in general as well.
• #14: So let’s get on with it: how do we do it? How do we provision a virtual infrastructure or “cloud” that is safe and always available?Since there are many different kinds of “clouds”, this aims mostly at the private or public clouds based on systems like Vmware, KVM or Xen. Most of it applies to Amazon’s EC² as well, but that’s not the main focus here.
• #15: No matter how much you talk about it and how much effort you put into it, your system can be hacked. It will always be able to get hacked.
• #16: A virtual infrastructure is an extension to your current physical one, and thus it has more layers than the physical world, more layers that need protecting.
• #17: Very obvious: but the hardware that forms your virtual infrastructure, needs to be stored securely.That means you need access lists to control who can have physicalaccess to your systems. Access logs to monitor. Perhaps even security on-site, camera’s, ...
• #18: We’d all like it, but allas: without power, no servers. And by extent, no cloud. But it’s not just power. You need redundant power circuits, with battery-rooms to cover a power outage, generators on standby.
• #19: Even though not a real “security” problem, it’s vital for you to run your cloud. No cooling equals no running servers.
• #20: Your location, power supply and cooling need to be secure. If someone can physically damage any of it, your cloud (or any IT infrastructure for that matter) is in direct danger.
• #21: The datacenter is something we take for granted. We usually already have it running when we start to think about virtualization.The real challenge is presented within our virtual infrastructure.
• #22: Virtual environments need physical equipment. Don’t forget to secure that. While it gets less focus, it’s increasingly more important.
• #23: If you think about how a cloud or virtual environment is built, there are 2 major components: the hardware powering it, and the software managing the actual ‘virtualization’.; your hypervisor. You can have Dell servers providing your cloud with CPU power, storage and memory, and Vmware providing you with the virtual aspects of your environment. Two entirely different things, each with their vulnerabilities. But needed to give shape to your virtual infrastructure. Let’s have a look at those different kind of layers.
• #24: Storage is the central part to any virtual infrastructure. It needs to be scalable, reliable and safe. The data on each (virtual) server is the actual value of your business. Your database with clients, products or invoices. Your generated images. Your processed CSV files. You name it.
• #25: WhenusinglargeStorageArea Networks, youwillprobablybeconnecting to yourstorage via NFS oriSCSI. Thatmeansstorage is passing a classic network.All traffic going to and from the storage needs to be either on a seperate network, or be logically seperated via VLANs. Make sure your traffic can not be sniffed by other devices on your network besides your storage network. You don’t want Virtual Machines seeing traffic from other VMs.But not like this, because it’s a mess.
• #26: But do it more like this. You need to be able to manage it. And show it off to your clients. ;)
• #27: Making sure no one can read your network traffic to/from storage is one thing. If someone actually manages to get physical hold of your disks, that’s something else. Or what about disks being broken and sent back to the supplier? Can he read your data? Did you do something to make the data unusable?Encryption can be considered in a number of ways. Either directly on your storage (where your storage software can provide it), or within the operating system of the virtual machine by using encrypted partitions such as Truecrypt. Each has their benefits. Each has their specifics.
• #28: If you’re thinking about doing so on your actual storage box, it usually means there’s an extra hardware appliance that needs to sit in front of your current hardware, to act as a “translater” to do the actual encrypting. That introduces an additional cost, because these things don’t come cheap.
• #29: If hardware encryption is not an option, you can look at encrypting some or all partitions within your virtual machine itself.There are limitations to this, especially if you’re using things like dynamic disks on Windows or when you’re running older Windows in general. If there’s ever the need to upgrade your OS, that’ll cause you headaches when the partition is encrypted. Or doing a system repair with a live boot CD? Encrypting/decrypting is hardly supported there.The encrypting/decrypting causes an additional load to your hypervisor as well, as that CPU will be tasked with doing so.
• #30: The moment you talk about encryption, the next logical burden to overcome is key management. Since they form the entry ticket to your storage, as a means of decrypting them, they arguably become the most important factor in your encrypted storage.How do you secure your keys? Do you often change them, and via what procedure? Does it mean re-encrypting all your current data to the new keys? How long would that take? Can you do it while in production? These are just some of the questions you need to ask yourself, and that force you into some kind of procedure to manage it.
• #31: Having secured storage is one thing. Surviving entire storage or datacenter failure means you need to replicate your storage to another (physical) location. If site A would get hit by an earthquake, and you loose it, you still have site B with the replicated storage box. All goodthingscome in pairs.
• #32: Surviving a disk failure means being prepared via RAID sets. Find the trade between performance, disk capacity and disk failures.While striping all your data will give you killer performance, just one disk failure means you’ll have a killer boss blaiming you for data loss. Looking at raid 50 or raid 60 means losing a lot of disk capacity, at the trade of having more reliable storage. Having spare disks add nothings to your capacity, but means you can relax knowing your raid will automatically rebuild. For you environment, look at your requirements. See what kind of read/write ratio’s you have on your storage, and consider making “fast” volumes but less secure, and slower but more protected volumes.
• #33: Make sure to have good and reliable back-ups, with a fast time-to-recovery. Not only from your entire storage system (say replication?), but from within each individual server, to have file-level and database-level back-ups. Having a full-blown site-recovery plan is great, but it sucks when you need to restore a single file or database.Make sure that is encrypted. A back-up is worth just as much as a production system. If someone tries to get hold of your data, and finds you go beyond limits to secure your Storage Area Network, but leave your back-ups just as plain-text on a not-so-secure off-site location, he’ll go after that. After all, the back-up contains the data you thought was important enough to take a back-up from. You’ve already determined for your attacker, that data is the most important for you. He doesn’t have to decypher that anymore. You’ve done it for him.That’s it for storage (-related) items.
• #34: Building a cloud or ‘going virtual’ means placing several virtual machines onto one or more physical servers. Those need bandwidth to communicate with other systems.
• #35: That means protecting them. The old-skool way is placing firewalls and limiting inter-VM traffic. That’s still valid, but expanded because of the level of virtualization.In the “old”, physical world, firewalls are limited to network blocks. All traffic within a subnet does not pass the gateway, and therefore does not pass the firewall.In virtual environments, you can expand this to place a virtual firewall around each VM. So even VMs in the same subnet are still firewalled, while not passing the gateway (aka firewall). This is a major advantage of doing firewalling within a virtual environment.
• #36: S
• #37: Your virtual firewalls are very much needed, but because they run as appliances within your virtual environment, they will be less performant than a physical firewall.That means more vulnerable to a DDoS attack. Consider having multiple layers of firewalls, where you still have your physical layer to block unwanted access and have more fine-grained control via the firewalls inside your cloud, to limit that inter-VM traffic.
• #38: Ideally, you will even secure your connection towards your cloud via VPN. Be it IPSEC, L2TP, PPTP or SSL, by encrypting your traffic to and from the server you are less vulnerable for network sniffing.The network at the cloud might be, secured, it doesn’t mean your office network is secure. All unencrypted traffic can be picked up at any point, and read in plain-text, regardless of where the interception occurs.As a general rule of thumb, stay away from all unencrypted protocols (FTP, Telnet, IMAP, POP3) and prefer the secure variants (sFTP, SSH, IMAPs, POP3s).
• #39: When you’re at the point you need to debug networking, make sure you know what happens. Not only between physical devices; that would pass your physical switches and you can use your current tools to debug that, but also the inter-VM traffic. The traffic that never leaves your hypervisor, but that is shared between virtual machines running on the same hardware server.You can not see that over your switch because it never gets there, you need specific tools to see that. Tools like Altor Networks (recently aqcuired by Juniper) and Trend Micro can offer you that.
• #40: The same logic applies to inter-VM intrusion detection or prevention. The classical way is to deploy IDS or IPS devices that handle that, but if your VM traffic doesn’t even leave your hypervisor, you cannot use those.
• #41: You need some kind if IPS or IDS that is “virtual aware”, that can detect inter-VM traffic as an appliance. If you don’t, someone could simply deploy a new VM and start corrupting your virtual infrastructure from within, because no one knows what is going on, or you cannot monitor it.
• #42: Many systems offer you IDS appliances (usually these are just based on snort), but IDS means you need to inspect them as well. Just for your ease, it’s more fun when there’s a nice GUI involved for a quick overview, and a powerful CLI at the backend to do your actual “work”.
• #43: Clear overviews, clear documentation.
• #44: To add to the complexity, your virtual firewall / IDS / IPS needs to be aware of the movement in your cloud. A VM could be running on one hardware node, and move to a second one an instant later. Your firewall needs to remain valid and up-to-date, no matter where your VM runs.That’s only the infrastructure.
• #45: There’s also your choice of virtualization: software or hardware virtualization? Either way, your hardware nodes are a very important part of your virtual infrastructure. So you need to ask yourself, will you run some kind of OS virtualization like OpenVZ or Xen Server, or prefer other flavors such as Vmware or KVM? Just remember that with software virtualization, gaining access to one of the virtual machines can be as easy as this.
• #46: List your containers, and simply enter the one you want. There, full access, without the owner of the container even knowing about it, and without resetting any kind of password. All that was needed was SSH access to the hardware node. *You* have that access, but what if someone else manages to get hold of it, or hack your box?
• #47: Who knows this many, and what he’s famous for?- Kevin Mitnick
• #48: Social engineering. Abusing the “good will” of people. Talking your way passed security, or talking your way into someone’s mind to give you information you need.Kevin Mitnick was famous for doing so, as he gained entrance to the NASA and the Pentagon through his Social Engineering.
• #50: No matter how secure your virtual infrastructure is, if your applications aren’t secure: you’re sill screwed. SQL Injection, Remote Code Execution, Buffer Overflows, Brute-Force enumeration, Cross Site Scripting (XSF) or Cross Site Request Forgeries (XSRF),
• #51: Dev’s that not only know the theory behind things like cross site scripting, but that know how to do it and protect applications from it.
• #52: Most (public) cloud vendors offer API’s to interact with the cloud. If there’s a security hole in that API, or if your developers don’t use that API as recommended, you can have a big problem. The API’s allow you to manage or change your cloud configuration, deploy new VMs and stop them, which means they are a very powerful medium that can be abused.
• #53: And management that can help you in building and maintaining a cloud, while remaining security conscious.But not this kind of management, as it’s utterly boring.
• #54: Not those men in suits. Real geeks that know how to configure, troubleshoot, deploy and finetune. The ones that don’t get out that much. The ones that, ultimately, have the coolest job ever.
• #55: Because they go beyond limits to get things done, in a good way. Geeks don’t have interests. They have passions.
• #56: Securing a virtual environment isn’t about securing a virtual environment. At least, not only. The physical world still takes up 80% of your “vulnerable surface”.All areas involved in your virtual infrastructure, from your * office connection to your storage* to your hypervisor and * guest operating system and applications in your VM* The developersThey all require the management of people that know how to secure it, that are conscious about configuring those things.
• #57: Q’s and hopefully some A’s.
• #58: Good bye.