LCC17 - Live Patching, Virtual Machine Introspection and Vulnerability Management - Lars Kurth & Cheng Zhang, Citrix
2 likes816 views
The document discusses the advancements in the Xen Project, focusing on topics like live patching and virtual machine introspection (VMI) to enhance security against malware and advanced persistent threats. It details collaborations among various companies to improve Xen's capabilities, including the integration of live patching in XenServer, which allows for security updates without downtime. Additionally, it outlines the processes for vulnerability management and the importance of responsible disclosure in maintaining system security.
LCC17 - Live Patching, Virtual Machine Introspection and Vulnerability Management - Lars Kurth & Cheng Zhang, Citrix
- 1. Lars Kurth Community Manager, Xen Project Chairman, Xen Project Advisory Board Director, Open Source, Citrix larskurth Presentation on xenbits.xenproject.org/people/larsk/ Cheng Zhang Software Engineer, Citrix Currently working for XenServer Livepatch integration and new packaging framework aiwei2013214
- 2. Simon Crosby, XenSource Inc. Xen Project: www.xenproject.org XenServer: www.xenserver.org
- 5. A new way to protect against malware
- 6. 2007 2009 2011 2013 2015 2017 Enablers: from xenaccess/xenprobes to LibVMI Interesting research topic Originally used for forensics (too intrusive for server virt) AA A Xenaccess/Xenprobes LibVMI VMI: enabling commercial applications Hardware assisted VMI solves the intrusion problem Collaboration between: Zentific, Citrix, Bitdefender, Intel and others VMI: HW Support (EPT, …) ARM, alt2pm, .. C CC
- 7. VM3 Guest OS App VMn Guest OS App VM2 Guest OS App Dom0 Dom0 Kernel Drivers Agent(s) Agent(s) Agent(s) Installed in-guest agents, e.g. anti-virus software, VM disk & memory scanner, network monitor, etc. Can be disabled by rootkits and advanced persistent threats (APT)
- 8. Several VM3 VMnVM2Dom0 Dom0 Kernel Drivers VM3 Guest OS App VMn Guest OS App VM2 Guest OS App Security Appliance VM1 Introspection Engine Protected area authentication mechanism to protect the IF Uses HW extensions to monitor memory (e.g. Intel EPT) è Low Intrusion Register rules with Xen to trap on and inspect suspicious activities (e.g. execution of memory on the dynamic heap)
- 9. All malware need an attack technique to gain a foothold Attack techniques exploit specific software bugs/vulnerability The number of available attack techniques is small Buffer Overflows, Heap Sprays, Code Injection, API Hooking, … Because VMI protects against attack techniques It can protect against entirely new malware Verified to block these advanced attacks in real-time APT28, Energetic Bear, DarkHotel, Epic Turla, Regin, ZeuS, Dyreza, EternalBlue1 … solely by relying on VMI WannaCry/EternalBlue blocked in real installations 1 businessinsights.bitdefender.com/hypervisor-introspection-defeated-enternalblue-a-priori
- 10. Rootkits & Advanced Persistent Threats Exploit 0-days in Operating Systems/System Software Can disable agent based security solutions (mask their own existence) VMI solutions operate from outside the VM Thus, it cannot be disabled using traditional attack vectors BUT: VMI is not a replacement, for traditional security solutions It is an extra tool that can be used to increase protection
- 11. Pratap Sankar @ Flickr Documentation wiki.xenproject.org/wiki/Virtual_Machine_Introspection Products Bitdefender HVI XenServer www.bitdefender.com Protection & Remedial Monitoring & Admin Citrix Ready Zentific Zazen (June 17) Xen & XenServer & … www.zentific.com Protection & Remedial Monitoring & Admin Forensics & Data gathering Malware analysis AIS Introvirt XenServer www.ainfosec.com
- 12. A tale of close collaboration within the Xen Project Community
- 13. 2015 2016 2017 4.84.7 P Why did we develop Live Patching? Cloud reboot affected AWS, Rackspace, IBM SoftLayer and others Deploying security patches may require reboots; Inconveniences users How did we fix this? 2015: Design with input from AWS, Alibaba, Citrix, Oracle and SUSE 2016: Xen 4.7 came with Live Patching for x86 2016: Xen 4.8 added extra x86 use-cases and ARM support 2017: XenServer 7.1 releases Live Patching in first commercial product D 10% 1%
- 14. const char *xen_extra_version(void) { return XEN_EXTRAVERSION; } push %rbp mov %rsp,%rbp lea 0x16698b(%rip),%rax leaveq retq const char *xen_extra_version(void) { return “Hello World”; } push %rbp mov %rsp,%rbp lea 0x29333b(%rip),%rax leaveq Retq Replacing compiled functions with new code, encoded in an ELF file called payload, while the hypervisor is running without impacting running guests. Design: xenbits.xenproject.org/docs/unstable/misc/livepatch.html
- 15. The exact source tree used to build the running Xen instance. The .config from the original build of Xen. A build-id onto which the livepatch will be applied. A source patch. livepatch- build- tools The exact same compilation toolchain used to build the running Xen. Livepatch payload
- 16. Supports stacking of different payloads; payloads depend on build-id Functionality: list: lists loaded and applied live patches upload: load & verify a live patch unload: unload a live patch apply: apply a live patch revert: un-apply a live patch Xen 4.8.1 XSA 213 XSA 214 XSA 215 Depends on build-id of 4.8.1 Depends on build-id of XSA 213 Depends on build-id of XSA 214
- 17. Target Dom0 & Guest Linux Kernel Hypervisor Technology Kernel Live Patching kPatch (RedHat) Xen LivePatch kSplice (Oracle) kGraft (SUSE) Function + Data ✔ ✔ ✔ Xen 4.7 ✔ ✔ Inline patching ✗ ✗ ✗ Future ✔ ✗ Data Structures ✗ ✔ via hooks ✔ ✔ ✗ Xen 4.8 via hooks XenServer LivePatch Integrates different solutions into a single user experience For Dom0 (CentOS) For Xen
- 18. Source patches + other build artifacts Hot Fixes contain Per valid patch level: a Xen or Dom0 Live Patch Matching RPMs for most recent patch level In case of a reboot or for Xen/Dom0 not capable of Live Patching Extensive Verification and Validation: The process of patching a live hypervisor or kernel is not an easy task. What happens is a little bit like open heart surgery. The patient is the hypervisor and/or Dom0 itself, and precision and care are needed to get things right. One wrong move and it is game over. Live Patch Live Patch Live Patch build for each patch level package Hot Fix LPsRPMs RPMRPMRPM build for most recent patch level Hot Fix LPsRPMs Publication SigningValidation Verification Q&A (livepatch-build or kpatch-build) (iso)
- 19. XAPI Toolstack Hot Fix LPsRPM downloadHot Fix LPsRPM Dom0 Dom0 Kernel (CentOS) Hypervisor XenCenter or xe Initiates host update SysAdmin Running System instance that supports live patching Disk updates (such that after reboot the patches are applied) works out correct LP & updates (using native live patching tools or APIs)
- 20. Pratap Sankar @ Flickr xenbits.xenproject.org/people/larsk/LCC17 - Build LivePatch.mp4 xenbits.xenproject.org/people/larsk/LCC17 - Apply LivePatch.mov
- 24. Pratap Sankar @ Flickr Xen Project LivePatch Specification & Status xenbits.xenproject.org/docs/unstable/misc/livepatch.html wiki.xenproject.org/wiki/LivePatch Xen Project LivePatch Presentations & Videos xenbits.xenproject.org/people/larsk/FOSDEM17-LivePatch.pdf (Short) people/larsk/XPDS16-LivePatch.pdf (Long) Xen Project LivePatch Videos fosdem.org/2017/schedule/event/iaas_livepatxen/ XenServer xenserver.org
- 27. Vulnerability data from cvedetails.com 0 50 100 150 200 250 300 350 2017* 2016 2015 2014 2013 2012 n Xen Project n Linux Kernel n QEMU/KVM *) Data up to May 31st, 2017
- 28. R: Vulnerability reported to security@xenproject.org P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org R P Fixing Security Bugs: Dedicated security team = security experts from within the Xen Project Community Security Team: Triage Creation of fix/patches Can a Livepatch can be created? No? If possible, re-write fix/patches Validation of fix/patches Assignment of CVE Issue description and risk analysis
- 29. AR P Fix their systems/software: Eligible Xen Project Users are informed under embargo of the vulnerability Eligible Users = Pre-disclosure list members: Product Companies, Open Source & Commercial Distros (e.g. Huawei, Debian) Service/Cloud Providers (e.g. Alibaba) Large Private Downstream (e.g. Google) Allowed to share information via xen-security-issues- discuss@lists.xenproject.org R: Vulnerability reported to security@xenproject.org P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa
- 30. A XR P General Publication: Information about vulnerability is made public Everyone else: Patches their systems either through security updates from distros/products or builds them from source. Users of service/cloud providers will not be impacted R: Vulnerability reported to security@xenproject.org P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa
- 31. A XR P Responsible Disclosure: fix critical systems/software before publication R: Vulnerability reported to security@... P: Vulnerability pre-disclosed to eligible users A: Vulnerability announced publicly F: Fix available Full Disclosure, immediate (no-fix): public disclosure without a fix A XR F A XR Full Disclosure, post-fix: public disclosure with a fix F F
- 32. 4) http://oss-security.openwall.org/wiki/mailing-lists/distros 5) http://www.openwall.com/lists/oss-security 6) No Chinese companies or distros on pre-disclosure list 7) Only handles x86 KVM bugs (no ARM or other bugs) 1) Is the CVE severity used to handle vulnerabilities differently? 2) Days embargoed (information is secret) 3) D = Distros/Products, S = Public Service, P = Private Downstream Responsible only Days 2 Who? 3FOSS Project Bug Severity 1 Process Type 14-19 D 6 Linux Kernel via OSS security distros 4 ≥ Medium – Critical Responsible Disclosure 14-19 D 6 QEMU/KVM via OSS security distros 4 OSS security 5 ≥ Medium – Critical ≤ Low Responsible Disclosure 7 Full Disclosure, no-fix 3-5 D, S, POpenStack OSSA OpenStack OSSN ≥ Medium – Critical ≤ Low Responsible Disclosure Full Disclosure, post-fix Xen Hypervisor Includes Linux & QEMU vulnerabilities in supported Xen configurations Low – Critical Responsible Disclosure 14 D, S, P 14-19 D 6 Linux Kernel via OSS security distros 4 OSS security 5 ≥ Medium – Critical ≤ Low Responsible Disclosure Full Disclosure, no-fix
- 33. Picture by Lars Kurth
- 34. Only Hypervisor with VMI Protection from new classes of malware Several security companies working with XenServer Live Patching Disruption free application of vulnerabilities Used by several cloud providers Used best in commercial products, e.g. XenServer Industry Leading Vulnerability Process Includes QEMU and Kernel XSAs Designed with input from Cloud Providers Stable number of CVEs BUT: the Xen Project cannot today distribute XSAs as Live Patches (the project delivers source code only) Sys Admins Extra protection = extra piece of mind Picture by Lars Kurth
- 35. xenbits.xenproject.org/people/larsk You can also contact Patrick Zhang (patrick.zhang@citrix.com) after the presentation Picture by Lars Kurth