SlideShare a Scribd company logo
Argo
Hypervisor-Mediated data eXchange
Christopher Clark
OpenXT Project
Xen Design and Developer Summit, 10th
July, 2019
Argo
● What it is
○ Overview
○ Origin
● Why
○ Key properties
○ What is different, distinctive about it
● How it relates to other hypervisors
○ HMX in other hypervisors
● A bit more about what it is
○ Details of the interface
○ Examples of it in use
● References
Argo : an interdomain communication transport
● An OpenXT technology, originally developed for XenClient.
○ Derived from v4v deployed in uXen in Bromium’s vSentry on millions of machines.
● Hypervisor-mediated data copies via private ring buffers with notifications.
○ Prioritizes isolation, access control and mandatory conformance to the transfer protocol.
○ Simple but powerful primitive to build upon.
● Included in Xen 4.12
○ Set the KConfig option CONFIG_ARGO=y to build it in,
then turn it on at the Xen boot command line: argo=1,mac_permissive=1
Argo : an Odyssey
The longest patch upstreaming process in Xen’s history. 2011 - 2019.
Argo : an Odyssey
The longest patch upstreaming process in Xen’s history. 2011 - 2019.
● 2010: XenClient project begins
● 2011: Ross reserves the hypercall number (39) in Xen. ← The timer starts here with the first patch integrated.
● 2012: Jean’s v4v patch series posted
● 2013: Phil’s Xen Summit presentation, Ross’s v4v patch series posted
● 2014: OpenXT project begins
● 2016: Clark presentation at Xen Summit
● 2017: HP ships laptops with Bromium micro-virtualization for Hyper-V and Windows 10 VBS
● 2018 - 2019: Argo patch series submitted, iterated, reviewed and accepted into Xen
● 2019: Design activity on Argo’s next development
HMX:
● 2019: Microsoft ships Windows Sandbox and SystemGuard in Windows 10
● 2019: IBM/RedHat microvm QEMU machine type inspired by Intel NEMU and AWS Firecracker
Argo : an Odyssey
One new hypercall, with four operations.
○ Register ring
○ Unregister ring
○ Sendv
○ Notify
Argo : Key Properties
● No memory is shared between guest VMs.
● Data is copied from the source to the destination.
○ Maintains spatial isolation between guest VMs.
● The Hypervisor performs the data transfer into guest-accessible memory.
○ Ensures that the transfer protocol in the ring structure is strictly adhered to.
○ Enables enforcement of mandatory access control policy by the hypervisor.
These properties will be maintained.
MILS Architecture Foundational Security Principals (2005)
● Data Isolation
Information in a partition is accessible only by that partition and private data remains private.
● Control of Information Flow
Information flow between partitions is from an authenticated source to authenticated recipients;
the source of information is authenticated to the recipient, and information only goes where intended.
● Periods Processing / Temporal Separation
Resources may be used by different components by time-slicing, where the system enforces that the
resource is cleaned to remove any trace of its previous use before being reassigned.
● Fault Isolation
Failure within a partition is prevented from cascading to any other partition.
Failures are detected, contained and recovered locally.
Argo : an interdomain communication transport
● Strong isolation between communicating domains
○ No memory is shared between VMs.
○ Does not use or require grant tables or XenStore.
● Strong mechanism for data transfer
○ Hypervisor performs the data movement. Memory accesses conform to the protocol.
● High performance suitable for sustained throughput
● A clean mapping to Linux and Windows native I/O primitives
● Clear separation from guest Operating System networking stacks
● A foundation for the future work that we intend to do
HMX : Hypervisor Mediated data eXchange
A term to describe:
Asynchronous authenticated message passing between VM partitions,
performed by the hypervisor.
Channels use no shared memory between the source and the receiver.
Enable enforcement of Control of Information Flow between domains
and preserve Data Isolation.
HMX: pattern for data delivery
Hypervisor
Message
Data
Receive memory
buffer
Delivery performed by the hypervisor:
● data delivered with context (size, origin)
● writes to the receive buffer, will conform to protocol / structure
VM : Sender VM : Receiver
DataContext
Hypervisor invoked
to send message
Inter-partition Communication: Xen (Grants)
Traditional PV communication channels using Xen’s primitives: Grants + Events
eg: PV network front and back ends.
Grants are most commonly used to share or copy pages between domains.
● Grant copy operations can ask hypervisor to write to a remote page, but:
○ Conformance to a ring access protocol is not enforced.
○ Hypervisor does not provide message header with metadata about the source.
=> HMX: no.
Aside: Grant table code has excessive complexity: Attributed responsibility for 5% of Xen’s XSA/CVEs.
Most recent grant table XSA was issued... yesterday.
Two API versions, switchable at runtime plus legacy operations... so OpenXT adds KCONFIGs to reduce it.
Inter-partition Communication: Hyper-V
VMBus : HvPostMessage
Messages are copied by the hypervisor into a private per-receiver message buffer
in hypervisor-owned private memory.
For delivery, individual messages are then copied by the hypervisor out into a
Message Page shared between the hypervisor and the receiver partition, when free
slots within the page are available.
=> HMX: yes.
since at least 2006.
Inter-partition Communication: uXen
Bromium vSentry uses uXen, which uses v4v.
● Open Source
● Has no legacy PV interfaces.
○ no grant tables
○ no XenStore
● All virtual devices use the simple, copy-based primitive.
○ No memory sharing of any kind.
Demonstrates that Xen’s problematic legacy interfaces can be replaced with an
approach built upon Argo.
=> HMX: yes.
Inter-partition Communication: Xen (Argo, since 4.12)
=> HMX : yes.
Argo: ring registration
Hypercall operations to establish and tear down a communication channel:
Register ring:
● Guest supplies memory to be used for receiving incoming messages.
● Specifies either: a specific other domain as sender, or receive from anyone.
● Access control policy check: is this channel allowed?
● Ring is mapped into the hypervisor address space.
Unregister ring
● Removes the hypervisor state. Message delivery ceases.
Argo: sendv
Sendv:
● Guest indicates the destination ring it wants to send a message to
○ and provides the data to be sent
● Access control check: is this transmission allowed?
● Is there sufficient space in the destination ring?
○ If not, interest is recorded and a notification will occur when sufficient space is free.
● Hypervisor synchronously copies the data into the ring.
○ Message header is written by the hypervisor, indicating the size and source.
Argo: notify
Notify:
● Queries for space availability in remote rings
○ Access control check: can this domain send to the specified ring?
○ Returns ring state indicator flags:
■ Exists
■ Shared
■ Empty
■ Sufficient space exists for the specified message size
■ Busy
○ Register sender’s interest: notification issued when available space becomes sufficient
● Initiates checking of space within own rings to trigger pending notifications
Argo: XSM/Flask controls
● enable
○ governs whether a domain may access any Argo operations at all
● ring registration
○ single-sender rings: allow / disallow registration of rings with the specified sender
○ any-sender rings: allow / disallow a domain to register any-sender rings
● send
○ allow sending a message to a specified domain
○ if the send permission is not granted, notify query operations will also be disallowed
Argo: use cases in OpenXT
Examples:
● User interface export from the control domain to the User Interface renderer VM
● Stubdomain toolstack helper services
● Interdomain DBUS service
○ Enables use of existing components maintained by other Open Source
communities, deployed in a multi-domain configuration
■ eg. NetworkManager and its user interface
● VM provisioning services for remote management
● Debugging support: ssh access to remote VM terminals
Argo: simple to use: ssh example
eg. Using the Linux device driver and userspace library to communicate between VMs:
# VM1: load the Argo kernel module:
insmod /lib/modules/*/extra/xen-argo.ko
# use the interposer to run the ssh server on port 8022
export INET_IS_ARGO=1
LD_PRELOAD=/usr/lib/libargo-1.0.so.0.0.0/usr/sbin/sshd -p 8022
# VM2: load the Argo kernel module:
insmod /lib/modules/*/extra/xen-argo.ko
# use the interposer to run the ssh client to connect to the server
export INET_IS_ARGO=1
LD_PRELOAD=/usr/lib/libargo-1.0.so.0.0.0/usr/bin/ssh 1.0.0.1 -p 8022
ssh example
ssh server process
libargo interposer library
process env: INET_IS_ARGO=1
Linux kernel driver: xen-argo.ko
/dev/argo_stream
Xen hypervisor
System calls using the Linux Argo device
Hypercalls: argo_op : REGISTER_RING,
SENDV, NOTIFY
Argo VIRQ
ssh client process
libargo interposer library
process env: INET_IS_ARGO=1
Linux kernel driver: xen-argo.ko
/dev/argo_stream
System calls using the Linux Argo device
Hypercalls: argo_op: SENDV, NOTIFY
Argo VIRQ
ssh example
Xen
Message
Data
Receive memory
buffer
Delivery performed by Argo in Xen:
● data delivered with context (size, origin)
● writes to the receive buffer, will conform to protocol / structure
VM with ssh client VM with ssh server
DataContext
Hypervisor invoked
to send message
Development items
● Sender domain context: hypervisor to provide metadata to message recipient
○ message header conveys sender’s XSM sid and bits indicating stubdomain relationship state
● Connection state: enforce and track ring pairings for bidirectional communication
○ Tracking bidirectional connections enables fine-grained access policy control
● Access control: run-time configurable firewall
○ Uses sender context and connection state. Replaces OpenXT’s existing v4vtables port.
● Nesting data transport
○ Sending data between VMs at different levels of nesting, with firewall policy enforcement
● Wildcard rings: protection of client forward progress
○ Increases resilience of the clients of any-sender rings to DoS by other clients
● Shutter rings: mechanism and policy controls for reducing hypervisor mappings
Research items
● Accelerated transport options leveraging available hardware
● Asynchronous message send primitive
● Reducing scope of notify wakeups to replace existing broadcast
● Integration with memory encryption technologies
References
Ian Pratt, Founder of the Xen Project, CTO of Bromium, at PSEC 2018: Hypervisor Security : Lessons Learned
Essential viewing. Strongly recommended.
References
● PSEC 2018: Hypervisor-Mediated data eXchange (HMX): Primitives for Authentic Communication
● Xen Summit 2013: XenClient Mechanisms to Strengthen Guest Separation
● Microsoft Hyper-V: Hypercall APIs Explained WinHEC 2006 presentation, slide 45 describes HvPostMessage.
○ Hypervisor Top Level Functional Specification, v5.0C, see HvPostMessage references, inc. section 11.11.
● OpenXT Summit 2016: Secure Virtual Platform
● Argo source code:
○ Hypervisor: argo.c in Xen 4.12
○ Linux drivers: current driver, and under-development vsock-based driver
○ Linux userspace: libargo, including LD_PRELOAD interposer enabling transparent use by socket applications
○ For a Windows driver, a port from v4v is required: see the xenv4v source code in OpenXT as a starting point.
● Argo documentation: argo.pandoc in Xen 4.12
● Argo next development: OpenXT wiki on Argo development items
● OpenXT: about the Project, project events, v4v documentation, v4vtables port for basic access control, with tooling
● uXen: v4v.c in uXen 4.1.6. uXen hypervisor source code repository: https://github.com/OpenXT/uxen/
Any questions? Please get in touch: #OpenXT on freenode IRC

More Related Content

PDF
XPDDS19: The Xen-Blanket for 2019 - Christopher Clark and Kelli Little, Star ...
PPTX
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
PDF
OSCON14: Mirage 2.0
PDF
XPDDS18: Unikraft: An easy way of crafting Unikernels on Arm - Kaly Xin, ARM
PPTX
XPDDS18: Windows PV Drivers Project: Status and Updates - Paul Durrant, Citri...
PDF
2018 Genivi Xen Overview Nov Update
PDF
Xen Project 15 Years down the Line
PPTX
ALSS14: Xen Project Automotive Hypervisor (Demo)
XPDDS19: The Xen-Blanket for 2019 - Christopher Clark and Kelli Little, Star ...
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
OSCON14: Mirage 2.0
XPDDS18: Unikraft: An easy way of crafting Unikernels on Arm - Kaly Xin, ARM
XPDDS18: Windows PV Drivers Project: Status and Updates - Paul Durrant, Citri...
2018 Genivi Xen Overview Nov Update
Xen Project 15 Years down the Line
ALSS14: Xen Project Automotive Hypervisor (Demo)

What's hot (20)

PDF
LCEU13: Securing your cloud with Xen's advanced security features - George Du...
PDF
XPDDS18: CPUFreq in Xen on ARM - Oleksandr Tyshchenko, EPAM Systems
PPTX
Scale17x: Thinking outside of the conceived tech comfort zone
PDF
Erlang on Xen: Redefining the cloud software stack
PDF
XPDDS18: Unleashing the Power of Unikernels with Unikraft - Florian Schmidt, ...
PPTX
LinuxCon Japan 13 : 10 years of Xen and Beyond
PPTX
Scale14x: Are today's foss security practices robust enough in the cloud era ...
PDF
XDF18: Heterogeneous Real-Time SoC Software Architecture - Stefano Stabellini...
PDF
Scale 12x Securing Your Cloud with The Xen Hypervisor
PDF
OSSEU18: From Handcraft to Unikraft: Simpler Unikernelization of Your Applica...
PDF
CIF16: Building the Superfluid Cloud with Unikernels (Simon Kuenzer, NEC Europe)
PDF
OSSEU18: NVDIMM and Virtualization - George Dunlap, Citrix
PDF
XPDS14 - Zero-Footprint Guest Memory Introspection from Xen - Mihai Dontu, Bi...
PDF
XPDS14: OpenXT - Security and the Properties of a Xen Virtualisation Platform...
PDF
XPDDS19: Xen API Archaeology: Creating a Full-Featured VMI Debugger for the...
PDF
Unikraft Landing Page Master Slides
PDF
Rootlinux17: An introduction to Xen Project Virtualisation
PDF
XPDDS18: Introducing ViryaOS: Secure Containers for Embedded and IoT - Stefan...
PDF
XPDDS18: LCC18: Xen Project: After 15 years, What's Next? - George Dunlap, C...
PDF
Xen Project: Windows PV Drivers
LCEU13: Securing your cloud with Xen's advanced security features - George Du...
XPDDS18: CPUFreq in Xen on ARM - Oleksandr Tyshchenko, EPAM Systems
Scale17x: Thinking outside of the conceived tech comfort zone
Erlang on Xen: Redefining the cloud software stack
XPDDS18: Unleashing the Power of Unikernels with Unikraft - Florian Schmidt, ...
LinuxCon Japan 13 : 10 years of Xen and Beyond
Scale14x: Are today's foss security practices robust enough in the cloud era ...
XDF18: Heterogeneous Real-Time SoC Software Architecture - Stefano Stabellini...
Scale 12x Securing Your Cloud with The Xen Hypervisor
OSSEU18: From Handcraft to Unikraft: Simpler Unikernelization of Your Applica...
CIF16: Building the Superfluid Cloud with Unikernels (Simon Kuenzer, NEC Europe)
OSSEU18: NVDIMM and Virtualization - George Dunlap, Citrix
XPDS14 - Zero-Footprint Guest Memory Introspection from Xen - Mihai Dontu, Bi...
XPDS14: OpenXT - Security and the Properties of a Xen Virtualisation Platform...
XPDDS19: Xen API Archaeology: Creating a Full-Featured VMI Debugger for the...
Unikraft Landing Page Master Slides
Rootlinux17: An introduction to Xen Project Virtualisation
XPDDS18: Introducing ViryaOS: Secure Containers for Embedded and IoT - Stefan...
XPDDS18: LCC18: Xen Project: After 15 years, What's Next? - George Dunlap, C...
Xen Project: Windows PV Drivers
Ad

Similar to XPDDS19: Argo and Hypervisor-Mediated Data eXchange (HMX) - Christopher Clark, OpenXT Project (20)

PPTX
Containerization & Docker - Under the Hood
PDF
Autopilot : Securing Cloud Native Storage
PDF
Why Choose Xen For Your Cloud?
PDF
Why Choose Xen For Your Cloud?
PDF
Heterogeneous multiprocessing on androd and i.mx7
PDF
OpenStack on SmartOS
PDF
Oscon 2012 : From Datacenter to the Cloud - Featuring Xen and XCP
PDF
OpenVZ Linux Containers
ODP
Performance characterization in large distributed file system with gluster fs
PPTX
Docker Swarm secrets for creating great FIWARE platforms
PDF
XPDS16: The OpenXT Project in 2016 - Christopher Clark, BAE Systems
PDF
CloudStack In Production
PDF
RPC in Smalltalk
PDF
FIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE Platforms
PDF
ACM_Intro_Containers_Cloud.pdf Cloud.pdf
PDF
Security of Linux containers in the cloud
PDF
State of ARM-based HPC
PDF
The lies we tell our code, LinuxCon/CloudOpen 2015-08-18
PPTX
Linux 开源操作系统发展新趋势
PDF
Kubernetes: My BFF
Containerization & Docker - Under the Hood
Autopilot : Securing Cloud Native Storage
Why Choose Xen For Your Cloud?
Why Choose Xen For Your Cloud?
Heterogeneous multiprocessing on androd and i.mx7
OpenStack on SmartOS
Oscon 2012 : From Datacenter to the Cloud - Featuring Xen and XCP
OpenVZ Linux Containers
Performance characterization in large distributed file system with gluster fs
Docker Swarm secrets for creating great FIWARE platforms
XPDS16: The OpenXT Project in 2016 - Christopher Clark, BAE Systems
CloudStack In Production
RPC in Smalltalk
FIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE Platforms
ACM_Intro_Containers_Cloud.pdf Cloud.pdf
Security of Linux containers in the cloud
State of ARM-based HPC
The lies we tell our code, LinuxCon/CloudOpen 2015-08-18
Linux 开源操作系统发展新趋势
Kubernetes: My BFF
Ad

More from The Linux Foundation (20)

PDF
ELC2019: Static Partitioning Made Simple
PDF
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
PDF
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
PDF
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
PDF
XPDDS19 Keynote: Unikraft Weather Report
PDF
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
PDF
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
PDF
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
PDF
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
PPTX
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
PPTX
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
PDF
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
PDF
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
PDF
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
PDF
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
PDF
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
PDF
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
PDF
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
PDF
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
PDF
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
ELC2019: Static Partitioning Made Simple
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Unikraft Weather Report
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE

Recently uploaded (20)

PDF
NewMind AI Weekly Chronicles - August'25 Week I
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
PPTX
Spectroscopy.pptx food analysis technology
DOCX
The AUB Centre for AI in Media Proposal.docx
PDF
Machine learning based COVID-19 study performance prediction
PDF
Encapsulation theory and applications.pdf
PPTX
sap open course for s4hana steps from ECC to s4
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
NewMind AI Weekly Chronicles - August'25 Week I
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Diabetes mellitus diagnosis method based random forest with bat algorithm
Unlocking AI with Model Context Protocol (MCP)
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
Chapter 3 Spatial Domain Image Processing.pdf
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Digital-Transformation-Roadmap-for-Companies.pptx
Spectral efficient network and resource selection model in 5G networks
Per capita expenditure prediction using model stacking based on satellite ima...
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Spectroscopy.pptx food analysis technology
The AUB Centre for AI in Media Proposal.docx
Machine learning based COVID-19 study performance prediction
Encapsulation theory and applications.pdf
sap open course for s4hana steps from ECC to s4
The Rise and Fall of 3GPP – Time for a Sabbatical?
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...

XPDDS19: Argo and Hypervisor-Mediated Data eXchange (HMX) - Christopher Clark, OpenXT Project

  • 1. Argo Hypervisor-Mediated data eXchange Christopher Clark OpenXT Project Xen Design and Developer Summit, 10th July, 2019
  • 2. Argo ● What it is ○ Overview ○ Origin ● Why ○ Key properties ○ What is different, distinctive about it ● How it relates to other hypervisors ○ HMX in other hypervisors ● A bit more about what it is ○ Details of the interface ○ Examples of it in use ● References
  • 3. Argo : an interdomain communication transport ● An OpenXT technology, originally developed for XenClient. ○ Derived from v4v deployed in uXen in Bromium’s vSentry on millions of machines. ● Hypervisor-mediated data copies via private ring buffers with notifications. ○ Prioritizes isolation, access control and mandatory conformance to the transfer protocol. ○ Simple but powerful primitive to build upon. ● Included in Xen 4.12 ○ Set the KConfig option CONFIG_ARGO=y to build it in, then turn it on at the Xen boot command line: argo=1,mac_permissive=1
  • 4. Argo : an Odyssey The longest patch upstreaming process in Xen’s history. 2011 - 2019.
  • 5. Argo : an Odyssey The longest patch upstreaming process in Xen’s history. 2011 - 2019. ● 2010: XenClient project begins ● 2011: Ross reserves the hypercall number (39) in Xen. ← The timer starts here with the first patch integrated. ● 2012: Jean’s v4v patch series posted ● 2013: Phil’s Xen Summit presentation, Ross’s v4v patch series posted ● 2014: OpenXT project begins ● 2016: Clark presentation at Xen Summit ● 2017: HP ships laptops with Bromium micro-virtualization for Hyper-V and Windows 10 VBS ● 2018 - 2019: Argo patch series submitted, iterated, reviewed and accepted into Xen ● 2019: Design activity on Argo’s next development HMX: ● 2019: Microsoft ships Windows Sandbox and SystemGuard in Windows 10 ● 2019: IBM/RedHat microvm QEMU machine type inspired by Intel NEMU and AWS Firecracker
  • 6. Argo : an Odyssey One new hypercall, with four operations. ○ Register ring ○ Unregister ring ○ Sendv ○ Notify
  • 7. Argo : Key Properties ● No memory is shared between guest VMs. ● Data is copied from the source to the destination. ○ Maintains spatial isolation between guest VMs. ● The Hypervisor performs the data transfer into guest-accessible memory. ○ Ensures that the transfer protocol in the ring structure is strictly adhered to. ○ Enables enforcement of mandatory access control policy by the hypervisor. These properties will be maintained.
  • 8. MILS Architecture Foundational Security Principals (2005) ● Data Isolation Information in a partition is accessible only by that partition and private data remains private. ● Control of Information Flow Information flow between partitions is from an authenticated source to authenticated recipients; the source of information is authenticated to the recipient, and information only goes where intended. ● Periods Processing / Temporal Separation Resources may be used by different components by time-slicing, where the system enforces that the resource is cleaned to remove any trace of its previous use before being reassigned. ● Fault Isolation Failure within a partition is prevented from cascading to any other partition. Failures are detected, contained and recovered locally.
  • 9. Argo : an interdomain communication transport ● Strong isolation between communicating domains ○ No memory is shared between VMs. ○ Does not use or require grant tables or XenStore. ● Strong mechanism for data transfer ○ Hypervisor performs the data movement. Memory accesses conform to the protocol. ● High performance suitable for sustained throughput ● A clean mapping to Linux and Windows native I/O primitives ● Clear separation from guest Operating System networking stacks ● A foundation for the future work that we intend to do
  • 10. HMX : Hypervisor Mediated data eXchange A term to describe: Asynchronous authenticated message passing between VM partitions, performed by the hypervisor. Channels use no shared memory between the source and the receiver. Enable enforcement of Control of Information Flow between domains and preserve Data Isolation.
  • 11. HMX: pattern for data delivery Hypervisor Message Data Receive memory buffer Delivery performed by the hypervisor: ● data delivered with context (size, origin) ● writes to the receive buffer, will conform to protocol / structure VM : Sender VM : Receiver DataContext Hypervisor invoked to send message
  • 12. Inter-partition Communication: Xen (Grants) Traditional PV communication channels using Xen’s primitives: Grants + Events eg: PV network front and back ends. Grants are most commonly used to share or copy pages between domains. ● Grant copy operations can ask hypervisor to write to a remote page, but: ○ Conformance to a ring access protocol is not enforced. ○ Hypervisor does not provide message header with metadata about the source. => HMX: no. Aside: Grant table code has excessive complexity: Attributed responsibility for 5% of Xen’s XSA/CVEs. Most recent grant table XSA was issued... yesterday. Two API versions, switchable at runtime plus legacy operations... so OpenXT adds KCONFIGs to reduce it.
  • 13. Inter-partition Communication: Hyper-V VMBus : HvPostMessage Messages are copied by the hypervisor into a private per-receiver message buffer in hypervisor-owned private memory. For delivery, individual messages are then copied by the hypervisor out into a Message Page shared between the hypervisor and the receiver partition, when free slots within the page are available. => HMX: yes. since at least 2006.
  • 14. Inter-partition Communication: uXen Bromium vSentry uses uXen, which uses v4v. ● Open Source ● Has no legacy PV interfaces. ○ no grant tables ○ no XenStore ● All virtual devices use the simple, copy-based primitive. ○ No memory sharing of any kind. Demonstrates that Xen’s problematic legacy interfaces can be replaced with an approach built upon Argo. => HMX: yes.
  • 15. Inter-partition Communication: Xen (Argo, since 4.12) => HMX : yes.
  • 16. Argo: ring registration Hypercall operations to establish and tear down a communication channel: Register ring: ● Guest supplies memory to be used for receiving incoming messages. ● Specifies either: a specific other domain as sender, or receive from anyone. ● Access control policy check: is this channel allowed? ● Ring is mapped into the hypervisor address space. Unregister ring ● Removes the hypervisor state. Message delivery ceases.
  • 17. Argo: sendv Sendv: ● Guest indicates the destination ring it wants to send a message to ○ and provides the data to be sent ● Access control check: is this transmission allowed? ● Is there sufficient space in the destination ring? ○ If not, interest is recorded and a notification will occur when sufficient space is free. ● Hypervisor synchronously copies the data into the ring. ○ Message header is written by the hypervisor, indicating the size and source.
  • 18. Argo: notify Notify: ● Queries for space availability in remote rings ○ Access control check: can this domain send to the specified ring? ○ Returns ring state indicator flags: ■ Exists ■ Shared ■ Empty ■ Sufficient space exists for the specified message size ■ Busy ○ Register sender’s interest: notification issued when available space becomes sufficient ● Initiates checking of space within own rings to trigger pending notifications
  • 19. Argo: XSM/Flask controls ● enable ○ governs whether a domain may access any Argo operations at all ● ring registration ○ single-sender rings: allow / disallow registration of rings with the specified sender ○ any-sender rings: allow / disallow a domain to register any-sender rings ● send ○ allow sending a message to a specified domain ○ if the send permission is not granted, notify query operations will also be disallowed
  • 20. Argo: use cases in OpenXT Examples: ● User interface export from the control domain to the User Interface renderer VM ● Stubdomain toolstack helper services ● Interdomain DBUS service ○ Enables use of existing components maintained by other Open Source communities, deployed in a multi-domain configuration ■ eg. NetworkManager and its user interface ● VM provisioning services for remote management ● Debugging support: ssh access to remote VM terminals
  • 21. Argo: simple to use: ssh example eg. Using the Linux device driver and userspace library to communicate between VMs: # VM1: load the Argo kernel module: insmod /lib/modules/*/extra/xen-argo.ko # use the interposer to run the ssh server on port 8022 export INET_IS_ARGO=1 LD_PRELOAD=/usr/lib/libargo-1.0.so.0.0.0/usr/sbin/sshd -p 8022 # VM2: load the Argo kernel module: insmod /lib/modules/*/extra/xen-argo.ko # use the interposer to run the ssh client to connect to the server export INET_IS_ARGO=1 LD_PRELOAD=/usr/lib/libargo-1.0.so.0.0.0/usr/bin/ssh 1.0.0.1 -p 8022
  • 22. ssh example ssh server process libargo interposer library process env: INET_IS_ARGO=1 Linux kernel driver: xen-argo.ko /dev/argo_stream Xen hypervisor System calls using the Linux Argo device Hypercalls: argo_op : REGISTER_RING, SENDV, NOTIFY Argo VIRQ ssh client process libargo interposer library process env: INET_IS_ARGO=1 Linux kernel driver: xen-argo.ko /dev/argo_stream System calls using the Linux Argo device Hypercalls: argo_op: SENDV, NOTIFY Argo VIRQ
  • 23. ssh example Xen Message Data Receive memory buffer Delivery performed by Argo in Xen: ● data delivered with context (size, origin) ● writes to the receive buffer, will conform to protocol / structure VM with ssh client VM with ssh server DataContext Hypervisor invoked to send message
  • 24. Development items ● Sender domain context: hypervisor to provide metadata to message recipient ○ message header conveys sender’s XSM sid and bits indicating stubdomain relationship state ● Connection state: enforce and track ring pairings for bidirectional communication ○ Tracking bidirectional connections enables fine-grained access policy control ● Access control: run-time configurable firewall ○ Uses sender context and connection state. Replaces OpenXT’s existing v4vtables port. ● Nesting data transport ○ Sending data between VMs at different levels of nesting, with firewall policy enforcement ● Wildcard rings: protection of client forward progress ○ Increases resilience of the clients of any-sender rings to DoS by other clients ● Shutter rings: mechanism and policy controls for reducing hypervisor mappings
  • 25. Research items ● Accelerated transport options leveraging available hardware ● Asynchronous message send primitive ● Reducing scope of notify wakeups to replace existing broadcast ● Integration with memory encryption technologies
  • 26. References Ian Pratt, Founder of the Xen Project, CTO of Bromium, at PSEC 2018: Hypervisor Security : Lessons Learned Essential viewing. Strongly recommended.
  • 27. References ● PSEC 2018: Hypervisor-Mediated data eXchange (HMX): Primitives for Authentic Communication ● Xen Summit 2013: XenClient Mechanisms to Strengthen Guest Separation ● Microsoft Hyper-V: Hypercall APIs Explained WinHEC 2006 presentation, slide 45 describes HvPostMessage. ○ Hypervisor Top Level Functional Specification, v5.0C, see HvPostMessage references, inc. section 11.11. ● OpenXT Summit 2016: Secure Virtual Platform ● Argo source code: ○ Hypervisor: argo.c in Xen 4.12 ○ Linux drivers: current driver, and under-development vsock-based driver ○ Linux userspace: libargo, including LD_PRELOAD interposer enabling transparent use by socket applications ○ For a Windows driver, a port from v4v is required: see the xenv4v source code in OpenXT as a starting point. ● Argo documentation: argo.pandoc in Xen 4.12 ● Argo next development: OpenXT wiki on Argo development items ● OpenXT: about the Project, project events, v4v documentation, v4vtables port for basic access control, with tooling ● uXen: v4v.c in uXen 4.1.6. uXen hypervisor source code repository: https://github.com/OpenXT/uxen/ Any questions? Please get in touch: #OpenXT on freenode IRC