Securely Serving Millions of Boot Artifacts a Day by João Pedro Lima & Matt Fleming
0 likes119 views
Cloudflare’s boot infrastructure dynamically generates and signs boot artifacts for nodes worldwide, ensuring secure, scalable, and customizable deployments. This talk dives into its architecture, scaling decisions, and how it enables seamless testing while maintaining a strong chain of trust.
Securely Serving Millions of Boot Artifacts a Day by João Pedro Lima & Matt Fleming
- 1. A ScyllaDB Community Securely Serving Millions of Boot Artifacts a Day João Pedro Lima Systems Engineer, Linux Team Matt Fleming Senior Systems Engineer
- 2. João Pedro Lima (he/him) ■ Previously Product Security @VMWare, Infrastructure Security @ Cloudflare ■ Interested in OS Security and Cryptography
- 3. Matt Fleming (he/him) ■ Former Linux Kernel maintainer ■ Focused on performance of OS, DB, and dist sys ■ Co-authored papers on Change Point Detection and testing distributed systems
- 4. ■ Cloudflare’s Fleet Architecture ■ Boot Process ■ Design Evolution ■ Future Work Presentation Agenda
- 6. Cloudflare has data centers in over 335 cities ■ Edge vs. control plane ■ Control plane data centers can have thousands of machines ■ Edge data centers can be big or small ■ Edge compute servers are called “metals” for historical reasons ■ OS is stateless and executes from ramdisk ■ Optimised, latest LTS Linux Kernel ■ Debian 12 (bookworm) ■ Secure Boot Fleet Architecture
- 7. ■ Each datacenter has a set of datacenter manager (DM) nodes ■ DM renders boot configuration for all nodes in datacenter from configuration management ■ Metal requests boot artifacts from DM ■ DM runs nginx ■ DM cryptographically signs all artifacts on render ■ Metal verifies artifacts before executing Fleet Architecture
- 8. Boot Process
- 9. Boot Process Power-on /Reset Chipset/CPU BIOS/UEFI firmware PXE/iPXE OS Kernel Userspace Baseboard management controller (BMC) iPXE scripts
- 10. ■ Secure boot keys configured when metal is provisioned into datacenter ■ iPXE used to pull kernel image and ramdisk via HTTP ■ Secure boot used to verify kernel images and modules Boot Process
- 11. #!ipxe :diag kernel ${boot_prefix}/vmlinuz initrd=diag-image.img console=tty0 imgverify vmlinuz ${boot_prefix}/vmlinuz.sig initrd ${boot_prefix}/diag-image.img imgverify diag-image.img ${boot_prefix}/diag-image.img.sig boot :updates imgfetch --name {{ hw_model }}/update.ipxe {{ hw_model }}/update.ipxe imgverify {{ hw_model }}/update.ipxe {{ hw_model }}/update.ipxe.sig imgexec {{ hw_model }}/update.ipxe :baseimg kernel vmlinuz imgverify vmlinuz ${boot_prefix}/vmlinuz.sig initrd ${boot_prefix}/baseimg.img imgverify baseimg.img ${boot_prefix}/baseimg.img.sig initrd ${boot_prefix}/{{ net_img }}.img imgverify {{ net_img }}.img ${boot_prefix}/{{ net_img }}.img.sig boot
- 12. Secure boot Power-on /Reset Chipset/CPU BIOS/UEFI firmware PXE/iPXE OS Kernel Userspace Baseboard management controller (BMC) iPXE scripts https://blog.cloudflare.com/anchoring-trust-a-hardware-secure-boot-story/ Platform Secure Boot/ HW root of trust UEFI Secure Boot
- 13. Secure boot + iPXE signing Power-on /Reset Chipset/CPU BIOS/UEFI firmware PXE/iPXE OS Kernel Userspace Baseboard management controller (BMC) iPXE scripts Platform Secure Boot/ HW root of trust UEFI Secure Boot
- 14. ■ Network boot (netboot) via DM ■ Default boot strategy for metals ■ Boot artifacts retrieved just-in-time ■ Local disk boot (localboot) from disk EFI partition ■ Needed to boot first DM in datacenter ■ Fallback strategy for metals if no DM is available ■ Boot artifacts are synced every time configuration management runs (~ 3 hours) Boot strategies
- 15. Architecture Control plane colo Config mgmt master Vault Boot information sources “Primary” DM Metals DMs Edge colo Boot render flow Boot flow
- 16. Design Evolution
- 17. Challenges ■ Too many nodes trusted to render and sign boot artifacts ■ Ideally go from all DMs to just a single identity ■ DMs have failover but it’s not elastic ■ DMs configuration management update time is dominated by boot artifact handling ■ Localboot pull model is costly and inefficient
- 18. Requirements ■ Must be able to generate artifacts for all nodes ■ Highly available service ■ Tolerant to the loss of part of/entire control plane ■ Some degradation is acceptable in extreme circumstances
- 19. Architecture Internal K8S cluster Boot service Vault Boot information sources DMs Metals Edge colo Internal backup K8S cluster Boot service Vault Boot information sources Internal S3 cluster Backup cloud S3 cluster
- 20. ■ Reduced trust domain ■ Only boot service is able to render and sign dynamic artifacts ■ Static artifacts are signed once and served from S3 afterwards ■ DM configuration management gains ■ High Availability and Elasticity ■ K8s service written in Go ■ Load balanced across multiple instances ■ Fallback to public cloud S3 for default artifacts if all K8S out ■ Localboot adopts push model ■ Only rendered and updated if node configuration changed Boot service
- 21. Future Work
- 22. Future Work ■ Cryptographically verify all executable code ■ Eliminate DMs completely ■ TPMs!
- 23. Thank you! João Pedro Lima jlima@cloudflare.com jopelima in/joaopedropaulinolima Matt Fleming mfleming@cloudflare.com fleming_matt mfleming