![kind: DaemonSet
...
hostNetwork: true
containers:
- name: condor-startd
image: .../cloud/condor-startd
command: ["/usr/sbin/condor_startd", "-f"]
securityContext:
privileged: true
livenessProbe:
exec:
command:
- condor_who
Sched
Negotiator
Collector
Host
StartD
...
StartD
...
StartD
...
kubefed init fed --host-cluster-context=condor-host ...
kubefed join --context fed tsystems
--host-cluster-context condor-host --cluster-context tsystems](https://image.slidesharecdn.com/kubeconeurope2018-cern-180507122303/85/Multi-Cloud-Federated-Kubernetes-at-CERN-15-320.jpg)
Multi-Cloud Federated Kubernetes at CERN
- 1. Multi-Cloud Federated Kubernetes at CERN Clenimar Filemon clenimar@lsd.ufcg.edu.br Ricardo Rocha ricardo.rocha@cern.ch
- 2. Founded in 1954 What is 96% of the universe made of? Fundamental Science Why isn’t there anti-matter in the universe? What was the state of matter just after the Big Bang?
- 7. ~40 MHz ~ 1PB/sec L1 Trigger ~ 100 kHz HL Trigger Collisions Hardware Filter Software Filter ~ 1 kHz Raw Data ~ 1-10 GB/s Huge Data Still Big Still Big
- 8. 320 000 Cores 3 300 Users 4 300 Projects 10 000 Hypervisors 210 Kubernetes Clusters 250 Petabytes
- 9. 200+ Sites 700 000 Cores ~400 000 Jobs Distributed Computing ~30 GiB/s CERN T1 T2 ... ... ... ... ... ... ... ... Reconstruction Calibration Simulation Analysis
- 10. CMS Higgs Event, May 2012 ATLAS Higgs Analysis, May 2012
- 11. Motivation for Federation Periodic load spikes International Conferences, Reconstruction Campaigns Simplification Monitoring, Lifecycle, Alarms Deployment Uniform API, Replication, Load Balancing Use Cases: CERN Batch System, RECAST Analysis
- 12. Sched Collector Negotiator StartD AcctGroup = "ATLAS" JobPrio = 0 RequestCpus = 2 RequestMemory = 4260 ... CERNEnvironment = “production” Datacenter = “meyrin” HasMPI = true TotalCpus = 8 TotalMemory = 22500 ... Matchmaking with ClassAds Fair Share Preemption Running Virtualized Extensive Experience in HEP External Storage and Networking
- 13. Sched Collector Negotiator StartD AcctGroup = "ATLAS" JobPrio = 0 RequestCpus = 2 RequestMemory = 4260 ... CERNEnvironment = “production” Datacenter = “meyrin” HasMPI = true TotalCpus = 8 TotalMemory = 22500 ... Matchmaking with ClassAds Fair Share Preemption Running Virtualized Extensive Experience in HEP External Storage and Networking
- 14. Sched Negotiator Collector Host kubefed init fed --host-cluster-context=condor-host ...
- 15. kind: DaemonSet ... hostNetwork: true containers: - name: condor-startd image: .../cloud/condor-startd command: ["/usr/sbin/condor_startd", "-f"] securityContext: privileged: true livenessProbe: exec: command: - condor_who Sched Negotiator Collector Host StartD ... StartD ... StartD ... kubefed init fed --host-cluster-context=condor-host ... kubefed join --context fed tsystems --host-cluster-context condor-host --cluster-context tsystems
- 16. REANA / RECAST Reusable Analysis Platform Workflow Engine (Yadage) Each step a Kubernetes Job Integrated Monitoring & Logging Centralized Log Collection https://github.com/reanahubhttps://github.com/recast-hep https://github.com/diana-hep/yadage
- 18. Thank You Great Community, Amazing Tools Credits CERN OpenStack Cloud and Batch teams (Spyros Trigazis and all) Lukas Heinrich, REANA / RECAST Kelsey Hightower
Editor's Notes
- #3: Dark energy + dark matter Quark gluon plasma, moments after the big bang
- #4: Location, lake, alps mont blanc, swiss-french border Complex of accelerators, higher and higher energy. 27km circumference Two beams of protons travelling on different directions, close to speed of light
- #5: Almost 10.000 magnets Kept in the ring thanks to these superconducting magnets Temperature kept at 1.9K (-271 celsius) to keep superconducting properties
- #6: Sibling of ATLAS, with similar goals but different design 14.000 tons 20 meters long, 15x15
- #7: Anti-matter Decelarator, creates anti-atoms to better understand its properties AMS experiment, launched on mission STS-134 (penultime shuttle mission), measures antimatter in cosmic rays
- #9: 2 floors Historical building, 50 years old, from mainframes to racks Internet backbone, biggest in late 80s and early 90s
- #10: Hierachical system, few big T1s and many smaller T2s
- #11: Not a physicist, but learned to look for patterns in plots
- #13: Based on htcondor, which HEP has decades of experience operating Component description, requests and resources published as classads Advanced functionality (fair share, pre-emption) Currently running mostly on virtualized resources Important: htcondor relies on an established storage and net infrastructure, handling compute only (which is what we try to federate)
- #14: StartD is our first containerization goal, deployed at scale
- #15: Host cluster, with the condor control plane One command only to establish federation
- #16: StartD deployment as a daemonset, meaning we get one instance on every host Clusters are added again with one single command CVMFS caching for software distribution speed-up Data access and networking outside the scope of this exercise