stackconf 2022: Cluster Management: Heterogeneous, Lightweight, Safe. Pick Three
0 likes17 views
The document discusses CI/CD pipelines and cluster management using OCaml, focusing on tools like Ocurrent for pipeline description, Obuilder for lightweight sandboxed builds, and Ocluster for worker management. It highlights the integration of webhooks, automatic caching, and reproducibility in builds, emphasizing the importance of a secure and maintainable programming environment. Resources and links to further documentation and CI/CD tools are provided for developer empowerment.
![OCurrent: build a Docker image from a git repo
head head commit build … run
let pull = false
let timeout = Duration.of_min 50
(* Run "docker build" on the latest commit in Git repository
* [repo]. *)
let pipeline ~repo () =
let src = Git.Local.head_commit repo in
let image = Docker.build ~pull ~timeout (`Git src) in
Docker.run image ~args:["./skynet.exe"; "--help"]
6](https://image.slidesharecdn.com/antonindecimoclustermanagementheterogeneous-lightweight-safe-220804141235-11487105/85/stackconf-2022-Cluster-Management-Heterogeneous-Lightweight-Safe-Pick-Three-6-320.jpg)
stackconf 2022: Cluster Management: Heterogeneous, Lightweight, Safe. Pick Three
- 1. CI/CD Pipelines and Cluster Management: Heterogeneous, Lightweight, Safe OCurrent, OBuilder, OCluster Antonin Décimo _ @Rucikir a @MisterDA July 19, 2022 Tarides https://tarides.com 1
- 2. Tarides & OCaml The OCaml programming language: general-purpose, industrial-strength, expressive, safe. let rec total t = match t with | Leaf -> 0 | Node (l, x, r) -> total l + x + total r (* val total : int tree -> int = <fun> *) let rec flip t = match t with | Leaf -> Leaf | Node (l, x, r) -> Node (flip r, x, flip l) (* val flip : 'a tree -> 'a tree = <fun> *) • type safe + memory safe = secure by construction! • maintainable programs 2
- 3. opam: the OCaml Package Manager A package manager and a package archive: • ~3900 different packages, ~24000 versions • explicit version constraints Goals: • test the ecosytem at scale • empower the developers 3
- 4. CI Pipelines opam-health-check weekly check of all ~24000 opam packages http://check.ocamllabs.io/ opam-repo-ci check packages for inclusion in the archive opam-docs-ci build and publish documentation for all packages ocaml-ci CI for OCaml projects (~430 projects) https://ci.ocamllabs.io/github/MisterDA/ocurrent docker-base-images OCaml & opam Docker images (19 systems × 13 OCaml versions) https://images.ci.ocaml.org/ 4
- 5. CI Pipelines in OCaml! OCurrent: an OCaml embedded Domain Specific Language to describe pipelines. • no more YAML or embedded JavaScript! • expressiveness of a real programming language • leverage the entire OCaml ecosystem https://github.com/ocurrent/overview 5
- 6. OCurrent: build a Docker image from a git repo head head commit build … run let pull = false let timeout = Duration.of_min 50 (* Run "docker build" on the latest commit in Git repository * [repo]. *) let pipeline ~repo () = let src = Git.Local.head_commit repo in let image = Docker.build ~pull ~timeout (`Git src) in Docker.run image ~args:["./skynet.exe"; "--help"] 6
- 7. OCurrent libraries current_incr • define changeable values and changeable computations • keep tracks of changes and propagate them • automatic caching current_* plugins: a toolbox for CI/CD pipelines 7
- 8. OCurrent plugins: monitoring inputs • git events Current_git.clone : string -> Commit.t Current.t • webhooks from GitHub and GitLab Current_github.Api.ci_refs : Current_github.Repo_id.t -> Commit.t list Current.t • new Docker images • web events 8
- 9. OCurrent plugins: outputs • Docker images • Notifications (Slack, Matrix, build statuses, ...) Current_slack.post : channel -> key:string -> string Current.t -> unit Current.t • Artifacts and logs • Web UI 9
- 10. Inspectable pull debian-11 4.14 x86_32 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.1 ✔ Analyse ✔ pull debian-11 4.03 x86_64 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.13 x86_64 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.14 x86_32 opam-2.0 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.0 ✔ pull debian-11 4.03 x86_64 opam-2.0 ✔ opam-vars ✔ pull debian-11 4.13 x86_64 opam-2.0 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.0 ✔ opam-vars ✔ ocaml/opam-repository:refs/heads/master head head commit debian-11-4.13 debian-11-4.14 (lint-opam) (lint-doc) (lint-fmt) build ✔ summarise build ✔ build ✔ build ✔ build ✔ 10
- 11. Reproductible All jobs generate equivalent Dockerfiles git clone --recursive "https://github.com/MisterDA/ocurrent.git" -b "master" && cd "ocurr cat > Dockerfile <<'END-OF-DOCKERFILE' FROM ocaml/opam@sha256:c9b4f14cd425a623c4ed33182b3a845175aa7494578d6997fbb9c71d4f2a8135 # debian-11-4.14 USER 1000:1000 RUN sudo ln -f /usr/bin/opam-2.0 /usr/bin/opam WORKDIR /src RUN sudo chown opam /src RUN cd ~/opam-repository && (git cat-file -e 8270cc5e433a2e14831089f1129059ef1077e5e5 || COPY --chown=1000:1000 current_web.opam current_slack.opam current_rpc.opam current_gitla RUN opam pin add -yn current_web.dev './' && opam pin add -yn current_slack.dev './' && o ENV DEPS="alcotest.1.5.0 alcotest-lwt.1.5.0 angstrom.0.15.0 ansi.0.5.0 asetmap.0.8.1 asn1 RUN opam depext --update -y current_web.dev current_slack.dev current_rpc.dev current_git RUN opam install $DEPS COPY --chown=1000:1000 . /src/ RUN opam exec -- dune build @install @check @runtest && rm -rf _build END-OF-DOCKERFILE docker build . 11
- 12. Obuilder: execute jobs A lightweight Docker build: takes a build script and performs build steps in a sandboxed environment. Linux Windows macOS Cache BTRFS/ZSF Docker images Rsync Sandbox runc Docker run User-level isolation 12
- 13. Obuilder jobs Docker-like syntax or calls to obuilder-spec library ((from ocaml/opam@sha256:c9b4f14cd425a623c4ed33182b3a845175aa7494578d6997fbb9c71d4f2a8135 (comment debian-11-4.14) (user (uid 1000) (gid 1000)) (run (shell "sudo ln -f /usr/bin/opam-2.0 /usr/bin/opam")) (workdir /src) (run (shell "sudo chown opam /src")) (run (cache (opam-archives (target /home/opam/.opam/download-cache))) (network host) (shell "cd ~/opam-repository && (git cat-file -e dcf32445c6c5322cbd5891bf0aa3cb6ba4 (copy (src current_web.opam current_slack.opam current_rpc.opam current_gitlab.opam curr (dst ./)) (run (network host) (shell "opam pin add -yn current_web.dev './' && opam pin add -yn current_slack.de (env DEPS "alcotest.1.6.0 alcotest-lwt.1.6.0 angstrom.0.15.0 ansi.0.5.0 asetmap.0.8.1 as (run (cache (opam-archives (target /home/opam/.opam/download-cache))) (network host) (shell "opam depext --update -y current_web.dev current_slack.dev current_rpc.dev c (run (cache (opam-archives (target /home/opam/.opam/download-cache))) 13
- 14. Pipelines with OCurrent! • integrated with webhooks and services • native and efficient build and sandboxing • automagic caching • reproducible • inspectable 14
- 15. OCluster: the cluster management system • manages pools of workers • a scheduler accepts jobs from clients and distributes then to workers • caching across machines • logs forwarding • global monitoring • estimated time of execution • communication using Cap'n Proto (RPC protocol) linux-x86_64 windows-x86_64 linux-arm64 worker 1 worker 2 worker 3 worker 4 worker 5 scheduler ocaml-ci 15
- 16. Refs & Thanks! All three independent components: OCurrent the eDSL to describe pipelines OBuilder the sandboxed build engine OCluster the cluster management system See our CI/CD pipelines at https://github.com/ocurrent/overview ! Started by Thomas Leonard and supported by contributors at Tarides. Thanks! any questions? 16