SlideShare a Scribd company logo

Migrating to a bazel based CI system: 6 learnings

O
Or Shachar

The document outlines six key learnings from migrating a CI system to a Bazel-based framework, focusing on using migration tools, measuring systems accurately, and enhancing local development experience. The author emphasizes the importance of optimizing speed, running parallel CI pipelines, and successfully managing poly-repo challenges. The overall goal is to improve build stability and deployable speeds while transitioning from a complex Maven setup to a streamlined Bazel environment.

Software
Related topics:
Continuous Integration
1 of 82
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
Or Shachar, Backend Engineer, Wix-CI Migrating to a Bazel-based CI system: 6 learnings ors@wix.com linkedin/or-shachar github.com/or-shachar
Check emailCoffee break Ping pong Reading news CAll your mom <Pending mode>
Check emailCoffee break Ping pong Reading news CAll your mom Building...
Hi, I'm Or. CI Build with Ant and Makefiles Maven CI Pipelines Backend Engineering … Backend Engineering for2017 2011 2014 2019
Building... Building... We migrated to Bazel
Codebase + build definition Build server feedback Build tool deployables
~5M LOC > 1000 repos 2000+ Maven modules 2017 Teamcity Maven feedback deployables
~5M LOC > 1000 repos 2000+ Maven modules 2017 Teamcity Maven feedback deployables building / broken blockedblocked blocked blocked
2017 Master stability Push to feedback Push to release Often broken Can take 1 hour Can take hours
And so we wait...
Find new machinery that’s stable, fast, and can scale.
Master stability Push to feedback Push to release 2019 Often broken Can take 1 hour Can take hours 2017 Master stability Push to feedback Push to release Mostly stable 5-10 minutes 15-25 minutes
6 LEARNINGS FROM MIGRATING OUR BUILD SYSTEM TO BAZEL.
#1 Use a Migration tool Massage it, validate it.
Migration 101 Bazel>>Maven >>
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Bazel>>Maven >>
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity 1:1:1 idiom - one target per one directory, representing a single package
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules java_library java_library java_test java_image
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> internal deps java_library java_library java_test java_image
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary @junit @jetty @guava20 Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> external deps java_library java_library java_test java_image @guava28
. ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary @junit @jetty Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> external deps java_library java_library java_test java_image @guava28
. [WORKSPACE.bazel] ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] [BUILD.bazel] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] [BUILD.bazel] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] [BUILD.bazel] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] [BUILD.bazel] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> Add deps >> Write Bazel files
Migrating to a bazel based CI system: 6 learnings
Code graph >> Targets graph Third party definitions >> >> ✓ wix/exodus >>
Automatic tooling is not enough >> >> ! bazel build bazel test migration trustees
✖ Missing compile/runtime deps ✖ Third party alignment ✖ Trying to read from ./src/test/resources ✖ Tests that try to download during test runtime ✖ More... TECHNICAL DEBT == MORE WORK exodus/troubleshooting
>> >> >> ?! >> bazel build bazel test Did I lose anyting?
>> << Test reports Test reports compare! >> >> >> ?! >>
>> >> >> ?! >> >> ✓ >> Exodus Overrides
1 Use a migration tool
#2 Measure your system but measure right.
2 3 4 5 1 2 3 4 1 2 3 1 2 7 0 1 0 1 WIX FRAMEWORK (2017) Lines of code 2 5 0 7 Maven modules 3 5 2 3 Bazel Targets 2 5
89% reduction Correct Build Times in Minutes * 8-cores machine
89% reduction Not Good Enough Correct Build Times in Minutes * 8-cores machine
Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
1 Use a migration tool 2 Measure the system
#3 Local dev experience don’t forget your users!
CI & Console runs 🤝 Build Tool
✖ Dependency management ✖ Code completion ✖ Refactoring ✖ MacOS cache issues Developers and IDEs 🤝 Build Tool
Local-dev velocity matters! Bad CILocal dev CIBad Local dev
✓ Bazel training ✓ IDE issues ✓ Develop solutions! Think local development! CIOkay Local dev Wix Bazel Local dev team: ○Tooling alignment ○Dev Cache ○Wix IntelliJ Bazel Plugin
1 Use a migration tool 2 Measure the system 3 Think local dev experience
#4 Introduce a new CI pipline but keep the old one too.
Code push Trigger Artifact Deployment backoffice inform GA/Testbed deployment CI Pipeline Production Server
Code push Trigger Artifact Deployment backoffice Inform GA/Testbed Deployment CI Pipeline Production Server Replace This part
✓ Remote cache and remote execution ✓ PR prechecks (no push to master) ✓ Automatic discovery of deployables ✓ Cloud based build server - no build queues New Bazel CI Pipeline
Inform Code push Trigger Artifact Deployment backoffice GA/Testbed Deployment CI Pipeline Production Server Trigger Bazel Artifact
Bazel Maven ✓ Two CI pipelines, side by side Self-service switch Bazel Maven✓
1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines
#5 Optimize for Speed e.g., remote execution cache
How to make faster bazel builds?
✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput 5-8 minutes - fully cached build
✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput ✓ Repository cache - Saves downloading http binaries ✓ Docker cache - Saves downloading docker images
✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput ✓ Repository cache - Saves downloading http binaries ✓ Docker cache - Saves downloading docker images ✓ Bazel flags - builds without the bytes, multithread digest...
When we started we compiled Bazel from sources Bazel keeps evolving Each release brought a lot of value, but also required us to adjust Tap to important issues, influence the priorities Become a Bazel activist!
1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed
#6 Migrating poly repo is hard but possible!
Bazel works well with mono repo All code in one place Single bazel execution ✓ Single CI job
It's too much for me! I'm not ready yet Mono-repo on Git You change too much! Mono RepoMercurial Poly-Repo on Git ✓
from 1000+ repos ...to just about 50.
Goal: Virtual mono repository java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@framework//src/main/java/com/wix/external", ], ) git_repository( name = "framework", url = "git@github.com:wix/framework.git", commit = "<auto-updated>", )
Virtual mono-repo: Virtual Mono-Repo Service holds all repos of wix with their latest HEAD version Bazel wrapper generates a local bzl file with `git_repository` definition of all repositories Each WORKSPACE file loads the definition from the generated bzl file. framework(031d53ce983d26a861d053c760190de669aa458d) users(1202d41c7dc0739deddde3176c7de34ed9b978b7) events(ab6d1d07477e3742fbc9f9c1e81e0ec402650b1b) external_repos.bzl git_repository( name = "framework", Commit = "031d53ce983d2 ) ... WORKSPACE.bazel 1 2 3
~50 migrations
External Dependencies Guava:25 Commons:7 Junit:3.2 wix-framework:1.0-SNAPSHOT wix-utils:1.0.SNAPSHOT ...
External Dependencies Guava:25 Commons:7 Junit:3.2 wix-framework:1.0-SNAPSHOT wix-utils:1.0.SNAPSHOT ... java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], )
Migration in "Isolation Mode" ✓ Faster move to Bazel CI ✓ Faster feedback ✓ Faster deployables ✖ Keep Maven builds green ✖ Reproducibility hit ✖ Stale SNAPSHOT locally
Isolation Mode Social Mode java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) scala_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@framework//src/java/com/wixpress/lib", ], )
✖ Complex ✖ Non-revertible java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@framework//src/java/com/wixpress/lib", ], )
java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) external_binary( name = "com_wix_framework", artifact = "com.wix.framework:1.0-SNAPSHOT" ) external_source( name = "com_wix_framework", exports = [ "@framework//src/java/com/wixpress/lib" ] ) workspace_rule
external_binary external_source @com_wix_framework Current repo in social mode? Requested module in social mode? no no yes yes
Social mode ✓ Global Third party alignment ✓ Missing runtime deps ✓ Testonly targets ✓ ...
1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed 6 Migrate poly-repo
Where are we now?
Safe haven CLOSED ✓ No more SNAPSHOT consumption ✓ All deployable from bazel ✓ All developers use bazel only
1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed 6 Migrate poly-repo
Thank You ors@wix.com 6 LEARNINGS FROM MIGRATING OUR BUILD SYSTEM TO BAZEL
Q&A ors@wix.com

More Related Content

PPTX
Migrating to a Bazel-based CI System: 6 Learnings - Or Shachar
Wix Engineering
 
PPT
Python Deployment with Fabric
andymccurdy
 
PDF
Integrated Cache on Netscaler
Mark Hillick
 
PDF
Deploying Symfony | symfony.cat
Pablo Godel
 
PDF
Australian OpenStack User Group August 2012: Chef for OpenStack
Matt Ray
 
PDF
Ansible v2 and Beyond (Ansible Hawai'i Meetup)
Timothy Appnel
 
PDF
Portland PUG April 2014: Beaker 101: Acceptance Test Everything
Puppet
 
PPTX
Automated Deployment with Fabric
tanihito
 
Migrating to a Bazel-based CI System: 6 Learnings - Or Shachar
Wix Engineering
 
Python Deployment with Fabric
andymccurdy
 
Integrated Cache on Netscaler
Mark Hillick
 
Deploying Symfony | symfony.cat
Pablo Godel
 
Australian OpenStack User Group August 2012: Chef for OpenStack
Matt Ray
 
Ansible v2 and Beyond (Ansible Hawai'i Meetup)
Timothy Appnel
 
Portland PUG April 2014: Beaker 101: Acceptance Test Everything
Puppet
 
Automated Deployment with Fabric
tanihito
 

What's hot (20)

PDF
Ansible 實戰：top down 觀點
William Yeh
 
PDF
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
Open Source Consulting
 
PDF
Modern PHP Ch7 Provisioning Guide 導讀
Chen Cheng-Wei
 
KEY
PyCon US 2012 - Web Server Bottlenecks and Performance Tuning
Graham Dumpleton
 
PPTX
Automated Deployments with Ansible
Martin Etmajer
 
PDF
Workshop: Know Before You Push 'Go': Using the Beaker Acceptance Test Framewo...
Puppet
 
PDF
Homer - Workshop at Kamailio World 2017
Giacomo Vacca
 
KEY
modern module development - Ken Barber 2012 Edinburgh Puppet Camp
Puppet
 
PDF
Symfony Live NYC 2014 - Rock Solid Deployment of Symfony Apps
Pablo Godel
 
PDF
The Modern Developer Toolbox
Pablo Godel
 
KEY
Puppet for dummies - ZendCon 2011 Edition
Joshua Thijssen
 
PPTX
Go Faster with Ansible (PHP meetup)
Richard Donkin
 
KEY
London devops logging
Tomas Doran
 
PPTX
Automate DBA Tasks With Ansible
Ivica Arsov
 
PDF
快快樂樂用Homestead
Chen Cheng-Wei
 
PDF
Ansible new paradigms for orchestration
Paolo Tonin
 
PDF
Zero Downtime Deployment with Ansible
Stein Inge Morisbak
 
PDF
Getting started with Ansible
Ivan Serdyuk
 
PDF
Taking Apache Camel For A Ride
Bruce Snyder
 
PDF
Mitchell Hashimoto, HashiCorp
Ontico
 
Ansible 實戰：top down 觀點
William Yeh
 
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
Open Source Consulting
 
Modern PHP Ch7 Provisioning Guide 導讀
Chen Cheng-Wei
 
PyCon US 2012 - Web Server Bottlenecks and Performance Tuning
Graham Dumpleton
 
Automated Deployments with Ansible
Martin Etmajer
 
Workshop: Know Before You Push 'Go': Using the Beaker Acceptance Test Framewo...
Puppet
 
Homer - Workshop at Kamailio World 2017
Giacomo Vacca
 
modern module development - Ken Barber 2012 Edinburgh Puppet Camp
Puppet
 
Symfony Live NYC 2014 - Rock Solid Deployment of Symfony Apps
Pablo Godel
 
The Modern Developer Toolbox
Pablo Godel
 
Puppet for dummies - ZendCon 2011 Edition
Joshua Thijssen
 
Go Faster with Ansible (PHP meetup)
Richard Donkin
 
London devops logging
Tomas Doran
 
Automate DBA Tasks With Ansible
Ivica Arsov
 
快快樂樂用Homestead
Chen Cheng-Wei
 
Ansible new paradigms for orchestration
Paolo Tonin
 
Zero Downtime Deployment with Ansible
Stein Inge Morisbak
 
Getting started with Ansible
Ivan Serdyuk
 
Taking Apache Camel For A Ride
Bruce Snyder
 
Mitchell Hashimoto, HashiCorp
Ontico
 
Ad

Similar to Migrating to a bazel based CI system: 6 learnings (8)

PDF
How to successfully migrate to Bazel from Maven or Gradle - JeeConf
Natan Silnitsky
 
PDF
How to successfully migrate to Bazel from Maven or Gradle - Riga Dev Days
Natan Silnitsky
 
PPTX
Building a Large Java Codebase with Bazel - Natan Silnitsky
Wix Engineering
 
PPT
Maven
Rajkattamuri
 
PDF
An Empirical Study of Build System Migrations in Practice (ICSM 2012)
Bram Adams
 
PDF
Bdsys icsm v3.5
SAIL_QU
 
PPT
Maven introduction in Mule
Shahid Shaik
 
PPT
Maven
Sunil Komarapu
 
How to successfully migrate to Bazel from Maven or Gradle - JeeConf
Natan Silnitsky
 
How to successfully migrate to Bazel from Maven or Gradle - Riga Dev Days
Natan Silnitsky
 
Building a Large Java Codebase with Bazel - Natan Silnitsky
Wix Engineering
 
Maven
Rajkattamuri
 
An Empirical Study of Build System Migrations in Practice (ICSM 2012)
Bram Adams
 
Bdsys icsm v3.5
SAIL_QU
 
Maven introduction in Mule
Shahid Shaik
 
Maven
Sunil Komarapu
 
Ad

Recently uploaded (20)

PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
PDF
System and Network Administraation Chapter 3
jaramharo
 
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PDF
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
PDF
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
System and Network Administration Chapter 2
jaramharo
 
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PDF
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
System and Network Administraation Chapter 3
jaramharo
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
Transform Your Business with a Software ERP System
Canada Software Company
 
System and Network Administration Chapter 2
jaramharo
 
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 

Migrating to a bazel based CI system: 6 learnings

  • 1. Or Shachar, Backend Engineer, Wix-CI Migrating to a Bazel-based CI system: 6 learnings ors@wix.com linkedin/or-shachar github.com/or-shachar
  • 2. Check emailCoffee break Ping pong Reading news CAll your mom <Pending mode>
  • 3. Check emailCoffee break Ping pong Reading news CAll your mom Building...
  • 4. Hi, I'm Or. CI Build with Ant and Makefiles Maven CI Pipelines Backend Engineering … Backend Engineering for2017 2011 2014 2019
  • 6. Codebase + build definition Build server feedback Build tool deployables
  • 7. ~5M LOC > 1000 repos 2000+ Maven modules 2017 Teamcity Maven feedback deployables
  • 8. ~5M LOC > 1000 repos 2000+ Maven modules 2017 Teamcity Maven feedback deployables building / broken blockedblocked blocked blocked
  • 9. 2017 Master stability Push to feedback Push to release Often broken Can take 1 hour Can take hours
  • 11. Find new machinery that’s stable, fast, and can scale.
  • 12. Master stability Push to feedback Push to release 2019 Often broken Can take 1 hour Can take hours 2017 Master stability Push to feedback Push to release Mostly stable 5-10 minutes 15-25 minutes
  • 14. #1 Use a Migration tool Massage it, validate it.
  • 16. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Bazel>>Maven >>
  • 17. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
  • 18. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
  • 19. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
  • 20. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity 1:1:1 idiom - one target per one directory, representing a single package
  • 21. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules java_library java_library java_test java_image
  • 22. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> internal deps java_library java_library java_test java_image
  • 23. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary @junit @jetty @guava20 Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> external deps java_library java_library java_test java_image @guava28
  • 24. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary @junit @jetty Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> external deps java_library java_library java_test java_image @guava28
  • 25. . [WORKSPACE.bazel] ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] [BUILD.bazel] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] [BUILD.bazel] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] [BUILD.bazel] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] [BUILD.bazel] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> Add deps >> Write Bazel files
  • 27. Code graph >> Targets graph Third party definitions >> >> ✓ wix/exodus >>
  • 28. Automatic tooling is not enough >> >> ! bazel build bazel test migration trustees
  • 29. ✖ Missing compile/runtime deps ✖ Third party alignment ✖ Trying to read from ./src/test/resources ✖ Tests that try to download during test runtime ✖ More... TECHNICAL DEBT == MORE WORK exodus/troubleshooting
  • 30. >> >> >> ?! >> bazel build bazel test Did I lose anyting?
  • 31. >> << Test reports Test reports compare! >> >> >> ?! >>
  • 32. >> >> >> ?! >> >> ✓ >> Exodus Overrides
  • 33. 1 Use a migration tool
  • 34. #2 Measure your system but measure right.
  • 35. 2 3 4 5 1 2 3 4 1 2 3 1 2 7 0 1 0 1 WIX FRAMEWORK (2017) Lines of code 2 5 0 7 Maven modules 3 5 2 3 Bazel Targets 2 5
  • 36. 89% reduction Correct Build Times in Minutes * 8-cores machine
  • 37. 89% reduction Not Good Enough Correct Build Times in Minutes * 8-cores machine
  • 38. Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
  • 39. Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
  • 40. Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
  • 41. 1 Use a migration tool 2 Measure the system
  • 42. #3 Local dev experience don’t forget your users!
  • 43. CI & Console runs 🤝 Build Tool
  • 44. ✖ Dependency management ✖ Code completion ✖ Refactoring ✖ MacOS cache issues Developers and IDEs 🤝 Build Tool
  • 45. Local-dev velocity matters! Bad CILocal dev CIBad Local dev
  • 46. ✓ Bazel training ✓ IDE issues ✓ Develop solutions! Think local development! CIOkay Local dev Wix Bazel Local dev team: ○Tooling alignment ○Dev Cache ○Wix IntelliJ Bazel Plugin
  • 47. 1 Use a migration tool 2 Measure the system 3 Think local dev experience
  • 48. #4 Introduce a new CI pipline but keep the old one too.
  • 51. ✓ Remote cache and remote execution ✓ PR prechecks (no push to master) ✓ Automatic discovery of deployables ✓ Cloud based build server - no build queues New Bazel CI Pipeline
  • 52. Inform Code push Trigger Artifact Deployment backoffice GA/Testbed Deployment CI Pipeline Production Server Trigger Bazel Artifact
  • 53. Bazel Maven ✓ Two CI pipelines, side by side Self-service switch Bazel Maven✓
  • 54. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines
  • 55. #5 Optimize for Speed e.g., remote execution cache
  • 56. How to make faster bazel builds?
  • 57. ✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput 5-8 minutes - fully cached build
  • 58. ✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput ✓ Repository cache - Saves downloading http binaries ✓ Docker cache - Saves downloading docker images
  • 59. ✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput ✓ Repository cache - Saves downloading http binaries ✓ Docker cache - Saves downloading docker images ✓ Bazel flags - builds without the bytes, multithread digest...
  • 60. When we started we compiled Bazel from sources Bazel keeps evolving Each release brought a lot of value, but also required us to adjust Tap to important issues, influence the priorities Become a Bazel activist!
  • 61. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed
  • 62. #6 Migrating poly repo is hard but possible!
  • 63. Bazel works well with mono repo All code in one place Single bazel execution ✓ Single CI job
  • 64. It's too much for me! I'm not ready yet Mono-repo on Git You change too much! Mono RepoMercurial Poly-Repo on Git ✓
  • 65. from 1000+ repos ...to just about 50.
  • 66. Goal: Virtual mono repository java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@framework//src/main/java/com/wix/external", ], ) git_repository( name = "framework", url = "git@github.com:wix/framework.git", commit = "<auto-updated>", )
  • 67. Virtual mono-repo: Virtual Mono-Repo Service holds all repos of wix with their latest HEAD version Bazel wrapper generates a local bzl file with `git_repository` definition of all repositories Each WORKSPACE file loads the definition from the generated bzl file. framework(031d53ce983d26a861d053c760190de669aa458d) users(1202d41c7dc0739deddde3176c7de34ed9b978b7) events(ab6d1d07477e3742fbc9f9c1e81e0ec402650b1b) external_repos.bzl git_repository( name = "framework", Commit = "031d53ce983d2 ) ... WORKSPACE.bazel 1 2 3
  • 70. External Dependencies Guava:25 Commons:7 Junit:3.2 wix-framework:1.0-SNAPSHOT wix-utils:1.0.SNAPSHOT ... java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], )
  • 71. Migration in "Isolation Mode" ✓ Faster move to Bazel CI ✓ Faster feedback ✓ Faster deployables ✖ Keep Maven builds green ✖ Reproducibility hit ✖ Stale SNAPSHOT locally
  • 72. Isolation Mode Social Mode java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) scala_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@framework//src/java/com/wixpress/lib", ], )
  • 73. ✖ Complex ✖ Non-revertible java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@framework//src/java/com/wixpress/lib", ], )
  • 74. java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) external_binary( name = "com_wix_framework", artifact = "com.wix.framework:1.0-SNAPSHOT" ) external_source( name = "com_wix_framework", exports = [ "@framework//src/java/com/wixpress/lib" ] ) workspace_rule
  • 75. external_binary external_source @com_wix_framework Current repo in social mode? Requested module in social mode? no no yes yes
  • 76. Social mode ✓ Global Third party alignment ✓ Missing runtime deps ✓ Testonly targets ✓ ...
  • 77. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed 6 Migrate poly-repo
  • 78. Where are we now?
  • 79. Safe haven CLOSED ✓ No more SNAPSHOT consumption ✓ All deployable from bazel ✓ All developers use bazel only
  • 80. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed 6 Migrate poly-repo
  • 81. Thank You ors@wix.com 6 LEARNINGS FROM MIGRATING OUR BUILD SYSTEM TO BAZEL