Parallel builds in Eclipse IDE workspace - EclispeCon Europe 2018
1 like1,438 views
The document discusses the implementation and benefits of parallel builds in Eclipse workspaces, emphasizing the importance of appropriate scheduling rules to minimize conflicts and increase parallelism. It highlights the issues with legacy builders that use the workspace root as their scheduling rule, which can block other builds and limit efficiency. Case studies demonstrate the effectiveness of applying specific scheduling rules, particularly a null scheduling rule, to enhance build performance and reduce build time.
Parallel builds in Eclipse IDE workspace - EclispeCon Europe 2018
- 1. Parallel builds in Eclipse Workspaces Value, making-of and optimal usages Mickael Istria - @mickaelistria CC0 – No rights reserved
- 2. Demo Tool For the demo/talk we use https://github.com/mickaelistria/eclipse-ide-parallel-builds-demo which – Defines the WaitBuilder – Allows to generate build Gantt chart – Allows to generate signature of build output (compare output) – Allows to control scheduling rule for JDT and M2E builders
- 3. Demo scenario 6 projects with WaitBuilder waiting for some time (Thread.sleep()) – Projects deps directly configured in .project – Builder duration configured in waitBuilder.properties (default 1000ms) – Builder schedulingRule configured in waitBuilder.properties (default current project) can be set to workspace root or some other project.
- 4. The basics: legacy (pre-Photon) Builds can only be sequential
- 5. The basics: Paralleling ● Create 1 job per project build ● Throttle on limited amount of threads using JobGroup API ● Preference used by good old Workspace.build() API ● Default in Photon and 2018-09 is disabled (nbThreads == 1)
- 6. The basics: Parallelized No dependencies, isolated scheduling rules
- 7. The basics: Benefits ● Shorter full time to completion (& less energy consumed) ● Faster availability of each project – A long project or chain of projects build doesn’t block other projects for too long – If you have handlers on project builds completion, they’re called way earlier ● Applies to any workspace-build solutions, including non- IDE offerings like JDT-LS
- 8. Relying on jobs→Usable progress View
- 9. Dependency Graph: no parallel Dependency graph is computed, and turned into a sequence (via topological sort) p3,p1,p2,p5,p4,p6
- 10. Dependency graph: Parallel Graph is used directly and jobs are scheduled for builds as soon as all ancestors are built
- 11. About Scheduling Rules ● 2 operations using conflicting scheduling rules rule1.isConflicting(rule2) can NOT run in parallel ● Scheduling rules are usually Resource Path which conflict when one path is a parent of the other: – a/b and a/b/c are conflicting and those scheduling rules prevent jobs for running in parallel – a/b and a/c are not conflicting and those schduling rules can allow jobs to run in parallel ● Builders are expected to implement IncrementalProjectBuilder.getRule(…) Default is workspace.getRoot() Here, p1 build rule conflicts with all other build rules → p1 won’t build in // of other projects
- 12. Scheduling rules don’t change sequential scheduling No jobs → no issue about conflicting scheduling rules
- 13. So why scheduling rules on builders? “Safety”: Scheduling rule are here to prevent some other thread to change critical content while builder is running. The scheduling rule defines what must NOT be changed externally while builder is running. GOLDEN RULE: To minimize conflicts and allow more parallelism, set the narrowest rule your builder can deal with.
- 14. (Bad) State of art Many builders do not specify a scheduling rule (defaulting to workspace root, conflicting with every workspace operation) → Building a project locks the whole workspace! Builder uses root as SR, conflicts with whole WS Change file in another unrelated Project + Save
- 15. (Bad) examples ● JDT Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=531554 ● M2E Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=538461 ● PDE Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=531555 … Use default workspace scheduling rule, preventing parallel execution and blocking user actions in unrelated files!
- 16. With finer scheduling rule... More specific scheduling rule, only necessary content is locked by builder. For example, just using getProject() as scheduling rule can often be enough. Builder uses project as SR, conflicts with only project content Change file in another unrelated Project + Save File just saved!
- 17. GOLDEN RULE: To minimize conflicts and allow more parallelism, set the narrowest rule your builder can deal with.
- 18. 2018-09, a new hope... JDT and m2e have a (hidden) preference to control the builder’s scheduling rule and let it be null. Hidden, but available for testing with the test wizard, or with typical preference customization strategy (API, plugin_customization.ini…) in your Eclipse-based package (like JDT-LS). Try them!
- 19. Scheduling rules and parallel builds All rules conflicting → No parallel builder execution
- 20. Current impl does not pass rules to schedule jobs and use rules at job exec. Some jobs are started for nothing ( ) and can block other builds. But could do a bit better
- 21. Now you got it ● 1 project build → 1 job ● Throttling with JobGroup ● Scheduled according to dependency graph ● Honoring scheduling rules GOLDEN RULE: To minimize conflicts and allow more parallelism, set the narrowest rule your builder can deal with.
- 22. Highlighted cases ● Long build chain vs standalone projet ● Dependency cycles ● 1 project builder uses workspace root as scheduling rule
- 23. Dependency chain vs standalone project 1 thread 4 threads p6 is available much much sooner Some project output available much sooner
- 24. With cycles, sequential Compute a best sequential order (using “Strongly connected component”) and apply it p1,p2,p3,p4,p5,p6
- 25. With cycles, parallel 1. Process graph until no node is ready (symptom of cycle) 2. Then process the 1st waiting node in computed sequential order 3. Then back to regular graph processing. 1 2 3
- 26. Workspace root scheduling rule ● For backward compatibility, if any builder uses workspace root scheduling rule, disable parallel builds (as the scheduler will use this rule in its thread) ● Anyway, workspace root would conflict with about everything… ● So most builders currently prevents parallel builds
- 27. GOLDEN RULE: To minimize conflicts and allow more parallelism, set the narrowest rule your builder can deal with.
- 28. Case studies 1. dotnet build builder 2. JavaBuilder (JDT) and MavenBuilder (m2e)
- 29. Case Study #1 6 independent C# projects building with dotnet build (builder scheduling root == current project) Sequential build takes 39s Parallel on 4 threads takes 26s Actual gain: 33% (cool, but...) Expected gain: 66% ?
- 30. Case study #1 - lesson learned Most real world operation have bottlenecks (I/O, network…) making concurrent executions slower. Sequential 4 threads But still a major win!
- 31. Case study #2: Java (JDT & m2e) JavaBuilder and MavenBuilder do use default workspace root scheduling rule by default → Disable/prevents parallel builds
- 32. GOLDEN RULE: To minimize conflicts and allow more parallelism, set the narrowest rule your builder can deal with.
- 33. Case study #2: Java (JDT & m2e) Applying the GOLDEN RULE on JavaBuilder and MavenBuilder shows that null might be a good scheduling rule https://bugs.eclipse.org/bugs/show_bug.cgi?id=531554#c22 . null is the perfect value to avoid conflict and enabled parallelism.
- 34. Effect of null scheduling rule for JDT and m2e effect: story #1 Legacy Scheduling Rule null Scheduling Rule Build with 4 threads
- 35. Effect of null scheduling rule for JDT and m2e effect: story #2 Legacy Scheduling Rule null Scheduling Rule Build with 4 threads
- 36. Mickael Istria - @mickaelistria CC0 – No rights reserved And remember GOLDEN RULE: To minimize conflicts and allow more parallelism, set the narrowest rule your builder can deal with. Questions?
Editor's Notes
- #2: Pre-reqs: Eclipse IDE workspace with org.eclipsecon.editor in it open Build Jobs == 1 Project Explorer + (empty) editor area + progress view JDT tweaked to enable graphical report