Building Mage-OS - MageTitans 2023

Mage-os.org 22. Apr. 2023
Building
Mage-OS
Or: DIY Magento Releases

Mage-OS is an open
initiative to ensure the
accessibility, longevity,
and success of the
Magento® platform
and ecosystem.
The Mage-OS
Distribution
is a community
owned fork
of Magento Open
Source.
Building a
Distribution

How are Mage-OS release packages built?

Back when we started: the first ideas…
Magento is many modules in a mono-repo
> Symfony uses splitsh/lite, maybe we can use it!
> There also are other existing tools…

First experiments
Splitting the magento repo into subrepos took ~36 hours!

Okay, something custom then!
“The Plan”
- Clone repo
- List all tags
- For each "version" tag…
- Iterate over all folders in app/code/Magento
- Zip the folder as a composer package using the name and version in the
composer.json file in each directory
- Use composer/satis to use the directory with the zips as an artifact type repo

Time to build!
Ivan: Let’s create a Rust repo-splitter that keeps the git repo
in-memory so no working copy is needed!
Vinai: Let’s keep it simple and write a shell script version, too!
> The more implementations we get, the better our understanding of the problem will be and the more
people will be able to work on it (rust, bash, js, …) <

One tool made it to production 🎉
The JavaScript implementation started as an
experiment.
Turned out to be good enough!
Also available as a docker image
magece/mageos-repo-js
to build your own Mage-OS repositories!

Beside the module & framework packages
magento/project-community-edition metapackage (e.g. 2.4.5-p2)
depends on ⬇️
magento/product-community-edition metapackage (e.g. 2.4.5-p2)
depends on ⬇️
EVERY MAGENTO PACKAGE included in 2.4.5-p2 in a specific version (e.g. 103.0.2)
and also ⬇️
magento/magento2-base (2.4.5-p2)

The magento2-base package
Contains base directory filesystem structure, i.e. app/code, bin/, dev/, ...

It depends on magento/magento-composer-installer.
It declares extra.map in composer.json.
The magento/magento-composer-installer
copies the extra.map files and folders
from the package into the project base.

There is no way to determine the extra.map contents programmatically.
From what is visible outside of Magento/Adobe, the map seems to be maintained manually.

Solution for mirror packages:
copy the magento2-base composer.json of
all past releases from upstream

There be Dragons Inconsistencies!
The git repos don’t match the old Magento releases exactly…
- Version mismatch in package composer.json and base package composer.json
- No version in package composer.json
- Missing release tags
- Branches with version name instead of a tag (did that branch change after the release??)
- For some releases, no commit in public repo matches the released code exactly
(file missing or file is different)

Workarounds
Each problem requires a
different workaround,
e.g. adding missing tags or files.

You did this for all versions of Magento 2?
Nope.
We only support releases since 2.3.7-p2.

Finally: it installs!

Why is there no 2FA for Admin login?

Ups! There is more than one git repo!
- https://github.com/mage-os/mirror-security-package
- https://github.com/mage-os/mirror-inventory
- https://github.com/mage-os/mirror-magento2-page-builder
- https://github.com/mage-os/mirror-inventory-composer-installer
- https://github.com/mage-os/mirror-adobe-ims
- https://github.com/mage-os/mirror-adobe-stock-integration
- https://github.com/mage-os/mirror-magento-composer-installer
- https://github.com/mage-os/mirror-composer-root-update-plugin
- https://github.com/mage-os/mirror-composer-dependency-version-audit-plugin
- https://github.com/mage-os/mirror-magento2-sample-data (not a direct dep)
- https://github.com/mage-os/mirror-commerce-data-export (not a direct dep)

Different repos, different Structures
- Single package repos, like mage-os/mirror-inventory-composer-installer
- Multiple packages in base, like mage-os/mirror-security-package
- Meta-package in _metapackage dir, like mage-os/mirror-security-package
- Special case subdirectory package, like
mage-os/mirror-magento2/tree/2.4-develop/lib/internal/Magento/Framework/Amqp
- Repos that change the directory structure, like mage-os/mirror-inventory
(first a clone of magento/magento2, then only app/code/Magento/* folders + _metapackage)

Additional dependencies!
The dependency on the packages from those repos are added to the magento/product-
community-edition package.

Bundled Core Extensions
- amzn/amazon-pay-and-login-magento-2-module
- dotmailer/dotmailer-magento2-extension
- klarna/m2-payments
- paypal/module-braintree
- temando/module-shipping
- vertex/product-magento-module
- yotpo/magento2-module-yotpo-reviews

Bundled Core Extensions
- Magento forked the public upstream repos
- These forks are not public
- Solution for mirror:
copy released packages from upstream
- Solution for Mage-OS distro:
no core bundled modules

Are you sure everything is working now?

Quality Assurance
- Install release from mirror.mage-os.org
- Install release from repo.magento.com
- Compare versions and files

We have mirror releases ✅

We have mirror releases ✅
Next step: nightly builds!

What is different from a mirror build?
Use the head of a branch instead of a git tag.
Figure out a nightly-build version for each package instead of relying on a version
in composer.json.

Nightly Package Versions
After considering many options, we now use
- get the latest release tag of the repo
- get the version of all packages
- if less than 4 parts are present: increment last part of the current version,
otherwise increment the last part
- if there is no stability suffix, add an -aYYYYMMDD suffix to version
- otherwise keep the stability suffix and add YYYYMMDD

What deps to use in the base package?
- clone repo
- run composer install
- use github.com/mage-os/php-dependency-list to scan every .php, .phtml and di.xml file
- For each referenced PHP class
- Use the composer autoloader to determine which package contains the class
- Record the package name and version as a composer dependency for the base package

Upstream Nightly Releases ✅
composer create-project magento/project-community-edition
--repository-url https://upstream-nightly.mage-os.org
--stability alpha ./current-nightly

Mage-OS Nightly Releases ✅
The same process as upstream-nightly works
for the mageos-nightly builds, too!

New upstream Magento Open Source release?
- copy project-community-edition, product-community-edition and base-package composer.json into
the respective resources/history/magento/* folder with the release version as the name
- run the build
- install and validate new version
- add workarounds for missing tags and such to build config, re-build and test until everything works
- PR to github.com/mage-os/generate-mirror-repo-js, run action against preview-mirror.mage-os.org
- Merge PR, new packages are built and deployed to mirror.mage-os.org

GitHub Actions

Time for a wee bit of code?

The Build Config
Everything is declared in
“the build config”.
Base config extended by build type
(mirror, upstream-nightly or mageos-nightly)
Config is processed by the build script to
generate the package ZIP files.

The Build Config

Next steps
Come! Pls halp!

Next steps
Active Projects:
- Generate a real Mage-OS release
- Faster Integration & Unit Tests - (hi Adam + Damien!)
- New Features - (hey Gowri + Ivan!)
- Content!

Join our Community
Adam Crossland (UK)
Alexander Buch (DE)
Andreas Mautz (DE)
Anton Siniorg (EE)
Arron Moss (UK)
Barbaros Selim Büyükelçi (TR)
Benno Lippert (DE)
Bernard Robbins (US)
Brent Peterson (US)
Christian Münch (DE)
Damien Retzinger (US)
Daniel Fahlke (DE)
Daniel Sloof (NL)
David Lambauer (DE)
Dominik Rammert (DE)
Erik Hanson (US)
Gowri Sankar (AU)
Jisse Reitsma (NL)
John Prendergast (IE)
Nicole Mentzen (DE)
Roland Haselager (NL)
Sergei Derzap (BY)
Tim Bucciarelli (US)
Timon de Groot (NL)
Vitaliy Golomozy (UA)
Willem Wigman (NL)
Wouter Steenmeijer (NL)
Thank you to the contributors!

Mage-OS
Fork it, Own it, Grow it, Better!
TOGETHER

Building Mage-OS - MageTitans 2023

More Related Content

What's hot (20)

Similar to Building Mage-OS - MageTitans 2023 (20)

More from vinaikopp (20)

Recently uploaded (20)

Building Mage-OS - MageTitans 2023

Editor's Notes