Design multi-manifest (aka multi-architecture or multi-RID) publishing

[baronfel]

It's possible for containers to be specified in a 'manifest list' - a set of container image manifests that represent the same application on different underlying OS/hardware configurations.

Fundamentally this would be something like a multitargeted build. For some selection of OS/OSVersions and Architectures we'd need to orchestrate

• building the app for that os/version/architecture
• publishing a container image for that os/version/architecture

then, once all of those were done, we'd need to

• create a new 'manifest list',
• add each of the created images to the list,
• ensure each one was 'annotated' with the correct os/platform/etc annotations, and
• push the manifest list to the registry

There are a couple hurdles we'd need to cover:

• if we would like to perform this by default for projects that specify multiple RIDs, then we may need a concept of a cross-RID/cross-TFM publish. This doesn't exist right now, and in fact is explicitly stopped by the cross-targeting targets in the .NET SDK.
• if we modeled this as a different, standlone target, then we wouldn't need to tie it to 'publish' necessarily, but we might lose the benefit of associating with 'publish' as a concept
• I'm unsure how much users would use this - is it worth pioneering a potentially-new publishing concept?
  • This would bring us parity with Jib and Ko - maybe parity is enough of a motivator

Other requirements:

• Need to ensure that tarball export works for multi-arch images as well.
• This work might also be useful/necessary for users on the 8 LTS, so we should strongly consider keeping the NuGet package and making that package work seamlessly with 8 and 9 SDKs. We should do this work off of the .NET SDK 8.0.4xx branch and merge forward to 9.0 as a result.

Proposal

The gesture we want users to perform for multi-arch manifest publishing is

dotnet publish -t:PublishContainer

i.e. the same gesture they use today. To do this, we should change the implementation of the current PublishContainer Target from its current behavior of 'publish a single image for a single RID' to more of a decision-making target.

PublishContainer should

• check if the project is currently in multi-TFM state. if so, error out. we require specifying a single specific TFM for now.
• check if the project is in a 'multi RID' state - meaning the project does not have a RuntimeIdentifier specified and does have either ContainerRuntimeIdentifiers or RuntimeIdentifiers specified. If so, invoke a new "_BuildMultiImageManifest" target
• otherwise, the project is in a single-RID, single-TFM state. In this state, invoke a new "_BuildSingleContainer" Target whose behavior is exactly the same as the single-image version of PublishContainer today.

An example of this per-scenario break-out is here.

Anticipated hurdles

Defaulted RIDs

The SDK does not have a concept of 'multi-RID' publish, and so today there are several places where it has assumed that the publish gesture implicates the desire for a single RID. The main way this negatively impacts us is the UseCurrentRuntimeIdentifier property, which is inferred as true here and ends up erroneously pinning us to a single RID. Setting it explicitly to false in the project files works around this.

PublishSingleFile

If PublishSingleFile is set and UseCurrentRuntimeIdentifier is not (as mentioned above), there is a mismatch in expectations. For now, for scenarios like our initial set, users may have to condition properties to only light up when the RID-specific build(s) are being done (for example, adding a Condition="'$(RuntimeIdentifier)' != ''" to several properties.

This is a symptom of the overall Publishing mechanisms of the .NET SDK not being designed for multi-RID publish today. In general, I think many SDK checks could be deferred to the 'inner RID' builds with no loss of intent, but we may have to push for this functionality in phases.

BuildMultiImageManifest

This target broadly should do two things

• orchestrate the building of N RID-specific images, collecting output information about them
  • N in this case is either the ContainerRuntimeIdentfiers Property (invented for this feature), or the RuntimeIdentifiers property (existing)
• combine those outputs into a single Manifest List structure
• orchestrate the export of the RID-specific images and the Manifest List to a registry, local daemon, or tarball in the correct order

Ideally, it would also unify any shared work that may happen during the multiple single-RID publishes into one unit of work that is shared. A specific example of this is

• determination of Base Image to and fetching of the various manifests for that Base Image
• tracking and parallelization of image layers that for the base Image(s)

Characteristics of the Manifest List

• It should use the OCI Image Index schema, not the Docker Manifest List structure, if at all possible
• It should contain N manifests, one for each RID created. These should map to the existing 'PlatformSpecificManifest` structure we know today
• It should contain annotations matching all of the conventionally-applied labels that we support for individual images

Visual Aids

flowchart TD
    A[Start Build] --> B[dotnet publish -t:PublishContainer]
    B --> D[Publish for linux-x64]
    D --> E[Package a linux-x64 container]
    B --> F[Publish for linux-arm64]
    F --> G[Package a linux-arm64 container]
    G --> I[Package both containers into an image index]
    E --> I
    I --> H[Push containers and image index to registry]

Loading

Work Stages/Milestones

We should have two phases of the work - initial MVP and then productizing.

Initial MVP

In this stage we implement the multi-RID aware publishing feature with external registries as the primary destination - so no pushing to local daemons or exporting to tarballs. This is the most well-known area of development. Once this is implemented, we can hand a preview nupkg over to the internal partner teams that want to test the feature so they can begin validation.

Productizing

In this stage we would implement tarball export and local-Daemon export of manifest lists, as well as full testing and error handling scenarios.

[baronfel]

I discussed this at MVP Summit this year, and feedback from folks was strong - we should do this so that we have parity with other ecosystems.

[baronfel]

There are two separate requirements here:

• create images for each architecture/platform that a user specifies that has a match with the base image chosen
• bundle those platform- and architecture-specific images into a single manifest list and publish it

The former is relatively straightforward today. We essentially want to 'multi-target' like you would with a TFM, but with RIDs:

A Directory.Build.targets file with a Containerize target that enables multi-rid container generation

<Project>
  <PropertyGroup>
    <!-- We have to build Publish AND PublishContainer because PublishContainer (and other
        PublishProfile-delivered targets) don't have an explicit Publish dependency. -->
    <_RequiredContainerPublishTargets>Publish;PublishContainer</_RequiredContainerPublishTargets>
  </PropertyGroup>

  <!-- Entrypoint, either from solution-level `/t:Containerize` or project-level `/t:Containerize` -->
  <Target Name="Containerize" Condition="'$(EnableSdkContainerSupport)' == 'true'">
    <!-- Strategy here is that we will figure out what proejct(s) to build the containerization targets(s) for
         based on project state. We use `AdditionalProperties` to customize the outputs of each of the builds. -->

    <!-- Properties set here:
      * TargetFramework - multitargeting - changes inference for base image based on TFM
      * VersionSuffix - without either explicitly setting `ContainerImageTag` or influencing the tag in some way
                        (I chose `VersionSuffix` because it lets folks still customize the 'base' of the version
                        and follows a Docker-ish convention of arch-specific info adding to the end of the tag)
                        we'll get the same 'tag' for each image, which would cause the images to override when
                        pushed to a registry or local daemon. NOTE: this is currently the RID, but it _should_
                        be a golang-style platform string (e.g linux-amd64 instead of linux-x64).
      * ContainerRuntimeIdentifier - if we're building for a specific RID, we need to set this so that the
                                     containerization targets know what RID to build for
      * RuntimeIdentifier - if we're building for a specific RID, we need to set this so that we get optimized 
                            RID-specific assets in the publish output

      NOTE: we could get away with setting `RuntimeIdentfier` here to control `ContainerRuntimeIdentifier` inference
            but this is also nice and explicit.
     -->

    <!-- TFMs but no TF -> multitarget, making image for each TFM -->
    <ItemGroup Condition="'$(TargetFrameworks)' != ''
                          and '$(TargetFramework)' == ''" >
      <_TFMItems Include="$(TargetFrameworks)" />
      <_SingleContainerPublish Include="$(MSBuildProjectFullPath)"
        AdditionalProperties="TargetFramework=%(_TFMItems.Identity);
                              VersionSuffix=$([MSBuild]::GetTargetFrameworkVersion('%(_TFMItems.Identity)', 2))" />
    </ItemGroup>
    
    <!-- TF but no TFMs -> single image (aka the default pathway) up until now -->
    <ItemGroup Condition="'$(TargetFramework)' != ''
                          and '$(RuntimeIdentifiers)' == ''">
        <_SingleContainerPublish Include="$(MSBuildProjectFullPath)" />
    </ItemGroup>

    <!-- TF with RIDs -> multi-arch, single image per arch -->
    <ItemGroup Condition="'$(TargetFramework)' != ''
                          and '$(RuntimeIdentifiers)' != ''">
      <_RIDItems Include="$(RuntimeIdentifiers)" />
      <_SingleContainerPublish Include="$(MSBuildProjectFullPath)"
        AdditionalProperties="ContainerRuntimeIdentifier=%(_RIDItems.Identity);
                              RuntimeIdentifier=%(_RIDItems.Identity);
                              VersionSuffix=%(_RIDItems.Identity);" />
    </ItemGroup>
    
    <MSBuild Projects="@(_SingleContainerPublish)" Targets="$(_RequiredContainerPublishTargets)" BuildInParallel="true" />
  </Target>
</Project>

Adding this target to a project lets you run dotnet build /t:Containerize and generate architecture- and platform-specific images. We should look at including something like this in the official build targets for the 7.0.400 time frame if at all possible. Adding such a target also enables a related use case: containerizing every project in a solution that can be containerized. This enables workflows like dotnet build /t:Containerize myapp.sln && docker-compose up, where there is a compose.yaml that specifies the relationships between the services in the usual way, just using image: instead of build: stanzas for the project.

A worked example of this can be seen with this diff of the eshoponcontainers project. The docker compose YAML specifically is a useful example.

[mu88]

I would love seeing this feature, as it's literally the last missing piece from throwing away my Dockerfiles and replacing them with the SDK Container Building Tools.

[baronfel]

Making multi-architecture images is pretty straightforward, as shown above. The next step is creating image manifests using those images. There's an example of this in my sdk-container-demo repository here that builds upon the snippet above by:

• building architecture-specific images with a certain labeling scheme
• pushing those images to the destination registry
• creating a container manifest using the docker CLI with each of those images
• pushing the generated manifest to the destination registry

Our tooling doesn't yet speak these manifests, but it could learn to.

[mu88]

That looks very promising @baronfel !

This of course sparks hope 😉 what's missing from adding it to the Container Building Tools?