Since we introduced the pull provider, we carried both yggdrasil and our own worker (foreman_ygg_worker) in our client repos. Starting with EL 9.5, yggdrasil is going to be included in EL repos. The version there (0.4.1) is much newer than the one we ship, and therefore "wins" when yggdrasil is being installed. This causes a problem, because our own worker is not compatible with this newer version of yggdrasil and because of architectural changes in yggdrasil itself.
This issue may pop up on other currently supported distributions, but for the purpose of this document we’ll limit ourselves on distributions from the EL family.
In the 0.2.z line, yggdrasil talked with workers over grpc.
In the 0.4.z line, yggdrasil talks with workers over dbus.
In the 0.2.z line, upstream yggdrasil used yggd.service as the name of the service, along with yggd as the command line utility that communicates with the daemon. The config file for yggdrasil was /etc/yggdrasil/config.toml.
Accidentally, we settled on yggdrasild.service (and yggdrasild as a tool), by setting variables in our rpm spec differently.
The 0.4.z line, introduced support for templated services yggdrasil@.service (which depends on yggdasil-bus@.service) and dropped the system-wide yggd.service in favor of yggdrasil.service. The system-wide instance uses /etc/yggdrasil/config.toml as its configuration file, while the templated ones use /etc/yggdrasil/yggdrasil-%i.toml, where %i is the name of the instance.
There was also a branded variant (rhc), which used rhcd.service and rhc as a command line utility to communicate with the daemon. This is now being phased out, in favor of rhc (as the cli tool) communicating with the daemon running as yggddrasil.service.
In the 0.2.z line, the stanza used to configure which brokers yggdrasil connects to was called broker.
In the 0.4.z line, the stanza is server.
Additionally, due to different compile-time options being set in our packaging and in RHEL packaging, the version coming from RHEL behaves slightly differently when started with the same configuration so we need to pass in additional configuration options to override the compile-time defaults and make it work as expected. data-host = "" needs to be added to prevent yggdrasil from overwriting the hostname part of the URL from which it should download the script to execute and to which it should report updates. prefix = "yggdrasil" needs to be added to make yggdrasil connect to the topic to which we publish messages.
In the 0.2.z line, yggdrasil started the workers.
In the 0.4.z line, the workers have to be started by other means, be it by a service manager or having them be bus-activated.
In the 0.2.z line, yggdrasil used make as the build system.
In the 0.4.z line, yggdrasil uses meson as the build system.
Foreman comes with a couple of tools (provisioning snippet, content source change and katello-pull-transport-migrate) that ease migration from previous implementation of a pull client (katello-agent) and that allow easy deployment of the client on newly provisioned or registered hosts. These tools rely on known names of yggdrasild.service and yggdrasild.
Foreman’s documentation carries procedures, which describe how certain behavior of the pull provider client can be changed on the client machines. This relies on yggdrasild being the name of the service and that this service spawns the workers as its children.
Foreman doesn’t have foreman-version-specific repositories. (Technically Foreman does, but some downstream derivatives of Foreman don’t.). This can be tricky, if the changes delivered in client repository need corresponding changes in Foreman.
deployment of yggdasil-0.2.z
deployment of yggdasil-0.4.z
in place update from yggdrasil-0.2.z to yggdrasil-0.4.z
EL7, EL8 and EL9 repositories would stilll carry yggdrasil-0.2.z as they do now, but new-enough EL9 the version from EL repos would win over the one from our repos. Our EL10 repos wouldn’t carry the yggdrasil package at all.
This proposal mostly revolves around having a compatibility rpm (something like yggdrasil-foreman-compat). This rpm would supplement yggdrasil >=0.4.0 so that it would get automatically installed and would provide various conveniences for migrating between versions.
@subpop sent us a PR (theforeman/foreman_ygg_worker#21) that makes foreman_ygg_worker compatible with both 0.2.z and 0.4.z of yggdrasil.
Because rhc uses the yggdrasil.service and the systemd-wide bus, we cannot switch to it, otherwise we’d risk conflicting with rhc‘s configuration. That means we’d have to switch over to the templated services. Here I would propose using foreman as the instance name, leading to these services being used:
yggdrasil-bus@foreman.serviceyggdrasil@foreman.servicecom.redhat.Yggdrasil1.Worker1.foreman@foreman.service
To resolve the config file name change on upgrade, the compatibility rpm would run a post script, which would copy /etc/yggdrasil/config.toml to /etc/yggdrasil/yggdrasil-foreman.toml, but only if yggdrasild.service is enabled.
We need to update the Foreman-side tooling to generate the config file with the additional options. For the server x broker part, we will need to put both into the config file to maintain compatibility with both versions.
The compatibility rpm would have to run a scriptlet in post to migrate the existing config file to the new values.
To resolve the issue of workers having to be spawned separately from yggdrasil (i.e. migrating from yggdrasild.service to the ones listed in Naming and branding), the compatibility rpm would define a yggdrasild.service, which would depend (probably BindsTo and After) on yggdrasil@foreman.service and com.redhat.Yggdrasil1.Worker1.foreman@foreman.service.
This does not affect us in any way. We keep yggdrasil-0.2.z where we already have it, newer versions will not be packaged by us.
Because we cannot really change the Foreman-side tooling in-sync with the client repo changes, the compatibility package would have to set things up so that the ways we are used to keep working.
The Naming and branding section above already covers the changes in service name.
To provide the possibility to still invoke yggdrasild status as we do in our templates, the compatibility package would ship yggdrasild script, which would invoke DBUS_SESSION_BUS_ADDRESS=unix:abstract=yggd_foreman yggd "$@"
In addition to what is already described in the Yggdrasil configuration file contents section above, we would also have to change our templates and katello-pull-transport-migrate to emit yggdrasil configuration into a location based on version of yggdrasil. The rpm post script way wouldn’t apply here as the rpm is expected to be installed before the configuration gets generated.
This is one of the two places where we can’t really do anything about. The documentation would have to have two different paths, depending on whether it is being applied on a system running yggdrasil-0.2.z or yggdrasil-0.4.z.
For 0.2.z the current procedures would keep working, for 0.4.z the work directory override would have to be applied to com.redhat.Yggdrasil1.Worker1.foreman@foreman.service instead of yggdrasild.service.
This is the other place where we can’t really do anything about. We just have to do our best to deliver the bits more or less at the same time everywhere.
Some of the bugs we’ve reported have been resolved in the current versions of yggdrasil. If we went down this path, these would get resolved only on some of the client machines.
remains unchanged, should work ootb
should work ootb, assuming the template changes land in sync with yggdrasil and worker changes
should be covered by the compatibility rpm described above
This revolves around updating yggdrasil in our client repo to 0.4.z to match the version that is coming in EL 9.5 on all currently supported EL*. This would allow us to reduce the problem space. EL7, EL8 and EL9 repos would carry our package till their respective EOLs. For EL10 we wouldn’t even start shipping yggdrasil package in our repos.
It shares many of the points with Proposal A, but without the overhead of having to decide things based on versions.
@subpop’s (theforeman/foreman_ygg_worker#21) still applies here, but could be reduced to switching to dbus, instead of trying to support both grpc and dbus.
See Naming and branding in Proposal A.
See Yggdrasil configuration file contents in Proposal A.
See Naming and branding in Proposal A.
This would require us to do some changes in packaging to bump the version. As of 0.4.1, yggdrasil requires meson >=0.58.0, which is available on all currently supported EL versions, except for EL7, which has meson-0.55.0. Chances are it would build it we reduced the version, but we won’t know until we try.
(unconfirmed) Another thing that might prove troublesome are the different versions of dbus available on currently supported EL versions.
See Proposal A, but with the version check removed.
Same as Proposal A, but the version check could be removed, only the new location would remain.
Same as Proposal A.
Some of the bugs we’ve reported have been resolved in the current versions of yggdrasil. If we went down this path, these would get resolved on all clients, no matter the os version.
By introducing newer yggdrasil version to all EL*, we risk that we might break things on EL*, whereas in Proposal A the potential breakage would be limited to EL9.5 and above.
On the flip side, this would allow us to resolve some long open bugs everywhere, instead of just resolving them on EL9.5 and above.
Same as fresh install on any EL9.y.
remains unchanged, should work ootb
should be covered by the compatibility rpm described above
This would also apply to updates on all EL versions.
This proposal is an extension of variants A&B. This extension sticks to the original proposal with one distinction - this extension wouldn't use the templated systemd units, it would use the system-wide yggdrasil.service instead.
Here we would choose to disallow users from running rhc and remote execution pull provider on the same machine. While both could use the yggdrasil.service and the systemd-wide bus, yggdrasil can be configured to connect to only one MQTT broker at a time. If we leveraged the system-wide services, we would end up using these:
dbus.serviceyggdrasil.servicecom.redhat.Yggdrasil1.Worker1.foreman.service
To resolve the issue of workers having to be spawned separately from yggdrasil (i.e. migrating from yggdrasild.service to the ones listed in Naming and branding), the compatibility rpm would define a yggdrasild.service, which would depend (probably BindsTo and After) on yggdrasil.service and com.redhat.Yggdrasil1.Worker1.foreman.service.
The Naming and branding and deployment changes sections above already cover the changes in service name.
To provide the possibility to still invoke yggdrasild status as we do in our templates, the compatibility package would ship yggdrasild as a symlink to yggd.
Because the config file format changed, we would still have to make changes on the Foreman side to be able to generate config file that would work with any version of yggdrasil. See first and last paragraphs in Foreman-side dependencies in Proposal B.
This is one of the two places where we can’t really do anything about. The documentation would have to have two different paths, depending on whether it is being applied on a system running yggdrasil-0.2.z or yggdrasil-0.4.z.
For 0.2.z the current procedures would keep working, for 0.4.z the work directory override would have to be applied to com.redhat.Yggdrasil1.Worker1.foreman.service instead of yggdrasild.service.