In the OpenStack team, and potentially elsewhere, there is a need to maintain stable as well as devel charms. With non-reactive charms, this is fairly simple.
However, with reactive charms, when a charm is rebuilt the dependencies are taken from the master branch of the various layers and interfaces, and the charm-helpers library, which is baked into layer-basic, is the latest available on PyPi.
The issue is serious: the OpenStack is unable to support stable/17.02 charms without completely manually rebuilding them based on downloading all of the layers and manually editing wheelhouse.txt files to ensure that the versions that were available at the time the stable was cut are the correct versions, along with any cherry picked patches.
The builds are starting to fail less than six months after the original freeze to stable, which means that patches cannot be cherry-picked back to stable charms.
- Abandon supporting stable charms.
- Build version support into the dependency management system.
The OpenStack team doesn't feel it is possible to abandon maintaining stable charms as this will impact on our support offer. Therefore, it seems necessary to explore what options are available to support versioning in the build tools, and how this might be implemented.
The dependencies in a built charm are:
- Python modules; defined in the layer's wheelhouse.txt
- layers and interfaces; defined in the layer.yaml
To take a typical example of an OpenStack API charm, the aodh charm (pronounced 'erhh'), has the following dependencies:
layer.yaml:
includes: ['layer:openstack-api', 'interface:mongodb']
options:
basic:
use_venv: True
include_system_packages: True
repo: https://github.com/openstack/charm-aodh
config:
deletes:
- verbose
no wheelhouse.txt
layer.yaml
includes: ['layer:openstack-principle', 'interface:mysql-shared',
'interface:rabbitmq', 'interface:keystone',
'interface:hacluster', 'interface:openstack-ha']
repo: 'https://github.com/openstack/charm-layer-openstack-api'
no wheelhouse.txt
layer.yaml
includes: ['layer:openstack']
repo: 'https://github.com/openstack/charm-layer-openstack-principle'
no wheelhouse.txt
layer.yaml
includes: ['layer:basic']
repo: 'https://github.com/openstack/charm-layer-openstack'
wheelhouse.txt
netifaces>=0.10.4
# charms_openstack - pull from github until first release
git+https://github.com/openstack/charms.openstack.git#egg=charms.openstack
Note: yes, the wheelhouse.txt
does refer to the master branch of the github
repo.
layer.yaml
defines:
packages:
type: array
default: []
description: Additional packages to be installed at time of bootstrap
use_venv:
type: boolean
default: false
description: >
Install charm dependencies (wheelhouse) into a Python virtual environment
to help avoid conflicts with other charms or libraries on the machine.
include_system_packages:
type: boolean
default: false
description: >
If using a virtual environment, allow the venv to see Python packages
installed at the system level. This reduces isolation, but is necessary
to use Python packages installed via apt-get.
wheelhouse.txt
pip>=7.0.0,<8.2.0
charmhelpers>=0.4.0,<1.0.0
charms.reactive>=0.1.0,<2.0.0
Note that the interface files don't declare any dependencies, although they could, I believe, have a wheelhouse.txt.
The charms for OpenStack require:
- The ability to tag all of the dependencies at the 'versions' they were when the charm was 'cut' as a stable charm. e.g. stable/17.02
- The ability to backport or cherry pick bug fixes back into both the charm layer (easy) and the dependencies (hard) so that a new 'stable/17.02' charm can be released with those bug fixes.
And this needs to work for layers/interfaces the OpenStack charms team doesn't control.
There are a number of ways that this could be solved. Some of the ideas that have been raised at one point or another is:
- Write a 'freeze' tool, that tracks down ALL of the dependencies, builds a
custom
layers
andinterfaces
directories, fix up the wheelhouse.txt to capture the current requirements. This would 'freeze' the charm by making it a really 'fat' charm layer with all the dependencies baked in. However, it would not be linked to the original repositories and thus would make it harder to cherry pick fixes into the charm. - Add a feature to the
layer.yaml
file that will allow a version/tag to be added to specify what should be downloaded to the charm during build. - Allow a
dependencies.txt
file to be submitted along with thecharm build
that would allow allow any layer's dependencies to be overridden. i.e. to override a particular charm-helpers version, or layer tag.
The ideal solution from the perspective of the OpenStack charms team is:
- The dependencies can be discovered/set for Python modules and layers separately.
- The dependencies are stored in a file in the charm layer and override any layers below the layer with the dependencies in it. This is to enable 'freezing' at a layer as well as at a top charm.
- The dependency can be specified as a tag/branch for layers (or Python modules via git/bzr) so that bug fixes to stable dependencies get built into the charm.