This repo ce-test-recipes
is a test repo for uploading to test pypi using
concourse so we can prepare for the move from Jenkins to concourse for
deployment when we are ready.
https://packaging.python.org/guides/using-testpypi/
We all need to create accounts on test pypi so we can upload the package.
We need to remember to specify the --repository-url
option or set the
TWINE_REPOSITORY_URL
environment variable when uploading so we don't affect
pypi.
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Have a look at the Jenkinsfile for what Jenkins does for us when we tag the package. Specifically the one step (formatted to make it easier to read):
sh "docker run --rm -e TWINE_USERNAME -e TWINE_PASSWORD -v ${WORKSPACE}:/src:rw --workdir /src python:3 /bin/bash -c \
\"apt-get update && \
apt-get install -y git && \
pip install -q twine && \
python setup.py bdist_wheel --universal && \
twine upload dist/* && \
rm -rf dist build *.egg-info versioneer.pyc\""
It uses the python 3 docker image, installs
git
and twine
, creates a python binary distribution wheel file
(--universal
means the package works for both python 2 and 3) and use
twine upload
to upload to pypi.
Now that you understand what Jenkins is doing, it is good to know how to do this manually in case concourse or Jenkins didn't work.
-
Do a fresh check out to have a clean environment:
cd /tmp/ git clone https://github.com/karenc/ce-test-recipes.git cd ce-test-recipes
-
Tag the repo and push to origin (just pick one that doesn't already exist):
git tag -a v0.0.1 git log --graph --oneline --decorate # check origin/master and tag: v0.0.1 are on the same line git push origin v0.0.1
The general rule I follow for the commit message is to have
Release v0.0.1
and then the content ofgit log --format='- %s' --reverse v0.0.0..
in the body of the commit.Look at the release on github: https://github.com/karenc/ce-test-recipes/releases/tag/v0.0.1
-
Create a python 3 virtualenv and activate it:
python3 -m venv py3 . ./py3/bin/activate
which python
should show/tmp/ce-test-recipes/py3/bin/python
-
Install
twine
andwheel
:pip install twine wheel
-
Build the binary distribution:
python setup.py bdist_wheel --universal
ls dist
should show something likece_test_recipes-0.0.1-py2.py3-none-any.whl
-
Upload the new version to test pypi:
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
You will need to put in your username and password for test pypi.
-
Now check on test pypi for the version you just uploaded:
Our target is to use concourse to upload to pypi instead of using Jenkins. What concourse needs to do is similar to Jenkins but everything is configured using the pipeline.yml. There is nothing on the server except to read the logs and trigger jobs manually.
Have a look at concourse/dist-upload/pipeline.yml:
First we need to declare all the resources for the pipeline. The only resource is the git repo:
resources:
- name: ce-test-recipes
type: git
source:
uri: https://github.com/karenc/ce-test-recipes.git
tag_filter: '*'
The tag_filter: '*'
line is for getting only tags.
The first step of the job is to get the repository and specify how the job is triggered:
jobs:
- name: dist upload
public: true
serial: true
plan:
- get: ce-test-recipes
trigger: true
Concourse pipelines can be triggered by a resource. Details can be found here in the concourse tutorial.
This job should be automatically triggered every time there is a new tag pushed to github.
The second (and last) step of the job is to create the binary distribution and upload it to pypi:
- task: twine upload
config:
platform: linux
image_resource:
type: docker-image
source:
repository: python
tag: 3
inputs:
- name: ce-test-recipes
run:
path: /bin/bash
args:
- -c
- |
cd ce-test-recipes && \
pip install -q twine && \
python setup.py bdist_wheel --universal && \
twine upload dist/*
params:
TWINE_USERNAME: ((twine-username))
TWINE_PASSWORD: ((twine-password))
# upload to test pypi
TWINE_REPOSITORY_URL: https://test.pypi.org/legacy/
This step uses a docker image python:3
from
https://hub.docker.com. It is going to use the
output from the previous step ce-test-recipes
as one of the inputs. The
run:
part just does something similar to Jenkins or what we would do
manually. params:
contains the environment variables we want to set for the
docker container. ((twine-username))
and ((twine-password))
are variables
that you can pass in. There are 3 ways to pass the variables in:
-
Pass the variables in as options to
fly
(not tested):-v twine-username=karen_chan -v twine-password=xxxxx
-
Create a yaml file with all the variables, e.g.
vars.yml
:twine-username: karen-chan twine-password: xxxxx
and add this option to
fly set-pipeline
:-l vars.yml
-
Fetch the variables dynamically from Credentials Manager (not tested). There is nothing special to do here except the server needs to be configured. See Secret Parameters with Credentials Manager in the concourse tutorial.
To add the pipeline to concourse:
fly -t <concourse-instance> set-pipeline -p ce-test-recipes-pypi -c concourse/dist-upload/pipeline.yml -l concourse/dist-upload/vars.yml
Unpause the pipeline.
Push a tag to ce-test-recipes and wait for the job to start.