[UPDATE] by Dzintars Klavins @ 2023-05-15
These instructions is no more quite accurate. Current setup is much more simpler. Probably I will update these instructions at some point.
[WIP] by Dzintars Klavins @ 2021-08-28
These instructions is written for myself as an reminder guide. It is higly possible that they are wrong, outdated, un-maintained. I do not carry any responsibility about following these instructions. Use at your own risk! If you have any suggestions, i will be happy to improve this document.
This setup utilizes podman play kube
tool on Fedora 34 WorkStation.
Intention of this setup is to use it for Kubernetes (Minikube) and Bazel (build system) for temporary image storage.
The alternatives is to use Minikubes built-in simple registry or some of the public offerings.
I don't like public services as I like to save some money and tinker with the new stuff.
- Make it rootless
- Handle volume persistance corectly
- Automate?
- Podman
- Local storage (ideally would be neat to use Minio S3 from my oswee.ansible.minio role, but I left this for later)
- Clean up the mounted volume directories if starting over. Don't forget to
sudo ranger
because files in those directories belongs to sudo user.
Get the active firewall zone
sudo firewall-cmd --get-active-zones
See which zone is used for your network interface. Most likely it will be public
zone.
Open the ports required for this setup
sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --permanent --zone=public --add-port=9981/tcp
sudo firewall-cmd --reload
Add the image registry domain to the /etc/hosts
file
echo '192.168.0.2 registry.domain.local' | sudo tee -a /etc/hosts
Or if you willing to add to the specific line then
sudo sed -i "2i192.168.0.2 registry.domain.local" /etc/hosts
where 2i
prefix is the line nuber in which you want that record to appear.
Create the ~/quay/config-pod.yaml
file with the content:
---
apiVersion: v1
kind: Pod
metadata:
name: quay
labels:
app: quay
spec:
restartPolicy: Always
containers:
- name: postgres
image: docker.io/postgres:latest
env:
- name: POSTGRES_USER
value: quayuser
- name: POSTGRES_PASSWORD
value: quaypass
- name: POSTGRES_DB
value: quay
securityContext:
allowPrivilegeEscalation: false
# privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- mountPath: /var/lib/postgresql/data:Z
name: postgres-data
- name: redis
image: registry.redhat.io/rhel8/redis-5:1
env:
- name: REDIS_PASSWORD
value: strongpassword
securityContext:
allowPrivilegeEscalation: false
# privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- mountPath: /var/lib/redis/data:Z
name: redis-data
- name: config
image: quay.io/projectquay/quay:latest
ports:
- containerPort: 8080
hostPort: 9981
protocol: TCP
args:
- 'config'
- 'secret'
volumes:
- name: postgres-data
hostPath:
path: /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
type: Directory
- name: redis-data
hostPath:
path: /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
type: Directory
Make sure the volume directories exists
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/config
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/storage
TODO: The volume location could be more conventional/generic
Run the sudo podaman play kube ~/quay/config-pod.yaml
Enable TRGM module
sudo podman exec -it quay-postgres /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm" | psql -d quay -U quayuser'
At this point you should have healthy Quay-Config, Redis and Postgresql containers running inside single quay
pod.
Now it's time to generate configuration file for the Quay registry.
Access the configuration page at http://localhost:9981
Because I am using HAProxy with TLS termination (my proxy encrypts all traffic with walid TLS certificates) in this setup i will skip any TLS part. Thou... in general it is bad idea to run unencripted traffic inside of the perimeter and probably i will fix that some day soon.
For the Redis and Postgresql hosts use 127.0.0.1
IP address because that is how containers inside the Pod
can comunicate.
For "Database Server:" field shold be 127.0.0.1:5432
.
You will instantly see the "Please specify a non-localhost hostname" warning. We will deal with that later.
For the "Redis Hostname:" field should be 127.0.0.1
Also add your username you will be using for Quay registry to the "Super Users:" field at the bottom of the page.
Now, the tricky part:
In order for get the "Validate Configuration Changes" button available, we need temporary to replace 127.0.0.1
IP address
for Redis and Postgresql with any valid domain, like example.com
Then we need to open Chrome or Firefox Developer Tools and to select the "Validate Configuration Changes" button.
In the source code we need to find the lines which looks like
<button class="btn btn-warning ng-scope ng-hide" ng-click="checkValidateAndSave()" ng-show="!configform.$valid">
<i class="fa fa-lg fa-sort"></i>
<!-- ngIf: configform.$error['required'].length -->
<!-- ngIf: !configform.$error['required'].length --><span ng-if="!configform.$error['required'].length" class="ng-scope">
Invalid configuration field
</span><!-- end ngIf: !configform.$error['required'].length -->
</button>
and we need to change ng-show="!configform.$valid"
to ng-show="configform.$valid"
This will prevent from hiding the "Validate Configuration Changes" button when we use 127.0.0.1
IP address as databases host addresses.
Once validation is successful you need to download the config tar.
Export environment variable
export QUAY=/home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data
Move the tar to the Quay config directory
sudo tar -xvf ~/Downloads/quay-config.tar.gz -C $QUAY/config
Remove the Quay Config container from the pod
sudo podman stop quay-config
This should stop and remove the config container form the pod but leave the pod networking intact.
So, now inject the real registry container in the same pod
sudo podman run --pod quay -d \
--name=registry \
-v $QUAY/config:/quay-registry/conf/stack:Z \
-v $QUAY/storage:/datastorage:Z \
quay.io/projectquay/quay:latest
As you see, I do not expose any ports there, because those ports already was exposed by Config container. We are lucky that Config and Registry containers uses 8080 port inside. :)
Check the logs
sudo podman logs quay-redis
If you see something like
1:M 29 Aug 2021 00:12:03.113 # Background saving error
1:M 29 Aug 2021 00:12:09.028 * 1 changes in 900 seconds. Saving...
1:M 29 Aug 2021 00:12:09.029 * Background saving started by pid 514
514:C 29 Aug 2021 00:12:09.029 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis/data) for saving: Permission denied
this means we have some problem with volume mount permissions.
Most likely we need to use podman unshare
and sudo setfacl -m u:26:-wx /some/volume/dir
to set the right permissions. (UNVALIDATED/TODO)
My main intention was to expose as less ports as possible and make the setup more "portable" if i could say so.
Managing single Pod is much more easier thand 3 or 4 separate conatiners. And I do really like play kube
:)
To restore the existing setup we need another Pod manifest because we will use real Quay Registry image there. All other paramaters we leave intact.
Create new file
touch ~/quay/registry-pod.yaml
with the content of
---
apiVersion: v1
kind: Pod
metadata:
name: quay
labels:
app: quay
spec:
restartPolicy: Always
containers:
- name: postgres
image: docker.io/postgres:latest
env:
- name: POSTGRES_USER
value: quayuser
- name: POSTGRES_PASSWORD
value: quaypass
- name: POSTGRES_DB
value: quay
securityContext:
allowPrivilegeEscalation: false
# privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- mountPath: /var/lib/postgresql/data:Z
name: postgres-data
- name: redis
image: registry.redhat.io/rhel8/redis-5:1
env:
- name: REDIS_PASSWORD
value: strongpassword
securityContext:
allowPrivilegeEscalation: false
# privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- mountPath: /var/lib/redis/data:Z
name: redis-data
- name: registry
image: quay.io/projectquay/quay:latest
ports:
- containerPort: 8080
hostPort: 9981
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
# privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- mountPath: /quay-registry/conf/stack:Z
name: quay-config
- mountPath: /datastorage:Z
name: quay-storage
volumes:
- name: postgres-data
hostPath:
path: /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
type: Directory
- name: redis-data
hostPath:
path: /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
type: Directory
- name: quay-config
hostPath:
path: /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/config
type: Directory
- name: quay-storage
hostPath:
path: /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/storage
type: Directory
Now, because we used volume mounts, all our data was persisted in those directories and new Pod will use the same configs and data. In theory... I haven't checked it yet. Writing this while i am doing all this. :)