We want to host a nice and secure container registry – to push docker images there and our kubernetes cluster would pull them from this registry.
So came the idea to use the gitea over ssl.
Part of Developer Tools: The Complete Guide to Modern Development Workflows
1) Gitea already has container registry
2) Apache as TLS terminating proxy will add HTTPS to our Gitea.
3) That how it’s started… Rot CA, self-signed certificates…
When we need the registry in k8s
Unfortunately registry in kubernetes cluster created by kubespray doesn’t work for me.
- to push into it we need to create temporary tunnel via kube-proxy
- after pushing into it mu current version of new cluster could not pull images from this internal registry
After spending some awesome two evenings trying to fix it,
I have decided just to use Gitea’s internal container registry, just need to configure HTTPS access to it.
Will make it public anyway, so docker and k8s doesn’t have to do any docker logins.
Maybe. Let’s see how we go.
Testing if it’s working
To test if container registry suits us we want to be able
- to push image there and
- to create deployment in k8s from that image
sudo docker pull alpine:3.12.0
sudo docker images
sudo docker tag a24bb4013296 localhost:5000/rg/alpine:version.3.12.0
sudo docker push localhost:5000/rg/alpine:version.3.12.0
Enter fullscreen mode Exit fullscreen mode
Now xed alp1.yaml or nano alp1.yaml depending on what mood you are in and
apiVersion: apps/v1
kind: Deployment
metadata:
name: alp-registry-test
spec:
replicas: 1
selector:
matchLabels:
app: alp-registry-test
template:
metadata:
labels:
app: alp-registry-test
spec:
containers:
- name: alpine-test
image: localhost:5000/rg/lpine:version.3.12.0
imagePullSecrets:
- name: registry-secret
Enter fullscreen mode Exit fullscreen mode
Here is this file available for download
Then creating this deployment
kubectl create -f alp1.yaml
kubectl get pods
kubectl describe po alp-registry-test-5f5cb94b97-njsp2
# or whatever pod was created or wasn't created
Enter fullscreen mode Exit fullscreen mode
Yes I know about this bit
imagePullSecrets:
- name: registry-secret
Enter fullscreen mode Exit fullscreen mode
registry-secret is the name of the secret kubespray created.
Cleanup
kubectl delete -f alp1.yaml
Enter fullscreen mode Exit fullscreen mode
How
A bunch of documentation is available on Gitea site: https-setup
And on that page: https://docs.gitea.com/administration/reverse-proxies
Step 1 – Install Apache and create simple test site
Install Apache
sudo apt install apache2
Enter fullscreen mode Exit fullscreen mode
Check what we have in the firewall
sudo ufw status
Enter fullscreen mode Exit fullscreen mode
If firewall is active think which port you want to expose via https and allow it.
Standart apache configs are
sudo ufw app list
Enter fullscreen mode Exit fullscreen mode
We might see something like
Available applications:
Apache
Apache Full
Apache Secure
OpenSSH
Enter fullscreen mode Exit fullscreen mode
and to enable only port 443 we execute
sudo ufw allow 'Apache Secure'
Enter fullscreen mode Exit fullscreen mode
ok. now check the service status
sudo systemctl status apache2
Enter fullscreen mode Exit fullscreen mode
Next – create simple virtual server to test the Apache
sudo mkdir /var/www/reg.homelab
sudo chown -R $USER:$USER /var/www/reg.homelab
sudo chmod -R 755 /var/www/reg.homelab
sudo nano /var/www/reg.homelab/index.html
Enter fullscreen mode Exit fullscreen mode
Put there
<html>
<head>
<title>Welcome to reg.homelab!</title>
</head>
<body>
<h1>Success! The reg.homelab virtual host is working!</h1>
</body>
</html>
Enter fullscreen mode Exit fullscreen mode
then
sudo nano /etc/apache2/sites-available/reg.homelab.conf
Enter fullscreen mode Exit fullscreen mode
and put there
<VirtualHost *:3080>
ServerAdmin webmaster@localhost
ServerName reg.homelab
ServerAlias www.reg.homelab
DocumentRoot /var/www/reg.homelab
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
Enter fullscreen mode Exit fullscreen mode
then disable default site, enable this one and check how we are doing
sudo a2ensite reg.homelab.conf
sudo a2dissite 000-default.conf
sudo apache2ctl configtest
Enter fullscreen mode Exit fullscreen mode
see something like this?
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Enter fullscreen mode Exit fullscreen mode
then
sudo nano /etc/apache2/apache2.conf
Enter fullscreen mode Exit fullscreen mode
add to the end:
ServerName reg.homelab
Enter fullscreen mode Exit fullscreen mode
And this is still not the end! now need to remove port 80 binding attempt
sudo nano /etc/apache2/ports.conf
Enter fullscreen mode Exit fullscreen mode
put there
Listen 3030
...
Listen 443
Enter fullscreen mode Exit fullscreen mode
And now
sudo systemctl restart apache2
sudo systemctl status apache2
journalctl -xeu apache2.service
curl localhost:3080
Enter fullscreen mode Exit fullscreen mode
OK! Now navigate to :3080
Step 2 – Converting this site to insecure reverse proxy 🙂 to Gitea
sudo nano /etc/apache2/sites-available/reg.homelab.conf
Enter fullscreen mode Exit fullscreen mode
putting there
<VirtualHost *:443>
ServerAdmin webmaster@localhost
ServerName reg.homelab
ServerAlias www.reg.homelab
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
ProxyPreserveHost On
ProxyRequests off
AllowEncodedSlashes NoDecode
ProxyPass / http://localhost:3000/ nocanon
</VirtualHost>
Enter fullscreen mode Exit fullscreen mode
Do some config test
sudo apache2ctl configtest
Enter fullscreen mode Exit fullscreen mode
Add some apache modules and restart Apache
sudo a2enmod proxy proxy_http ssl
sudo systemctl restart apache2
sudo systemctl status apache2
Enter fullscreen mode Exit fullscreen mode
OK, now navigate to or curl
# Yes, it is still a http, but on the 443 port
curl http://localhost:443
http://<Server_IP_Address>:443/
Enter fullscreen mode Exit fullscreen mode
Step 3 Self – signed root CA and site cert
SweetHome-RootCA.
CANAME=MostImportant-RootCA
# optional, create a directory
mkdir $CANAME
cd $CANAME
# generate aes encrypted private key
openssl genrsa -aes256 -out $CANAME.key 4096
# create certificate, 1826 days = 5 years
openssl req -x509 -new -nodes -key $CANAME.key -sha256 -days 1826 -out $CANAME.crt -subj '/CN=My Root CA/C=AT/ST=Vienna/L=Vienna/O=MyOrganisation'
# create certificate for service
MYCERT=reg.homelab
openssl req -new -nodes -out $MYCERT.csr -newkey rsa:4096 -keyout $MYCERT.key -subj '/CN=My Firewall/C=AT/ST=Vienna/L=Vienna/O=MyOrganisation'
# create a v3 ext file for SAN properties
cat > $MYCERT.v3.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = reg.homelablab
DNS.2 = gitea.homelablab
IP.1 = 192.168.0.10
IP.2 = 192.168.0.11
EOF
openssl x509 -req -in $MYCERT.csr -CA $CANAME.crt -CAkey $CANAME.key -CAcreateserial -out $MYCERT.crt -days 730 -sha256 -extfile $MYCERT.v3.ext
Enter fullscreen mode Exit fullscreen mode
On machines connecting to gitea / registryRegister root certificate on linux:
sudo cp MostImportant-RootCA.crt /usr/local/share/ca-certificates
sudo update-ca-certificates
Enter fullscreen mode Exit fullscreen mode
Registre root certificate on windows:
- Doubleclick file MostImportant-RootCA.crt
- Import to local user
- Select Truster Root CA.
- When prompted about importing untrusted certificate – click yes
When git pull on windows says about
Unable to resolve "unable to get local issuer certificate...
Enter fullscreen mode Exit fullscreen mode
you can tell git to use windows networking layer by
git config --global http.sslbackend schannel
Enter fullscreen mode Exit fullscreen mode
Step 4 – Securing proxy with self-signed certificate
https://httpd.apache.org/docs/2.4/ssl/ssl_howto.html
Create a self-signed certificate IF you didn’t in step 3
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-subj "/CN=reg.homelab" \
-addext "subjectAltName = DNS:reg.homelab" \
-keyout /etc/ssl/private/apache-selfsigned-reg.homelab.key \
-out /etc/ssl/certs/apache-selfsigned-reg.homelab.crt
Enter fullscreen mode Exit fullscreen mode
Or just take it from previous step
sudo cp reg.homelab.crt /etc/ssl/certs/apache-selfsigned-reg.homelab.crt
sudo cp reg.homelab.key /etc/ssl/private/apache-selfsigned-reg.homelab.key
Enter fullscreen mode Exit fullscreen mode
Again, open virtual host config
sudo nano /etc/apache2/sites-available/reg.homelab.conf
Enter fullscreen mode Exit fullscreen mode
Add there to the bottom the SSL section with certs
<VirtualHost *:443>
ServerAdmin webmaster@localhost
ServerName reg.homelab
ServerAlias www.reg.homelab
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
ProxyPreserveHost On
ProxyRequests off
AllowEncodedSlashes NoDecode
ProxyPass / http://localhost:3000/ nocanon
SSLEngine on
SSLCertificateFile /etc/ssl/certs/apache-selfsigned-reg.homelab.crt
SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned-reg.homelab.key
</VirtualHost>
Enter fullscreen mode Exit fullscreen mode
Check config, restart server, check status and navigate to our Gitea over SSL
sudo apache2ctl configtest
sudo systemctl restart apache2
sudo systemctl status apache2
# goto: http://<Server_IP_Address>:443/
# or
curl -k -v https://localhost
Enter fullscreen mode Exit fullscreen mode
the browser will warn about the self-signed certificate
Your connection is not private
Attackers might be trying to steal your information from reg.homelab (for example, passwords, messages or credit cards). Learn more
NET::ERR_CERT_AUTHORITY_INVALID
Enter fullscreen mode Exit fullscreen mode
But we will ignore it, for now, till we use Let’s Encrypt.
Note: Once your Gitea SSL setup is working, consider backing up your server. See Backup and Restore Gitea server for detailed instructions.
Now k8s test
Do some DNSing…
On each k8s node:
sudo nano /etc/hosts
Enter fullscreen mode Exit fullscreen mode
and add there
192.168.18.200 gitea.homelab
192.168.18.200 reg.homelab
Enter fullscreen mode Exit fullscreen mode
Root CA
On each k8s node:
sudo cp SweetHome-RootCA.crt /usr/local/share/ca-certificates
sudo update-ca-certificates
Enter fullscreen mode Exit fullscreen mode
And restart now
Create secret with registry credentials
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
sudo docker login reg.homelab
kubectl create secret generic regcred --from-file=.dockerconfigjson=/home/rg/.docker/config.json --type=kubernetes.io/dockerconfigjson
Enter fullscreen mode Exit fullscreen mode
or
kubectl create secret docker-registry regcred --docker-server=your-registry-server --docker-username=your-name --docker-password=your-pword --docker-email=your-email
Enter fullscreen mode Exit fullscreen mode
New docker image and k8s deployment
sudo docker pull alpine:3.12.0
sudo docker images
sudo docker tag a24bb4013296 reg.homelab/rg/alpine:version.3.12.0
sudo docker push reg.homelab/rg/alpine:version.3.12.0
Enter fullscreen mode Exit fullscreen mode
Now nano alp2.yaml, file is [avaliable]avaliable
apiVersion: apps/v1
kind: Deployment
metadata:
name: alp-registry-test
spec:
replicas: 1
selector:
matchLabels:
app: alp-registry-test
template:
metadata:
labels:
app: alp-registry-test
spec:
containers:
- name: alpine-test
image: reg.homelab/rg/alpine:version.3.12.0
imagePullSecrets:
- name: regcred
Enter fullscreen mode Exit fullscreen mode
Here is this file available for download:
Then creating this deployment
kubectl create -f alp2.yaml
kubectl get pods
kubectl describe po alp...
Enter fullscreen mode Exit fullscreen mode
Cleanup
kubectl delete -f alp2.yaml
Enter fullscreen mode Exit fullscreen mode