In the future, I'll have a raspberry pi cluster. I'm toying in advance with installing kubernetes and various apps, to learn more about k8s and see if it's all worth it for homelabbing.

I settled on installing Talos linux and I've seen they have a magic cluster management thing called Omni. This post is about installing in on a local PC (note, this is a docker thing which runs outside the cluster itself, because we use it to set up the cluster. So meta!)

Prepare Gitea

Install

We install gitea via the following command:

sudo docker run -it \
    -v $PWD/certs/privkey.pem:/data/gitea/key.pem \
    -v $PWD/certs/fullchain.pem:/data/gitea/cert.pem \
    -v $PWD/gitea/app.ini:/data/gitea/conf/app.ini \
    -v $PWD/gitea/data/:/data/gitea/ \
    -p 3000:3000 \
    gitea/gitea:1.19.3

Gitea setup

This is just the bare minimum setup to run Omni. Gitea has many additional configuration options and security measures to use in accordance with your industry’s security standards. More information on the configuration of Gitea can be found (here)[https://docs.gitea.com/].

Create a user

Click the Register button at the top right corner. The first user created will be created as an admin and permissions which can be adjusted accordingly afterwards if you like.

Create organizations

After registering an admin user, the organizations, can be created which will act as the package repositories for storing images. Create the following organizations:

  • siderolabs
  • keycloak
  • coredns
  • etcd-development
  • registry-k8s-io-proxy
NOTE: If you are using self-signed certs and would like to push images to your local Gitea using Docker, you will also need to configure your certs.d directory as described here.

Prepare images

The documentation tells us we need to prepare the images. I've picked up a gitea-based configuration, because that's what I'm using.

First, we download the images (docker pull):

line="docker.io/gitea/gitea:1.19.3 quay.io/keycloak/keycloak:21.1.1 ghcr.io/siderolabs/imager:v1.4.5 ghcr.io/siderolabs/flannel:v0.21.4 ghcr.io/siderolabs/install-cni:v1.4.0-1-g9b07505 docker.io/coredns/coredns:1.10.1 gcr.io/etcd-development/etcd:v3.5.9 registry.k8s.io/kube-apiserver:v1.27.2 registry.k8s.io/kube-controller-manager:v1.27.2 registry.k8s.io/kube-scheduler:v1.27.2 registry.k8s.io/kube-proxy:v1.27.2 ghcr.io/siderolabs/kubelet:v1.27.2 ghcr.io/siderolabs/installer:v1.4.5 registry.k8s.io/pause:3.6"

arr=($line)

for i in ${arr[@]}; do docker pull $i; done

We save all images in a tarball:

docker save -o registry/all_images.tar \
  docker.io/gitea/gitea:1.19.3 \
  quay.io/keycloak/keycloak:21.1.1 \
  ghcr.io/siderolabs/imager:v1.4.5 \
  ghcr.io/siderolabs/flannel:v0.21.4 \
  ghcr.io/siderolabs/install-cni:v1.4.0-1-g9b07505 \
  docker.io/coredns/coredns:1.10.1 \
  gcr.io/etcd-development/etcd:v3.5.9 \
  registry.k8s.io/kube-apiserver:v1.27.2 \
  registry.k8s.io/kube-controller-manager:v1.27.2 \
  registry.k8s.io/kube-scheduler:v1.27.2 \
  registry.k8s.io/kube-proxy:v1.27.2 \
  ghcr.io/siderolabs/kubelet:v1.27.2 \
  ghcr.io/siderolabs/installer:v1.4.5 \
  registry.k8s.io/pause:3.6

Push Images to Gitea

Now that all of our organizations have been created, we can push the images we loaded into our Gitea for deploying Keycloak, Omni, and storing images used by Talos.

For all of the images loaded, we first need to tag them for our Gitea.

sudo docker tag original-image:tag gitea:3000/new-image:tag

For example, if I am tagging the kube-proxy image it will look like this:

NOTE: Don’t forget to tag all of the images from registry.k8s.io to go to the registry-k8s-io-proxy organization created in Gitea.
docker tag registry.k8s.io/kube-proxy:v1.27.2 ${GITEA_HOSTNAME}:3000/registry-k8s-io-proxy/kube-proxy:v1.27.2

Finally, push all the images into Gitea.

docker push ${GITEA_HOSTNAME}:3000/registry-k8s-io-proxy/kube-proxy:v1.27.2

Install Keycloak

Omni depends on keycloak, so we need to install it:

sudo docker run -it \
    -p 8080:8080 \
    -p 8443:8443 \
    -v $PWD/certs/fullchain.pem:/etc/x509/https/tls.crt \
    -v $PWD/certs/privkey.pem:/etc/x509/https/tls.key \
    -v $PWD/keycloak/data:/opt/keycloak/data \
    -e KEYCLOAK_ADMIN=admin \
    -e KEYCLOAK_ADMIN_PASSWORD=admin \
    -e KC_HOSTNAME=${KEYCLOAK_HOSTNAME} \
    -e KC_HTTPS_CERTIFICATE_FILE=/etc/x509/https/tls.crt \
    -e KC_HTTPS_CERTIFICATE_KEY_FILE=/etc/x509/https/tls.key \
    ${GITEA_HOSTNAME}:3000/keycloak/keycloak:21.1.1 \
    start

Install Omni

Once all these preparations are done, we can install Omni. First, we need an omni account:

export OMNI_ACCOUNT_UUID=$(uuidgen)

With all that was set up before, we run a (very long) docker command:

# Run omni

sudo docker run \
  --net=host \
  --cap-add=NET_ADMIN \
  -v $PWD/etcd:/_out/etcd \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v $PWD/certs/fullchain.pem:/fullchain.pem \
  -v $PWD/certs/privkey.pem:/privkey.pem \
  -v $PWD/certs/omni.asc:/omni.asc \
  ${GITEA_HOSTNAME}:3000/siderolabs/omni:v0.12.0 \
    --account-id=${OMNI_ACCOUNT_UUID} \
    --name=omni \
    --cert=/fullchain.pem \
    --key=/privkey.pem \
    --siderolink-api-cert=/fullchain.pem \
    --siderolink-api-key=/privkey.pem \
    --private-key-source=file:///omni.asc \
    --event-sink-port=8091 \
    --bind-addr=0.0.0.0:443 \
    --siderolink-api-bind-addr=0.0.0.0:8090 \
    --k8s-proxy-bind-addr=0.0.0.0:8100 \
    --advertised-api-url=https://${OMNI_HOSTNAME}:443/ \
    --siderolink-api-advertised-url=https://${OMNI_HOSTNAME}:8090/ \
    --siderolink-wireguard-advertised-addr=${OMNI_HOSTNAME}:50180 \
    --advertised-kubernetes-proxy-url=https://${OMNI_HOSTNAME}:8100/ \
    --auth-auth0-enabled=false \
    --auth-saml-enabled \
    --talos-installer-registry=${GITEA_HOSTNAME}:3000/siderolabs/installer \
    --talos-imager-image=${GITEA_HOSTNAME}:3000/siderolabs/imager:v1.4.5 \
    --kubernetes-registry=${GITEA_HOSTNAME}:3000/siderolabs/kubelet \
    --auth-saml-url "https://${KEYCLOAK_HOSTNAME}:8443/realms/omni/protocol/saml/descriptor"

Conclusion

This is a quite manual exercise and has a license limitation (can't be used for free in production environments), so its benefits are arguable, even for homelabbing. I'd wait for a process based on docker compose, ideally with a setup script for a toy environment.

HTH,