From Regular Docker to Docker Swarm to Kubernetes to Podman to Regular Docker Again

I have a nasty habit of taking something is working perfectly fine using a technology that is perfectly fine then hopping to a bunch of different things out of boredom or curiosity or following “best practices”.

When first starting my home server / lab, I followed what most people did and setup Docker + Docker Compose on Debian to run various self-hosted applications in containers. I was already pretty familiar with Docker since I am a software engineer so it made sense and I was able to get running pretty fast. I learned a ton doing this and had fun, in fact it became a borderline obsession.

After much trials and tribulations I finally managed to get it to a decent and stable state. Services like Syncthing, Nextcloud, Gitea, Jellyfin, and etc became extremely beneficial to me. But of course once everything was stable, I got curious other ways of container orchestration methods and potentially doing things in a “better” way.

Docker Swarm

Enter Docker Swarm. Docker Swarm is the built-in method for orchestrating Docker containers in a multi-node (multiple computers) environment. Sort of similar to Kubernetes (more on that later), but also not really because Kubernetes is VASTLY more complex and introduces even more layers of abstraction.

To be honest I had no reason to setup Docker Swarm. I only have one server, although I did experiment with a multi-nodal environment by setting up my server as the manager node and couple of my laptops as worker nodes. My primary motivation was learning and making my setup feel more “professional”.

One cool thing Docker Swarm has that Docker Compose does not is a proper secrets manager. To use secrets in Docker Compose you have to write them to a text file and then refer to that text file in the compose file. With Docker Swarm you can just use docker secret. So I wanted to try as a potentially more secure way of keeping my secrets rather than just leaving them in text and .env files.

Obviously learning is great but where I messed up was immediately trying to convert my already working Docker Compose setup to Docker Swarm in production (production being in use by me), without a solid way to revert back or doing a more thorough analysis on whether I will have all the same features I rely on in Docker Compose in Docker Swarm. Turns out no, no I would not have all the same features I needed, Docker Swarm is behind Docker Compose in a few ways despite.

Where I really began to have issues with was with the torrenting setup. I was using both Deluge and qBittorent for torrenting Linux ISOs on my server, and had these services running behind a Gluetun VPN container for privacy.

Gluetun is a really amazing project that runs a VPN client in a Docker container, which then you can connect other containers to proxy that container’s traffic through the VPN on Gluetun. It supports most of the popular VPN providers and custom configurations for OpenVPN and Wireguard.

Anyway, to quote myself from earlier notes I had written while trying to migrate this service to Docker Swarm:

Turns out trying to do something as simple a route a container’s traffic through a vpn is really fucking hard in swarm. Everything else about swarm was nice but this was the major gotcha I ran into. Swarm services do not have NET_CAP capabilities so gluetun doesn’t work, I tried using swarm-launcher as wrapper for gluetun which gives it access to options availability to docker compose services but not docker swarm, but then it doesn’t work with docker swarm secrets. There is little help online for this specific use-case other than using a .env file which I really did not want to do since I just moved all secrets off of .env files and into a proper secret manager. So fuck it, I give up, I’m just keeping the VPN clients off of swarm.

So yeah, I was running this frankenstein system where some of my applications were running in Docker Compose and some where running on Docker Swarm. I wrote some scripts to try and manage it but eventually I got tired of this crap and just wanted stuff to work normally so I went back to Docker Compose.

This is not to say I hate Docker Swarm though. It is probably a good option if you need to host your project or app in a multi-nodal environment, already using Docker, not doing anything complicated with networking and do not want to learn and maintain the beast that is Kubernetes. But for my use case: no, not worth it. Was cool to learn though.

Despite this, I still had not learned my lesson on multi-nodal container orchestration, as you will see if you continue to read my notes:

I had no plans to switch over to Kubernetes if Docker Swarm worked well enough but it turns out no it’s not going to work well enough, so I will try Kubernetes in the future.

Oh no…

Kubernetes

Now, if you are in anyway involved with software engineering or IT you’ll probably know Kubernetes is hot shit right now. I went to a developer conference last year and most of the biggest tech companies there were trying sell either their Kubernetes solution or AI solution. Around this time I was also personally very interested in Kubernetes.

If you do not know however, Kubernetes is a platform for the orchestration of containerized applications at scale across many nodes/computers (i.e. a “cluster”). It was developed and later open-sourced by Google to solve Google-sized problems. Despite not being Google nor having Google-sized problems, many other smaller companies and individuals want to use it for their own projects, me included.

The company I work for uses Kubernetes to deploy their applications. Lots of companies do. At this time I was doing some Kubernetes training at work. Naturally I wanted to apply it (the specific work I was doing was not at all Kubernetes related), so I thought what I great idea it would be to setup a cluster on my home server and migrate all my stuff to Kubernetes since I had so much frustration with Docker Swam but still wanted to try out a more “production-grade” I guess method of container orchestration.

Now one of the first bad assumptions I made was that Kubernetes was just like Docker Swarm but without the annoying issues I was facing because Kubernetes has way more popularity and support than Swarm. WRONG. Kubernetes basically adds a whole new layer of abstraction on top of all the Linux and containerization knowledge I was already familiar with. Being even just a bit familiar with the underlying technology may make it easier to learn, but Kubernetes really feels like a totally new subject in its own right rather than just a step up from Docker Compose.

I did manage to setup a cluster, get kubectl running on my laptop, deploy some of the services I use as pods, setup up some services and etc, but once I got into learning Kubernetes networking, MetalLB, volumes, and etc I began to realize this was massively overkill and I was setting myself up for a lot more work and maintaintence than I bargained for.

Now if I just wanted to learn Kubernetes it would have made sense to just spin a few VMs and create a cluster using that and then deploy a web app or two on that cluster. But I was trying to use this to run my home server which at this point I was using for all sorts of things like file storage, Nextcloud, Gitea for git repo storage, and etc.

I realized the error of my ways pretty early on before I got too deep into it and threw it at all away and destroyed the cluster. Back to Docker Compose.

But alas, I still was not done tinkering with and blowing up my homelab. Enter Ansible and Podman.

Ansible and Podman

To preface, I do not regret putting time in to learning Ansible, Ansible is an awesome tool and has been a great help for automating the setup and configuration of my Linux devices (I have a lot of Linux computers). Ansible is an automation and infrastructure-as-code tool developed by software engineer Michael DeHaan and soon after acquired by Red Hat focused mainly on Unix-like systems. You write YAML files to describe how to configure your system and Ansible uses SSH to just do it. The advantage of this is being able to configure massive amounts of Linux systems in a reproducible way from just a single command. Ansible is also agentless meaning you do not need to install Ansible or anything else special on any of the hosts you are targeting (other than I think Python which to my knowledge should already be installed on any modern Linux distro).

That being said Ansible is not a perfect tool, I won’t really get into my gripes with it here but if I write about Ansible more in the future then I will.

Podman is another Red Hat product, it is another containerization platform like Docker that aims to be a (almost) drop in replacement for Docker while running containers as a regular user rather than root, being daemonless, and integration with systemd (you can run containers as systemd services). Podman also as some integrations with Kubernetes like including pods and providing the ability to generate Kube files to serve as a stepping stone for getting into full blown Kubernetes.

My interest in Podman began when I wanted to try and improve the security on my home server. Running containers as root is a potential security hole since it really is unnecessary and violates the principle of least privilege. If a bad actor were to gain access to one of your running containers and then managed to escape the container to the host, they would have root privileges, which is obviously not good. There is Docker Rootless but it is not the default way to run Docker. I was also really curious about the ability to run containers as systemd services rather than using the Docker daemon. It felt like a better way to integrate containers as apart of the whole system so I wanted to try it.

I had also received the book Podman in Action for free from some Red Hat guys at the conference I attended which had also gotten me very excited about Podman.

It was around this time I was actually building a brand new server. Previously I had running everything on a used Lenovo ThinkCentre M720s SFF PC I picked up from eBay. It served me very well but I really wanted to build my own server from scratch (I had just built my first desktop PC and immediately fell in love with PC building) and to have a rack server instead of an old office PC since I had also just gotten a server rack. So I built the server, installed Debian, setup RAID with two 12 TB hard drives to use as my main data drives, racked it, then just before setting up my services again, I thought this would be a great opportunity to give Podman a try. What could go wrong? Turns out a lot.

My first screw up was using Debian if I really was dead set on using Podman. You see, Debian being Debian, of course ships with an older version of Podman than what upstream as released as the latest stable version. Normally this isn’t a issue for me as I usually don’t care for running the latest and greatest shiny new version of things. But with Podman it turns out this was kind of a big deal because the version of Podman that ships with Debian does not include Quadlets.

Quadlets is basically Podman’s equivalent of Docker Compose. It does not use yaml files however, instead these .container files which have a syntax most similar to that of systemd unit files.

Not the end of the world though. I could still use Docker Compose with Podman. However, these containers would not start automatically at boot like with docker compose because there is no Docker daemon. One potential solution would to have been to use Docker Compose to create the container, then use the podman systemd generate command to generate a systemd file. That could have worked fine, but I opted to make things harder and less maintainable for myself by setting up the containers and generate systemd services using Ansible.

Not that Ansible is a bad tool for this, it just honestly made the setup less maintainable and added a lot of friction to adding new services to my home server, since everyone on the internet making self-hosted services just assumes you’re using Docker Compose and Podman is usually an after thought at best. After a while of having set up everything on Podman running kinda sort of ok but not really, I just got sick of it all and the extra steps and I am currently going back to Docker Compose.

Beyond the annoying system I had come up with for orchestration my containers, the migratation of all my data and container setups from Podman to Compose was not very smooth. For one it just was tedious to move over all the volumes and check that everything was moved over ok, and second I ran into some problems with various containers due to them now having to run either in a rootless environment or with Podman specifically.

The biggest problem was Nextcloud. I spent so much time trying to get the Nextcloud AIO container to work on Podman, in a Pod, deployed with Ansible. Eventually I just finally gave up and accepted Nextcloud AIO does not work with Podman, and tried the now community maintained regular Nextcloud Docker image (Nextcloud now officially recommends and pushes the AIO solution). This of course created a new huge migration task of moving my stuff that had been in the Nextcloud AIO setup to a regular Nextcloud setup. As a side note, I found that the Nextcloud AIO project is great IF you just follow one of the happy paths laid out for you in their documentation, doing anything too custom or weird like using Podman just seems to become a major PITA.

So guess what, I finally got the community Docker image to work and all of my files I had in Nextcloud AIO where there, but still I never got it work as smoothly as I did when I had it running with the AIO container in Docker Compose. The last straw was when I was try to reorganize my notes in Nextcloud. I have them synced to my computer using the Nextcloud desktop client, so I was just in the terminal trying to move around some files and I started to run into really weird permission issues. I was getting a 403 error in the client just trying to rename files with mv and nothing would sync. This behavior was present on all my Nextcloud clients.

At this point I had dealt with enough bullshit trying to get Nextcloud to work so I said screw it, I’m just going to go back to Docker Compose. I’m tired of futzing with and breaking it and I just want my stuff to work now so I can move on and invest time in other projects. So that is what I am in the process of doing right now, moving all my stuff back to regular Docker on Docker Compose and then I will leave it the hell alone.

To conclude though, Podman is not a bad tool it’s just I would not run it on Debian again, only on something in the RHEL family or something that always gets the latest software like Arch, and also it is important to understand there will be some mishaps trying to move all your stuff to rootless since most container images you are going to use will probably assume that Docker is being run as root. So do it in a testing environment first and decide if it is worth it.

Conclusion

Potentially a lot of the issues I faced were due to my own skill issues and inadequate planning, but like I said for now I just my stuff to work so I can move on to other projects instead of stressing over an IT dumpster fire of my own making (it’s suppose to be fun, not a second job!). In the future if I want to play with a new technology for my server (NixOS, FreeBSD…?) I will do so in a testing environment, try my most critical services first to make sure I have will have all the capabilities I need and then decide if the switch over is really worth it.

If you are setting up a Debian home server for yourself, I recommend just making things easy for yourself and just using Docker and Docker Compose, that’s what everyone does, and it’s the most supported, has the most tutorials and guides on it, and will be the easiest to troubleshoot for online.

As for security, to be honest I think I have been a bit over paranoid. My server is not exposed to the internet and is behind a firewall. I should not have to worry about someone from the outside hacking into my containers. However potentially pulling a bad container image in an update is something that still worries me, but I suppose that is the same worry as pulling any software from an upstream source.

Thanks for reading.