I recently picked up a used Dell OptiPlex 3070 to replace my aging Pi, and went down a (fun!) rabbit hole of automating provision/management using a combination of Ansible and Docker Compose.

Setup

I started fresh from a debian-standard ISO image and configured the net interface. Docs were consulted heavily (thank you Archwiki/Debian docs), as it has been awhile since I did this! Turns out the ethN etc. interface names are all deprecated, and interfaces now have more unique names. I had to dig around to figure out what to use, via:

$ ls /sys/class/net

That gave me enp1s0, which I stuck into /etc/network/interfaces:

auto enp1s0
allow-hotplug enp1s0
iface enp1s0 inet dhcp

And with that, I now have a working network connection. The next step was setting up SSH keypair access (ssh-keygen + ssh-copy-id), and then I let Ansible take over.

I use an Ansible playbook to automate the server config. It does the bare host-system setup: timezone/NTP, disables IPv6 (because my ISP doesn't support it... in 2024; though I could just easily use it on my internal network), installs Docker, etc.

All services are managed through Docker Compose that gets invoked via Ansible.

Management

Everything is managed through Ansible. Services are containerized and managed
by Docker Compose (also via Ansible). This means that the directory structure looks like:

  • ./ansible/ - Ansible-related manifests
  • ./services/<service-name> - Service-related files, i.e.:
    • A docker-compose.yml file - manages/runs the service(s)
    • An Ansible playbook.yml file - deploys the service
    • Any service-related files (e.g. configuration)

You can find the Ansible task responsible for provisioning a service here.

Reverse Proxy & DNS

I use nginx as a reverse proxy to expose services over my internal network through the following domains:

I also run PiHole. It comes with dnsmasq that I leverage as an all-in-one DNS
solution. Each DNS record is set up via Ansible tasks in ./services/pihole-and-dns/playbook.yml#L24-L50.

You can also do this via PiHole's UI under Local DNS > DNS Records, but I prefer keeping mine programmatic and idempotent.

Unblocking iCloud Private Relay

PiHole blocks iCloud Private Relay by default. You can unblock it by setting BLOCK_ICLOUD_PR=false in its FTL conf file. I did it via an Ansible task here.

SSL via LetsEncrypt

I registered the domain gallifrey.sh and stuck it on Cloudflare, then used Certbot to auto-provision LetsEncrypt certs via Cloudflare DNS challenge. This means I get free SSL over my internal network without having to deal with any self-signed certs.

You can find my certbot command in services/nginx-certbot/docker-compose.yml#L12-L21.

Spinning up new services

Since everything's managed via Ansible, adding a new service means creating a
new directory under ./services/<service-name>, tossing in a docker-compose.yml, and an Ansible playbook to deploy it.

Check out these commits to see how I deploy an observability stack using Grafana, cAdvisor, NodeExporter, and VictoriaMetrics:

After the two commits above, I run the following playbooks:

# Deploy observability stack
$ ansible-playbook \
    -i ansible/inventory.yml \
    services/observability/playbook.yml \
    --ask-become-pass
# Update nginx conf
$ ansible-playbook \
    -i ansible/inventory.yml \
    services/nginx-certbot/playbook.yml \
    --ask-become-pass
# Update DNS
$ ansible-playbook \
    -i ansible/inventory.yml \
    services/pihole-and-dns/playbook.yml \
    --ask-become-pass

Et voila:

Improvements

  • The Ansible stuff is rather repetitive. I guess that's YAML for you - I'm sure I can cut down on some of the repetition, but that's a rabbit hole for another day
  • Make a "deploy all" Ansible playbook? The reverse proxy + DNS stuff is repetitive and running three playbooks to deploy a single service is a bit jarring
  • Logs! I should set up something to collect logs from all my services
  • More services! Now that my Pi's spec is no longer a limiting factor, I could toss in Jellyfin/Sonarr etc. and have everything running smoothly
  • Separate DNS/PiHole using my old Pi? A single host means a single point-of-failure, and DNS is more critical than others
  • Figure out a better way to provision services? I'm not a fan of wrangling YAML in general

You can find all of my configuration in this repository: half0wl/homelab.