Self-hosting side projects: principles and silent failures

My side projects used to each be deployed their own way: some were cloned onto the server and built there, some ran from a docker-compose file that only existed on that machine, and environment variables lived wherever they happened to land. It all ran, but every change meant first remembering how I'd done it last time.

This post covers how I brought it all under one roof, and the real mistakes I made along the way. If you run a server for your own projects, I hope a few of these save you an evening or two of debugging.

Put everything in git

The foundation: no hand edits on the server. Compose files, config, alert rules and dashboards all live in one repo. To change something, change it in the repo, push, and let CI apply it. Images are built in each app's CI, not on the server.

That gives you three things:

• History: every change has a commit, a reason, and can be reverted.
• Rebuildability: if the server is lost, you rebuild from the repo and a backup, not from memory.
• Drift detection: the rebuild script, run in --dry-run mode against the live server, should report "nothing to change". If it doesn't, git and reality have drifted apart.

One small mistake shows why versions must live somewhere readable: for a while, the database versions sat inside an encrypted secrets file. Diffs were unreadable, the dependency update tool couldn't see them, and some containers ended up running tags like latest that differed from what the file said, without anyone noticing. Now every image is pinned to an exact version right in the compose file.

Expose as little as possible to the internet

Little effort, large payoff: every service exposed to the internet is something to patch, monitor and worry about. A few ways to cut that down:

• Use a tunnel instead of opening ports. Services such as Cloudflare Tunnel let the server connect outward, so you don't need to open web ports on the machine. DDoS filtering and rate limiting happen before traffic reaches you.
• Separate the admin path from the user path. Admin access should go over a private network (a VPN), not the public internet.
• Don't route what's only used internally. Databases, caches and /metrics endpoints only need to talk to each other on Docker's internal network.
• Give each app its own database user, with rights on its own database only, instead of a shared admin account.

None of this makes a system "secure" in any absolute sense. It just reduces the number of places to worry about, so you can worry about the rest more carefully.

Deploys you can back out of

The current flow:

1. The app's repo runs its tests and builds an image.
2. The running version of each app is recorded in a file in the infrastructure repo, so the deploy history is simply that file's git log.
3. The new version first runs as a canary: a separate container that takes no traffic and has to pass its health check. Only then does it replace the running version; otherwise it rolls back on its own.

The biggest win isn't automation, it's that rolling back becomes boring: revert a commit and you're done.

Two real bugs I hit while building this, neither of which raised an error:

• cmd | grep -q in a script with set -o pipefail. grep -q exits as soon as it finds a match, the command before it gets SIGPIPE, and the whole pipeline counts as failed. The deploy script therefore reported "already up to date" instead of deploying. The fix: capture the output in a variable first, then grep the variable.
• Sharing a concurrency group between GitHub Actions workflows. As the GitHub Actions concurrency docs describe, GitHub keeps only one pending run per group by default, and a newer run cancels the pending one before it. Some deploys simply vanished. Each workflow now has its own group.

Secrets: no plain text in git

There are a few options, depending on scale:

• Encrypt secrets before committing them, with a tool such as sops, and decrypt only where they're used.
• Or use the secret store of the CI or hosting platform you're on.

Whichever you choose, three habits are worth keeping: give each component exactly the access it needs; write down how to rotate each secret before you need to; and rotate a secret as soon as it shows up somewhere it shouldn't, even if it's "probably fine".

Alerts: few, but right

• Alert on symptoms users feel (errors, slowness, unreachable), not causes. High CPU that hurts nobody isn't worth bothering anyone about.
• Check from the user's point of view. A probe that calls the public URLs along the same path users take (including TLS certificate expiry) catches failures that metrics inside the server can't see.
• Treat alert rules as code, with unit tests (Prometheus has promtool test rules).
• Check metric names against the real exporter before writing a rule. This is the mistake I find most dangerous: a rule that uses a metric that doesn't exist never fires and never reports an error. I had an "Elasticsearch is down" alert like that. It sat there looking reassuring, and could never have gone off.
• Monitor the monitoring. If monitoring runs on the server it watches, it goes quiet when the server dies. A regular "I'm alive" signal sent to an outside service solves that.

Backups only count if they restore

The rule I follow: a backup only counts once it has been restored successfully.

• Test-restore automatically and regularly into a temporary environment, using the same database versions as production, and check the data.
• Alert when a backup fails, when the latest one is too old, and when a test restore fails.
• Keep at least one copy off the server, and encrypt it.
• For upgrades that change a schema one way (such as Keycloak upgrades), try them first on a copy loaded from a backup, and take a fresh backup right before upgrading.

Other silent failures worth knowing

The mistakes that cost me the most time all had one thing in common: no error message.

• Bind-mount a file that doesn't exist yet and Docker creates a directory with that name, and the container runs as if nothing happened (Docker's bind mount docs say that with -v a missing source is always created as a directory, while --mount refuses to start). A custom MySQL config of mine was "applied" that way for 8 months, which is to say never.
• rsync --filter=':- .gitignore' only applies the rules on the receiving side if a .gitignore already exists there. A sync nearly deleted data and .env files because of it. Every sync with deletes now goes through a script that does a dry run first and refuses when it's about to delete anything that looks like data or a secret.
• Before moving an app to a new setup, compare the old and new configuration. One missing environment variable can switch off a feature without any error showing up.

Summary

• Keep compose files, config, alert rules and dashboards in one git repo and never edit the server by hand; a --dry-run rebuild that reports "nothing to change" shows git and the server still agree.
• Pin every image to an exact version in a file people and tools can read, not inside an encrypted secrets file.
• Expose as little as possible: a tunnel instead of open ports, admin access over a VPN, internal services only on Docker's internal network, and one database user per app.
• Deploy through a canary that must pass its health check, and record versions in git so rolling back is reverting one commit.
• Alert on symptoms users feel, test alert rules, check metric names against the real exporter, and monitor the monitoring from outside the server.
• A backup only counts once a test restore has succeeded.
• None of this takes much money: Cloudflare Tunnel and the Prometheus and Grafana stack have free tiers that are plenty for side projects. The hard part is discipline: no hand edits "just this once", and always asking "if this broke, would I know?".

If you've run into silent failures like these on your own server, I'd love to compare notes. My email is on the home page.