Self-Hosting Security: Hardening a Single VPS the Hard Way

First-hand security write-ups from one self-hosted VPS: a brute-forced SSH login, two crypto miners, the fail2ban/Docker trap, and the exact configs that closed the holes.

This page is an incident log, not a checklist

Everything on this page happened to one machine: a single Contabo VPS that runs a WordPress blog (igornews.com), a handful of FastAPI tools, and a few trading bots behind a single Caddy reverse proxy. Over the last month it survived a brute-forced SSH login, hosted two crypto miners, and taught me that some of the most common “hardening” advice is quietly useless once Docker is in the picture. I’m writing it down because the sanitized version you find in generic guides is not what actually keeps a box alive.

The threat model is boring on purpose. A VPS with a public IPv4 and SSH on port 22 starts getting scanned within minutes of coming online. Almost everything that hits it is automated credential-stuffing and opportunistic port scans, not a targeted human. So the whole game is to make the automated stuff fail loudly and early, and to notice quickly when something slips through anyway. If you do that, the targeted stuff never gets a foothold to aim at. It’s not exciting, and that’s exactly the point — a quiet box with a loud alarm is the whole win condition.

The breaches: two miners in a month

There were two, and they were unrelated. The first was the classic one — an unresponsive VPS that turned out to be a crypto miner dropped through a brute-forced SSH login. I wrote the full before-and-after numbers in SSH Brute-Force to Crypto Miner: Hardening a VPS the Hard Way. The short version: the attacker got a password I thought was strong enough, and the first thing I knew about it was the box falling over under load — the miner announced itself by consuming the CPU that my own services suddenly didn’t have.

The second was sneakier, and it’s the one that taught me the most. A Docker container spiked to 381% CPU — a miner hiding in a container, not on the host. It took three days of misdiagnosis because the obvious culprit (the ps output blaming a service by name) was lying to me. That story is in Crypto Miner in Docker: The Postgres Port I Forgot to Close, and the specific trick that fooled me — a container UID colliding with a host user so ps named the wrong process — is in The Miner in ps Wasn’t Ollama, It Was a Container.

The lesson that connects both: a breach is not one mistake, it’s a chain of small ones. A weak password, a port left open, a monitoring blind spot, a tool you didn’t know how to read. You don’t fix a breach by fixing one thing. You fix it by shortening every link in the chain at once, and by accepting that the next attack won’t look like the last one — it’ll look like the one thing you haven’t checked yet.

The firewall that was lying to me

After the SSH breach I did what everyone does: I installed fail2ban and pointed it at the brute-force attempts. And then I discovered that on a Docker host, the default fail2ban setup bans exactly nothing. The ban gets inserted into the wrong iptables chain, Docker’s own rules process the packet first, and the “banned” IP keeps hammering the port. The full walk-through — including how to confirm your bans are actually doing something — is in fail2ban Bans That Quietly Do Nothing on Docker Ports.

The fix is one directive: point the jail’s action at the DOCKER-USER chain instead of the default INPUT. Here is the actual jail running on this box, for a WordPress login brute-force jail that reads Caddy’s JSON access log:

[igornews-wp]
enabled   = true
filter    = igornews-wp
logpath   = /opt/igornews/logs/access.log
action    = iptables[type=allports, chain=DOCKER-USER, blocktype=DROP]
bantime   = 1h
findtime  = 10m
maxretry  = 6

Two things worth pointing out. First, chain=DOCKER-USER — that single token is the difference between a firewall that bans and a firewall that logs bans nobody ever felt. Second, the log it reads is Caddy’s JSON access log, which has a fractional epoch timestamp embedded in the middle of each line, so the filter needs a custom datepattern. Here’s the filter that makes fail2ban understand it:

[Definition]
datepattern = "ts":{LEPOCH}

failregex = ^\{"level":"info".*"msg":"handled request".*"remote_ip":"<HOST>".*"method":"POST".*"uri":"/wp-login\.php".*"status":200
            ^\{"level":"info".*"msg":"handled request".*"remote_ip":"<HOST>".*"method":"GET".*"uri":"[^"]*\?author=\d+

The two patterns catch different stages of the same attack. A failed WordPress login POSTs to /wp-login.php and gets a 200 back (a successful login returns a 302 redirect, so it’s deliberately not matched). The second pattern catches GET /?author=N — the username-enumeration step attackers run before they start guessing passwords. Blocking the recon step is worth more than blocking the guesses, because it kills the attack before it even knows which usernames exist. As of writing, this single jail has logged 10 failed attempts and issued 1 ban — small numbers, but they’re small because the box stopped being an easy target, not because nobody’s trying. A jail that never fires because the attack stopped coming is a jail that’s working.

Rootkit scans and the files that were mine

After a miner, the natural impulse is to run a rootkit scanner on everything. I ran rkhunter. It flagged four hidden paths. All four were mine. The full list and the reasoning is in rkhunter Flagged Four Hidden Paths. All Four Were Mine, but the point that matters here is the discipline, not the list.

A rootkit scanner that you can’t interpret is worse than no scanner, because it either trains you to ignore its warnings (and then you miss the real one) or it panics you into reverting something legitimate. The correct workflow is: run the scan, read every single hit, and for each one decide why it’s there. Only after you’ve confirmed a path is genuinely yours do you whitelist it. Here’s what the whitelist looks like on this box:

ALLOWHIDDENDIR=/usr/.qwen
ALLOWHIDDENFILE=/usr/bin/.summarizer-*
ALLOWHIDDENFILE=/etc/.resolv.conf.systemd-resolved.bak
ALLOWHIDDENFILE=/etc/.updated

Every one of those has a specific, boring explanation: /usr/.qwen is a dot-directory a local model tool creates, /usr/bin/.summarizer-* is a symlink a summarizer utility leaves behind, and the other two are files the system itself writes. If I’d blindly whitelisted them on day one, I’d never have built the habit of reading the output — and the habit is the actual security control. The scanner is just the thing that forces you to look.

The layered checklist, with real configs

There is no single tool that secures a VPS. There’s a stack of cheap layers, each one catching what the one above it misses. Here’s the exact stack on this machine, in order of how much pain each one has saved.

1. SSH: keys only, root login crippled

PermitRootLogin prohibit-password
PasswordAuthentication no

That’s the whole SSH hardening, and it’s the highest-leverage two lines on the box. prohibit-password still allows root login with a key (useful for automation) but kills the password path the attacker used. PasswordAuthentication no removes the credential-stuffing surface entirely — there’s nothing to guess. If you do nothing else on this page, do these two.

2. ufw: default-deny, and the ports you forgot

Incoming is deny by default; only four ports are open — 22 (SSH), 80 and 443 (the Caddy edge), and 5678 (n8n). Two more ports are explicitly denied rather than just left closed: 11434 (Ollama) and 5432 (Postgres). Those two got their own deny rules because they were the ones the miner actually used — they were reachable once through a misconfigured container, and now they can’t be, even if a container binds them again. On top of that, there are 14 manual DENY IN rules pinning specific brute-force source IPs that kept coming back.

The takeaway isn’t “use ufw.” It’s “default-deny, and write an explicit deny for any port that was ever a problem.” A port that’s simply not opened is one config typo away from being open. A port with an explicit deny rule is one typo away from still being denied. The difference is whether a mistake fails open or fails closed.

3. fail2ban: the DOCKER-USER chain

Covered above — but it earns its own slot because the jail is only useful if it’s pointed at the right chain. The jail, the JSON log, and the filter are all shown in the section above. The one thing I’ll repeat: verify your bans actually drop packets, don’t just trust that fail2ban “is running.”

4. The edge: Caddy does the security headers, once

Every request to the blog and the tool subdomains terminates at one Caddy container. That’s the single place to put security headers, and it’s also the single place to kill xmlrpc.php — a WordPress brute-force amplifier that has no business being reachable:

igornews.com, www.igornews.com {
    @xmlrpc path /xmlrpc.php
    respond @xmlrpc "Forbidden" 403

    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "SAMEORIGIN"
        Referrer-Policy "strict-origin-when-cross-origin"
        Permissions-Policy "geolocation=(), microphone=(), camera=(), interest-cohort=()"
        -Server
        -X-Powered-By
    }
    ...
}

The -Server and -X-Powered-By lines strip the version-stamp headers that make automated scanners’ job easier. The @xmlrpc block is the important one for WordPress: it answers XML-RPC with 403 at the edge, before the request ever reaches PHP, so the amplification vector just doesn’t exist. And because Caddy is the edge and terminates TLS itself, it sees the real client IP — which is the whole reason the fail2ban jail above works at all. Behind Cloudflare’s proxy (orange cloud) or a naive reverse proxy, every request would appear to come from one IP and the jail would ban nothing useful.

5. The app layer: a WordPress mu-plugin

Inside WordPress, a single must-use plugin does the version-strip (removes the generator tag), disables XML-RPC and pingbacks, removes the RSD/WLWManifest link noise, and turns off the /wp/v2/users REST endpoint so username enumeration is dead even if someone reaches the app directly. None of this is visible to a visitor, but it removes the cheap recon surface that the ?author=N jail and the xmlrpc block target from the outside. Defense in depth isn’t “do the same thing twice”; it’s “make the attacker clear a different, harder hurdle at each layer.”

6. Patching: a watchdog that applies the fix the day it ships

When the PHP “PHP2Shell” RCE advisories (CVE-2026-109618/109619) landed in late October, the official wordpress:php8.3-apache image didn’t carry the fix yet, and a naive docker compose pull would have recreated the container onto an equally-vulnerable build. The answer was a small watchdog cron that polls the image, reads the actual PHP_VERSION from its environment, and only re-deploys when the patched version actually exists. It stays silent until there’s something to apply, then applies it and pings me. Patching a containerized PHP app is a process, not a one-time action — the floating tag means “latest” can lag the fix for hours or days, and “I pulled the latest image” is not the same as “I’m running the fixed version.”

The container layer deserves its own paragraph

Sixteen containers run on this box at any given time — WordPress, Caddy, MariaDB, n8n, three Hummingbot pieces, OctoBot, a paper-trading bot, SimStudio, a couple of FastAPI tools. The single most dangerous assumption I made was that Docker somehow sandboxes the network. It doesn’t. A container that binds a port to 0.0.0.0 publishes that port to the host, and the host firewall is the only thing standing between that port and the internet. The Postgres port that the second miner used wasn’t opened by me — it was published by a container I’d misconfigured, and because I’d been thinking “it’s in Docker, it’s contained,” I didn’t check what the container was actually exposing.

The flip side of the same lesson: the artifacts Docker leaves around can look like malware when they’re not. The transient /tmp/runc-process* files that appear during docker exec are a legitimate part of how runc works — I nearly chased them as a compromise. The full story is in Don’t Panic Over Docker’s Exec Artifacts. The takeaway is the same in both directions: Docker is not a security boundary, and you have to know what “normal” looks like for your own containers before you can tell “weird” from “compromised.”

Detection: how I’d notice the next one

Prevention fails eventually. What saved me both times was that the miners were noisy — they burned CPU, and the slowdown was visible. But “wait until the box feels slow” is not a detection strategy. Here’s what actually watches this box now:

  • A daily security scan runs every morning and reports over Telegram — container state, auth-log anomalies, unexpected listeners, and the fail2ban/rkhunter status. The point is that it’s scheduled and external, not something I remember to do. A control you have to remember is a control that stops existing the week you get busy.
  • A silent-failure watchdog checks every few hours that the things that are supposed to be running still are. A miner often isn’t a new process — it’s a legitimate service that’s been replaced or repurposed, and the first sign is something stopping, not something starting.
  • The patch watchdog from the previous section doubles as an early-warning system: if a vendor fixes a CVE, I learn about it the moment the patched image ships, not when I next read the news.

The through-line is that none of these require me to be paying attention at the right moment. Automation is the only detection that works for a box you self-host in your spare time.

Recovery and the discipline of backups

A miner usually means you can’t trust anything on the box until you’ve rebuilt it. The only thing that makes that survivable is a restore path you’ve actually tested. The one data-integrity scare I had — a SQLite database that went corrupt mid-operation — is written up in Repairing a Corrupt SQLite Database with .dump and COMMIT. The lesson there applies to security too: WAL-mode writes and a tested .dump snapshot are the difference between a five-minute recovery and a lost week. I keep the “corrupt original” file around after a repair instead of deleting it, on the principle that a damaged artifact is still evidence — you never know what you’ll need to look at again.

If the box were fully compromised tomorrow, the recovery is: re-provision, re-run the container definitions (they’re all in version-controlled compose files), restore the data volumes from snapshot, and re-apply the layers on this page. None of that is glamorous, and all of it only works because the configs live in files I can re-run, not in state I’d have to remember.

What to choose, and why

Every tool above has an alternative people will tell you is better. Here’s why I ended up where I did, and when you should pick the other thing.

SSH authentication: password vs key vs 2FA

Method Trade-off
Password Guessable, stuffable, and the exact vector the first miner used. Never on an internet-facing SSH.
Key only The default for a reason. Nothing to guess, but a lost key locks you out — keep an emergency console path.
Key + 2FA Adds a real second factor, at the cost of friction on every login and more moving parts. Worth it only when the box is high-value enough to justify the pain.

I run keys-only with a console recovery path as the backstop. 2FA on SSH is genuinely stronger, but for a solo box the operational overhead outweighed the added protection once password auth was already dead — the marginal attacker who defeats a key is rare enough that the friction wasn’t paying for itself.

fail2ban vs CrowdSec vs ufw alone

Tool What it does When to pick it
fail2ban Parses logs, bans offending IPs via iptables/nftables You have a known log pattern (SSH, wp-login) and want a simple, transparent ban. Zero external dependency.
CrowdSec Same idea, but with a shared community blocklist and a reputation model You’re willing to run one more daemon and want IP reputation shared across the internet, not just your logs.
ufw alone Static allow/deny rules Only as a base layer. It can’t react to an attack; it’s the default-deny floor the reactive tools sit on top of.

I run fail2ban because the attack surface here is a known, small set of log patterns and I want to be able to read exactly what a ban did. CrowdSec’s shared blocklist is a real advantage for a box that sees novel attacks, but it’s one more moving part to debug when a ban misfires. ufw is not an alternative to either — it’s the layer underneath.

rkhunter vs chkrootkit vs Lynis

Tool Scope What it’s good at
rkhunter Rootkits + hidden files/processes Quick, targeted rootkit triage. Noisy on a busy box — needs whitelisting discipline.
chkrootkit Rootkits Older, smaller signature set. Fine as a second opinion, rarely the primary.
Lynis Full host audit The broad health check — permissions, config, patching. Slow but thorough; run it for the audit, not for the panic.

rkhunter is the daily driver because the specific fear after a miner is “is there something hiding now,” and that’s exactly what it targets. Lynis is the quarterly deep audit. chkrootkit I keep around as a cheap second opinion, but it hasn’t found anything rkhunter didn’t.

Caddy vs nginx/traefik for a single edge

Edge Why you’d pick it
Caddy Automatic TLS, headers and redirects in one readable file, no separate cert bot. The right default for a small self-hosted stack.
nginx Maximum control and the biggest ecosystem, at the cost of hand-managing TLS and more verbose config.
traefik Native Docker service discovery and labels. Wins when you have many short-lived containers that need dynamic routing.

I picked Caddy for the single reason that TLS is automatic and the config stays in one human-readable file — which is exactly where the security headers and the xmlrpc block live, one glance away. If this stack grows to dozens of dynamic containers, traefik’s label-based routing starts to pay for itself; until then, Caddy is less to maintain and fewer moving parts to misconfigure.

What I got wrong

Three things, in the order they cost me:

  • I trusted a password. The first miner got in through SSH with a password I’d convinced myself was long enough. It wasn’t. Keys-only would have stopped it cold, and I only switched after the fact.
  • I trusted ps. The second miner hid behind a container UID collision that made ps name the wrong process. The fix was learning to read the /proc and cgroup truth instead of the human-friendly label.
  • I trusted the firewall default. fail2ban was “working” for days — logging bans — before I realized the bans landed in a chain Docker ignores. Monitoring the effect of a control, not just its existence, is the habit I had to build.

The deep-dives

Each of these is the full incident, with the exact commands and the exact numbers. Read them in order — they build on each other.