...blog

Here is a weird one.

I have a server exposing web ports to the internet. As most (all?) servers do, this one gets a fair amount of scans ant things that don’t really add any value to me. Mostly nmap path enumerations and other kinds of scripts looking for something that was never there.

I’d like them not to. Since that’s not really up to me, I’d like to block them.

I already use fail2ban to block any brute force attempt on the ssh service on the same server, so it clearly it should be easy peasy. Right?

Caddy

I use Caddy as the web server, mostly because it has stupid-easy to deploy tls certificates. Caddy encoders are not really fail2ban-friendly: console encoder is not really meant to output to a file - and json encoder outputs… json, fail2ban is regex based, so you can make it work, but it’s not ideal.

But it’s what I have, so it’s what I’ll do.

Fail2ban filter

I found some regex online that said that worked on the standard caddy logs, but none really filtered. Digging in, the problem was not on the regex on the log line, but on the pattern for the date. In the end, i made it work with something like this, that matched the timestamp logged from caddy:

[user@localhost]# cat /etc/fail2ban/filter.d/caddy.conf
[Definition]
failregex = ^.*"remote_ip":"<HOST>",.*?"status":(?:401|403|500),.*$
datepattern = ^.*"ts":{EPOCH}\.
[user@localhost]#

a tip - fail2ban has an utility - fail2ban-regex made exactly for this. The documentation could be friendlier, but all the info is there.

after trying this, fail2ban recognized the entries on the log file, and deployed the ban action… but not really. The server was still available to the offending origin.

…back to the drawing board.

Docker

My instance of caddy runs on a docker container. docker does all it’s networking magic on iptables.

I found a couple of issues on github related to docker, and they pointed me to the wiki, where it says that you should change the iptables chain

great! lets do that:

[user@localhost]# cat /etc/fail2ban/jail.d/caddy.local
[caddy]
enabled     = true
port        = http,https
filter      = caddy
logpath     = /var/logs/caddy_logs/access.log
maxretry    = 10
findtime = 1h
bantime = 24h
chain = DOCKER-USER

one restart later… and still no ban.

but I had a clue… there was no blocking rule on iptables -L

Fail2ban and firewalld

I checked and checked fail2ban configuration.

The default clearly stated that the ban-action was deployed by touching iptables rules:

[user@localhost]# grep -e "^banaction" /etc/fail2ban/jail.conf
banaction = iptables-multiport
banaction_allports = iptables-allports
(...)
[user@localhost]#

Since I had not configured a special action on the fail2ban side, this is where I expected the block to be. But it was not there.

On my system (Centos 9 Stream) firewalld is also running (actually, it’s the default!). I’m not conversant enough on iptables to mind on way over the other, but since it’s the default, I’m not keen on removing it, not without understanding the problem.

At this point, I was clear that fail2ban was blocking by way of firewalld. But I could not find out why.

It took me way more time than I care to admit to find it, but find it I did.

It does that by adding a file that modifies the default rules:

[user@localhost]# cat /etc/fail2ban/jail.d/00-firewalld.conf
# This file is part of the fail2ban-firewalld package to configure the use of
# the firewalld actions as the default actions.  You can remove this package
# (along with the empty fail2ban meta-package) if you do not use firewalld
[DEFAULT]
banaction = firewallcmd-rich-rules
banaction_allports = firewallcmd-rich-rules
[user@localhost]#

This file was right next to the configuration for the caddy ban, but I had not payed any attention to it!

two banactions later on the caddy jail definition, and the block was finally working:

[user@localhost]# cat /etc/fail2ban/jail.d/caddy.local
(...)
banaction = iptables-multiport
banaction_allports = iptables-allports
[user@localhost]#

sometimes it comes easy.

but not this time.