Proxmox security – fail2ban

Fail2Ban ban hosts that cause multiple authentication errors.
It's done by scanning log files like /var/log/auth.log and bans IP addresses conducting too many failed login attempts. It does this by updating system firewall rules to reject new connections from those IP addresses (IPv4 and IPv6), for a configurable amount of time. Set up services to use only 2FA, or public/private authentication mechanisms if you really want to protect your services.

All your Proxmox nodes and the servers running on them should have Fail2ban and use the internal Firewall. All user that login to any of them shall utilize SSH keys, access Tokens or 2FA to authenticate. Don't allow root login.

Networking

A separate cluster network is recommended, especially if you have shared storage.
It's not the speed that is important for this network but the latency.

VLAN

VLANs are to be used for the Management, Cluster, and Storage Networks as for all other groups of services. Segregation of duty is still indispensable.

Proxmox Firewall

Yes, it is to be active on all devices and control access to and from devices. Link

Fail2ban setup

Fail2ban shall be installed on all nodes and server to make brute force attacks harder, this will protect against internal and external attacks.

Installation

apt update && apt install fail2ban -y

Configuration files for Fail2ban

We need to edit one file and add 3 file pairs (jail and filter):

1. The main configuration /etc/fail2ban/jail.local
2. The Generic Pam configuration
   1. /etc/fail2ban/jail.d/pam-generic.conf
   2. /etc/fail2ban/filter.d/pam-generic.conf
3. The SSH configuration
   1. /etc/fail2ban/jail.d/sshd.conf
   2. /etc/fail2ban/filter.d/sshd.conf
4. The Proxmox configuration
   1. /etc/fail2ban/jail.d/proxmox.conf
   2. /etc/fail2ban/filter.d/proxmox.conf

The main configuration – jail.conf

First, we create the main configuration file by copy the generic jail configuration.

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
cp

Edit the file and add your basics

...
[INCLUDES]
before = paths-debian.conf
[DEFAULT]
backend = auto
...
banaction = iptables-multiport
banaction_allports = iptables-allports
...

Create the file pair for PAM authentication

The jail configuration

nano /etc/fail2ban/jail.d/pam-generic.conf

# Jail.conf for PAM authentication

[pam-generic]
enabled = true
backend = systemd
banaction = iptables
findtime = 14d
bantime = 30d
maxretry = 1
ignoreip = 127.0.0.1/8 ::1

The filter configuration

nano /etc/fail2ban/filter.d/pam-generic.conf

# Filter for PAM authentication

[INCLUDES]

before = common.conf

[Definition]

# if you want to catch only login errors from specific daemons, use something like
#_ttys_re=(?:ssh|pure-ftpd|ftp)
#
# Default: catch all failed logins
_ttys_re=\S*

__pam_re=\(?%(__pam_auth)s(?:\(\S+\))?\)?:?
_daemon = \S+

prefregex = ^%(__prefix_line)s%(__pam_re)s\s+authentication failure;(?:\s+(?:(?:logname|e?uid)=\S*)){0,3} tty=%(_ttys_re)s <F-CONTENT>.+</F-CONTENT>$

failregex = ^ruser=<F-ALT_USER>(?:\S*|.*?)</F-ALT_USER> rhost=<HOST>(?:\s+user=<F-USER>(?:\S*|.*?)</F-USER>)?\s*$

ignoreregex =

datepattern = {^LN-BEG}

Create the file pair for OpenSSH

If you want to protect OpenSSH from being brute forced by password authentication, then get public key authentication working before disabling PasswordAuthentication in sshd_config.

The SSH jail configuration

nano /etc/fail2ban/jail.d/sshd.conf

# Jail.conf for OpenSSH

[sshd]
enabled = true
filter = sshd
banaction = iptables
backend = systemd
maxretry = 1
bantime = 30d
findtime = 14d
ignoreip = 127.0.0.1/8 ::1

The SSH filter configuration

nano /etc/fail2ban/filter.d/sshd.conf

# Filter to OpenSSH

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from common.local
before = common.conf

[DEFAULT]

_daemon = sshd

# optional prefix (logged from several ssh versions) like "error: ", "error: PAM: " or "fatal: "
__pref = (?:(?:error|fatal): (?:PAM: )?)?
# optional suffix (logged from several ssh versions) like " [preauth]"
#__suff = (?: port \d+)?(?: \[preauth\])?\s*
__suff = (?: (?:port \d+|on \S+|\[preauth\])){0,3}\s*
__on_port_opt = (?: (?:port \d+|on \S+)){0,2}
# close by authenticating user:
__authng_user = (?: (?:invalid|authenticating) user <F-USER>\S+|.*?</F-USER>)?

# for all possible (also future) forms of "no matching (cipher|mac|MAC|compression method|key exchange method|host key type) found",
# see ssherr.c for all possible SSH_ERR_..._ALG_MATCH errors.
__alg_match = (?:(?:\w+ (?!found\b)){0,2}\w+)

# PAM authentication mechanism, can be overridden, e. g. `filter = sshd[__pam_auth='pam_ldap']`:
__pam_auth = pam_[a-z]+

[Definition]

prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID>%(__pref)s<F-CONTENT>.+</F-CONTENT>$

cmnfailre = ^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \S+)?%(__suff)s$
            ^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>%(__suff)s$
            <cmnfailre-failed-pub-<publickey>>
            ^Failed <cmnfailed> for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
            ^<F-USER>ROOT</F-USER> LOGIN REFUSED FROM <HOST>
            ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>%(__suff)s$
            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because not listed in AllowUsers%(__suff)s$
            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because listed in DenyUsers%(__suff)s$
            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because not in any group%(__suff)s$
            ^refused connect from \S+ \(<HOST>\)
            ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$
            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because a group is listed in DenyGroups%(__suff)s$
            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups%(__suff)s$
            ^<F-NOFAIL>%(__pam_auth)s\(sshd:auth\):\s+authentication failure;</F-NOFAIL>(?:\s+(?:(?:logname|e?uid|tty)=\S*)){0,4}\s+ruser=<F-ALT_USER>\S*</F-ALT_USER>\s+rhost=<HOST>(?:\s+user=<F-USER>\S*</F-USER>)?%(__suff)s$
            ^maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?%(__suff)s$
            ^User <F-USER>\S+|.*?</F-USER> not allowed because account is locked%(__suff)s
            ^<F-MLFFORGET>Disconnecting</F-MLFFORGET>(?: from)?(?: (?:invalid|authenticating)) user <F-USER>\S+</F-USER> <HOST>%(__on_port_opt)s:\s*Change of username or service not allowed:\s*.*\[preauth\]\s*$
            ^Disconnecting: Too many authentication failures(?: for <F-USER>\S+|.*?</F-USER>)?%(__suff)s$
            ^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>%(__on_port_opt)s:\s*11:
            <mdre-<mode>-other>
            ^<F-MLFFORGET><F-MLFGAINED>Accepted \w+</F-MLFGAINED></F-MLFFORGET> for <F-USER>\S+</F-USER> from <HOST>(?:\s|$)

cmnfailed-any = \S+
cmnfailed-ignore = \b(?!publickey)\S+
cmnfailed-invalid = <cmnfailed-ignore>
cmnfailed-nofail = (?:<F-NOFAIL>publickey</F-NOFAIL>|\S+)
cmnfailed = <cmnfailed-<publickey>>

mdre-normal =
# used to differentiate "connection closed" with and without `[preauth]` (fail/nofail cases in ddos mode)
mdre-normal-other = ^<F-NOFAIL><F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)%(__authng_user)s <HOST>(?:%(__suff)s|\s*)$

mdre-ddos = ^Did not receive identification string from <HOST>
            ^kex_exchange_identification: (?:read: )?(?:[Cc]lient sent invalid protocol identifier|[Cc]onnection (?:closed by remote host|reset by peer))
            ^Bad protocol version identification '.*' from <HOST>
            ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:
            ^Read from socket failed: Connection <F-MLFFORGET>reset</F-MLFFORGET> by peer
            ^banner exchange: Connection from <HOST><__on_port_opt>: invalid format
# same as mdre-normal-other, but as failure (without <F-NOFAIL> with [preauth] and with <F-NOFAIL> on no preauth phase as helper to identify address):
mdre-ddos-other = ^<F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET> (?:by|from)%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$
                  ^<F-NOFAIL><F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)%(__authng_user)s <HOST>(?:%(__on_port_opt)s|\s*)$

mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No(?: supported)? authentication methods available
            ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching <__alg_match> found.
            ^Unable to negotiate a <__alg_match>
            ^no matching <__alg_match> found:
# part of mdre-ddos-other, but user name is supplied (invalid/authenticating) on [preauth] phase only:
mdre-extra-other = ^<F-MLFFORGET>Disconnected</F-MLFFORGET>(?: from)?(?: (?:invalid|authenticating)) user <F-USER>\S+|.*?</F-USER> <HOST>%(__on_port_opt)s \[preauth\]\s*$

mdre-aggressive = %(mdre-ddos)s
                  %(mdre-extra)s
# mdre-extra-other is fully included within mdre-ddos-other:
mdre-aggressive-other = %(mdre-ddos-other)s

# Parameter "publickey": nofail (default), invalid, any, ignore
publickey = nofail
# consider failed publickey for invalid users only:
cmnfailre-failed-pub-invalid = ^Failed publickey for invalid user <F-USER>(?P<cond_user>\S+)|(?:(?! from ).)*?</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
# consider failed publickey for valid users too (don't need RE, see cmnfailed):
cmnfailre-failed-pub-any =
# same as invalid, but consider failed publickey for valid users too, just as no failure (helper to get IP and user-name only, see cmnfailed):
cmnfailre-failed-pub-nofail = <cmnfailre-failed-pub-invalid>
# don't consider failed publickey as failures (don't need RE, see cmnfailed):
cmnfailre-failed-pub-ignore =

cfooterre = ^<F-NOFAIL>Connection from</F-NOFAIL> <HOST>

failregex = %(cmnfailre)s
            <mdre-<mode>>
            %(cfooterre)s

# Parameter "mode": normal (default), ddos, extra or aggressive (combines all)
# Usage example (for jail.local):
#   [sshd]
#   mode = extra
#   # or another jail (rewrite filter parameters of jail):
#   [sshd-aggressive]
#   filter = sshd[mode=aggressive]
#
mode = normal

#filter = sshd[mode=aggressive]

ignoreregex =

maxlines = 1

journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd

Create the file pair for Proxmox

The jail configuration

nano /etc/fail2ban/jail.d/proxmox.conf

# Jail.conf for Proxmox

[proxmox]
enabled = true
filter = proxmox
backend = systemd
banaction = iptables
maxretry = 1
findtime = 14d
bantime = 30d
ignoreip = 127.0.0.1/8 ::1

The filter file configuration

nano /etc/fail2ban/filter.d/proxmox.conf

# Filter to the Proxmox Jail

[Definition]
failregex = pvedaemon\[.*authentication failure; rhost=<HOST> user=.* msg=.*
ignoreregex =

Restart Service

To enable the new config, use: systemctl restart fail2ban to and arm fail2ban for protection of the Proxmox VE API.

Test for success

Make a failed attempt to login by PAM. Then issue the command:

fail2ban-regex systemd-journal /etc/fail2ban/filter.d/proxmox.conf

You should now have at least a Failregex: 1 total at the top of the Results section and 1 matched at the bottom

Check for jailed attempts

• Proxmox fail2ban-client status proxmox
• SSH fail2ban-client status sshd

References

fail2ban scans log files and journals (using specified regular expressions also known as filter-rules) and executes configured actions to ban failures having too many attempts (matched specified filter-rules). It does this e. g. by updating system firewall rules to reject new connections from those IP addresses, for a configurable amount of time. But you can write resp. configure your own action to ban something other than host/IP, like user or e-mail. Homepage ^[1], Wiki pages ^[2]
Proxmox Documentation ^[3], Wiki pages ^[4]