Proxmox Firewalls and VLAN's

The firewall is the key to successful security, but not the only thing. You need to have many things set right to have a chance in today's hostile environment.
Outh there are hundred of tousends of criminals, government sponsored hackers (spices and saboteurs) and normal hackers and they see any server on the internet as a prime target. And they will attack like a starving lion attacks a lame antelope. But they cant afford to spend to much time on a seconddary target. This is why Proxmox have Firewalls implemented on 3 levels: Datacenter, Host and VM - use them all!

👉 There is no such thing as to many layers of firewalls!

Set up the Firewall

We will use the internal firewall on all machines and most VM's too.

Ports used by Proxmox

• Web interface: PVE8006 PBS 8007 (TCP, HTTP/1.1 over TLS)
• VNC Web console: 5900-5999 (TCP, WebSocket)
• SPICE proxy: 3128 (TCP)
• SSHD (used for cluster actions): 22 (TCP)
• rpcbind: 111 (UDP)
• sendmail: 25 (TCP, outgoing)
• corosync cluster traffic: 5405-5412 UDP
• live migration (VM memory and local-disk data): 60000-60050 (TCP)

Datacenter level

This is where the master switch for all firewall is located. It's always off at initial boot up, so we do not lock our self out. You need to make an anti lock-out rule for the GUI on port 8006. Other ports may or may not need to be setup depending on your setup, see the Proxmox documentation on Firewalls.

Node level

Some nodes have special use cases and also special users and or groups assigned to them as users that need to be locked out from a network or added some access. You can also set the log levels individually per rule or Security Group (see below).

VM level

Some VMs have special users and or groups assigned to them as users that need to be locked out from the local networks. You can also set the log levels individually per rule or Security Group (see below).

Firewall Settings

For future you to understand the Firewall Rules, these will help to make them easier to debug or change. These are very convenient and powerful tools. I prefer the GUI, but you can edit the Firewall by sudo nano /etc/pve/firewall/cluster.fw

• The cluster-wide firewall configuration: /etc/pve/firewall/cluster.fw
• Host related configuration : /etc/pve/nodes/<nodename>/host.fw
• VM firewall configuration: /etc/pve/firewall/<VMID>.fw

IP Sets

IP sets can be used to define groups of networks and hosts.

IP sets can be used to define groups of networks and hosts. You can refer to them with +name in the firewall rules source and dest properties.

The following example allows HTTP traffic from the management IP set.

IN HTTP(ACCEPT) -source +management

Standard IP set management
This IP set applies only to host firewalls (not VM firewalls). Those IPs are allowed to do normal management tasks (Proxmox VE GUI, VNC, SPICE, SSH).

The local cluster network is automatically added to this IP set (alias cluster_network), to enable inter-host cluster communication. (multicast, SSH, …)

IP Alias

To call things with friendly names. IP Aliases allow you to associate IP addresses of networks with a name. You can then refer to those names in your rules. I use things like GW (for web access or not), LAN, SRV, MGMT, guest, cluster

Security Group

A security group is a collection of rules, defined at cluster level, which can be used in all VMs' rules. For example, you can define a group named “webservers” with rules to open the HTTP and HTTPS ports.

Now you can call this group to a VM’s firewall. Makes Firewall Rules easier to understand.

Options

This is used to set cluster-wide firewall options.

Log levels

There are 9 levels of logging: no log, emergency 0, alert 1, critical 2, error 3, warning 4, notice 5, info 6 and debug 7.

Use them well but be vice. Levels 6 and 7 are for finding issues but creates a storm of output. Level 5 means you need to look into it and anything lover needs to be addressed now - not tomorrow - the shit has hit the fan already.

Practical advice

It's impossible to give detailed advice how to use them due to the complexity of the setup of a level 1 hypervisor like Proxmox. You need to understand the basics of networking to be able to protect your internal setup and a little more in dept to protect any exposed services.

Internal Firewalls protects against compromised VM's and guests accessing the LAN by mistake or on purpose.

Don't lock yourself out

You need to set the port 8006 open (but securely) in Datacenter Firewall and the activate the firewall. By activating the Datacenter Firewall also the Host and VM Firewalls will be functional.

Use VLAN's

Segregating the Network is one small step to secure intrusion from internal or compromised VM's. The most gain is in less cluttered traffic not security, but it might keep kids away from experimenting with your stuff. Two examples:

• add a Linux Bridge and set the Bridge ports as ens1.10 for VLAN 100
• add a Linux VLAN and name it vmbr0.10 for VLAN 100 on vmbr0

You can use only one interface or one bonded interface using VLAN's even if you run a Firewall / Router in a VM on your Proxmox box. Assign WAN to on VLAN and LAN and OPT's to other VLAN's.

Configure the Notifications

You should set up Gotify to send error and warning alarms to your phone.

Management Interface on a VLAN ✅

Delete the settings on vmbro and add the IP and GW to vmbr2.100 or eno1.100. This is to make the management interface more secure from teenagers or uninvited guests trying to access the server.

• Create a bond0 device of two of your NIC ports and set the Bond Mode to what your switch can do and what is best for your networking needs. Bond0 shall also have Autostart and VLAN aware checked ☑. Another VLAN on bond0 could be one of the cluster interfaces.
• E.g. bond0 Slaves: enp2s0f0 enp6s0
  • Mode LACP (802.3ad) Hash policy: layer2+3
  • create vmbr2 with Bridge ports: bond0 Comment: VLAN bond bridge
• Make a Linux VLAN vmbr2.100
  • IPv4/CIDR: 10.100.100.100/24 GW: 10.100.100.1 Comment: MGMT VLAN
• Create a second VLAN for Servers and VM's from the rest of the NIC ports.
  • Bond1 Slaves enp2s0f1 enp2s0f2 enp2s0f3 enp6s1 enp6s2 enp6s3
  • Mode LACP (802.3ad) Hash policy: layer2+3
  • create vmbr1 with Bridge ports: bond1 Comment: SRV and VM VLAN

Block WLAN access ✅

WLAN are notorious easy to break in to and should be left to the IoT and other crap. If you need to have WLAN access, get devices with VLAN's to segregate IoT and Management from other WLAN traffic.

Use token and keys ✅

Proxmox has made token creation super easy for a reason - you should use them. Also SSH Keys are easy to use and should be mandatory, ed225519 should be used.

Passwords 🙅

Passwords are the Security's worst nightmare. Where you need one, use a super complicated one and make it a long one. There are a lot of password managers out there to make it easy to use 20 long passwords. Database passwords can be tricky with special characters, read the documentation for your database.

🆗 The correct way is to always use SSH keys and Tokens.
🤝 Tokens can easily be created in Proxmox.

Use Two-Factor Authentication whenever it's possible!

For a general overview of Two-Factor authentication (2FA) in PVE see the wiki User Management. A complete step-by-step guide to set up OATH OTP 2FA,
see the documentation Two-Factor Authentication. And a demonstration setup for Yubico OTP see the YubiKey article.

Jump Server set up, see this blog post and also the series of Hardening Servers.

Ngingx Proxy Manager

NPM Expose web services on your network · Free SSL with Let's Encrypt  · Designed with security in mind  · Perfect for home networks. Expose your private network Web services and get connected anywhere.

GUI based on Tabler, the interface is a pleasure to use. Free SSL by built in Let’s Encrypt support allows you to secure your Web services. The certificates even renew themselves.

Built as a Docker Image, Nginx Proxy Manager only requires a database.

Multiple Users can be configured to either view or manage their own hosts. Full access permissions are available.

Authelia

Authelia is an open-source full-featured authentication server providing a login portal and treating authentication requests in cooperation with NGINX. Authelia is build to Protect your applications with Single Sign-On and 2 Factor.

• Authelia offers a login portal to allow your users to login once and access everything.
• Authelia supports time-based one-time passwords generated by Google Authenticator.
• Per-resource authorizations. Authelia let you define fine-grained authorizations for every resources served by your various subdomains.
• Authelia regulates the number of login attempts made by a user to avoid brute force attacks.

References

Proxmox Firewall ^[1] Notifications ^[2] Networking ^[3]