Proxmox networking - part 1

Networking is key to success in HomeLabbing and Proxmox has it all switches and bridges in many forms. This is some of them.

What are the networks needed with Proxmox.

A hypervisor allows you to run multiple machines, virtual machines (VMs), on one physical device, also known as the host. For us with home-labs, there could be many different VM's each for different tasks. Workstations, Linux VM's, windows VM's, others might be hosting NGINX reverse proxy and web servers, others run media servers and home automation and then we have the wonderful word of Docker and K8S. We will configure our Proxmox servers to use different networks and also VLAN's so that you can group related VM's onto their own sub-nets.

1. One network card is OK for a basic setup. With or without VLAN's.
2. VLANs make labs coming to a new level. You may want to segregate the lab from the house net or divide the net into smaller part for better security and/or performance. You need Level 2 or Level 3 switches and basic knowledge of networking.
3. 3 or 5 or even more nodes are needed for High Availability and Ceph set-ups. For High Availability systems you need (3 nodes) minimum of 3 and preferably 6 network cards to ensure VM's can migrate freely between nodes, one single or dual 10G card is recommended. ‌‌These are more relevant in production environments but you might want to secure your reverse proxy, DNS or DHCP, Home Assistant or some other critical VM. In commercial set-ups you install 3 or 5 identical servers with identical ZFS-pools of correct size and identical network configurations. A cluster of 5 is better in production and use of single or dual 10G cards is recommended.
4. 4 or more if your VM's need dedicated network cards. You also need to have CPU's and MB with IOMMU (Vt-d) to pass the appliance to the VM. Many modern enterprise servers do have 4 network cards as standard. You might want to add a single or dual 10G card for critical loads.

The hardware needed

File systems

To start with the disks you should always set up 1 - n ZFS-pool's on your Proxmox for stability, speed and to use for snapshots or replications of your VM's.

An alternative or add-on is to use ZFS pool's and VDEV's on SAN/NAS for VM/CT storage. On any NAS a 10G card is highly recommended.

VM's can live migrate - containers can't.

Backups

There is no more important thing than backups except the rule of 3-2-1.

For all kinds of backups the best and easiest is to use a Proxmox Backup Servers. Use any old PC as a base and add disks and network cards.

RAID is no Backup - not even ZFSraid-2 or ZFSraid-3

Shared storage

Many times we use shared storage for ISO and template-files. Shared storage for VM disks make sens if you want ho have swift migration. Remember redundancy and back-ups.‌‌ TrueNAS Core is a fantastic solution due to the share power of it. But in a home-lab if you like Linux you could run TrueNAS Scale, even in a VM on Proxmox (isn't good practice to run hypervisors on hypervisors). For shared storage I recommend 10G networks and a Lightweight NAS with NFS and with or without GUI.

For HomeLabs and especially Proxmox based ones - a Lightweight NAS is what it takes and for more experienced users skip the GUI part.

Network switches

You should use smart/manageable switches Level 2, Level 2+ or Level 3 (preferable, can do routing in the switch). Older enterprise grade switches are inexpensive but power hungry solutions. Today you do find smart switches with VLAN's for 40 - 100€.

Migration in HA-systems can be controlled and maintained on 1 or 2.5G networks with VLAN separation and prioritization for max speed. 10G is better.

Small 10G switches are better and also affordable, they are needed between nodes and shared storage.

The Role of VLAN's

The basic network shall be built with several VLAN's for speed and security. Setting up VLAN infrastructure is not too complicated. But it's very flexible but, it can be strange the first time. There is different ways to use the vids so please read the documentation of your network gear, Proxmox do bridge-vids 2-4094 as standard.

auto vmbr0
iface vmbr0 inet static
        bridge-ports ens1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4092

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.10.10.45/24
        gateway 10.10.10.1
#MGMT Network

Firewall and routing

We have have two ways to go HW or SW but we need it. Servers are always the prime target for hackers. Try to avoid exposing the server to the web, only expose services trough reverse proxies or tunnels.

I use a Level 3 switch and sw routers like SDN.

Hardware solutions

HW solutions to recommend is Netgate pfSense Security Gateway Appliances 1100 or 2100 Link. Also Sophos, Ubiquity and Microtic have extremely nice gear.

Software solutions

There is a multitude of solutions out there, some are already outdated and dangerous to use. I just picked my current faworites to use as examples.

As I promote DIY and Opensource: Sophos Firewall Home Edition, pfSense, OPNsense or Untangle (now Arista NG, from 50$/year) or to do your basic network management and firewall tasks. I run them as VM's.

Note Sophos needs registrations and to download the Firewall OS for KVM zip file. Then extract the qcow2 files and import to a VM. Recommended 4 cores and 4G ram and the 2 downloaded disks 16 G+80 G.
Sophos is hungry for resources!

pfSense and OPNsense is very lightweight and runs nicely in a VM. Recommended 2 cores 1-2 G ram and 8-16 G disk for a basic firewall.

pfSense documentation is second to non!

My current configuration

1. One ISP router (junk but mandatory). WiFi user and guest
2. Sophos or pfSense (can't decide witch to use) for the main internal network‌‌ (more on this setup later) and an OPNsense for a test LAN
3. Redundant Pi-stack for the family network
4. Redundant Ubuntu based Bind9 DNS for the lab network
5. 1 Level 3, 6 Level 2 switches
6. Mesh WiFi user and guest
7. 3 Proxmox Backup Servers ( 1 off site)
8. 1 TrueNAS Core‌‌ as Cluster storage for data: ISOs, VMdisks, Files, Documents (to be replaced by a Lightweight NAS or maybe a Ultra Lihgthweight NAS.
9. 1 Proxmox cluster of 5
10. 4 Stand-alone Pve's

For testing and special labs

• 2 stand alone Proxmox servers
• 1 TrueNAS Scale for testing and some old stuff
• 2 Lightweight NAS server

Monitoring and alarms

Monitoring is a big must and sorry to say the Proxmox GUI is not good at all.

It's also important to have alarms setup to send messages to your phone and stats and warnings to the admin e-mail.

• Zabbix is my favorite and the best IMHO
• Grafana is beautiful, used with Influx DB and/or Prometheus
• Slack or Telegram are easy to use
• Uptime Kuma is fun and Hompage is more fun
• Homepage is also showing status info

Desktop setup

1. Linux DT (Arch or POP-OS) 2 x 32" screens
2. MacBook Pro with 2 x 27" screens
3. iPad
4. iPhone

I like to have many views of the cluster and it's running processes at any time.

GUI is easy to have a overview but terminals are the main tool to work with. I also do a number of project at the same time and need to follow many things in real time.‌‌ Do you need this number of screens - no. I just happen to have plenty of stuff to use. ‌‌Dual screen is mandatory, IMHO, been using dual screens from the 90's. The productivity is so much better and the ergonomics too.

Ergonomics is one of the things you should pay attention too. In the old day's when I sat in front of the screen 16 h a day - I learned it the hard way.