Proxmox networks - Install

This is how to install the networks, not why. (Why is described in posts Proxmox networking 1-3). In Proxmox we have physical network cards , a level 2 software switch, VLAN aware bridges, bridges and bonded interfaces.
Out of the box only vmbr0 is configured.

Network Cards

On enterprise servers you have 2 or 4 Ethernet ports as standard and the reserved machine management port. Will need as a minimum

• 1 for management and general vlan use
• one for the cluster network traffic
• Nice if we also have one for NFS/iSCSI traffic - 10G
• install and use the ethtool if needed to adjust the NIC
• For HA add a second NIC and configure the pairs
• Cluster management do not need speed but needs low latency

Plan your VLAN's

Special VLANS are 1 and 4093, 4094. Sometimes you can only use 1024 VLAN's. To block all unwanted traffic on empty rj-45 jacks assign them to a special port and block them in the /etc/network/interfaces file. We prefer to use tagged Ethernet frames as the IEEE 802.1Q header, or the VLAN tag.

Level 3 switches are fast in routing but routers can do much more. Meaning a Level 3 switch is good to have in larger and more complex labs, they usually also do Link Aggregation Control Protocol (LACP) for Ethernet, defined in IEEE 802.1AX or the previous IEEE 802.3ad .

A single VLAN can contain multiple IP subnets. IEEE 802.1ad extends the number of VLANs supported by adding support for multiple, nested VLAN tags. IEEE 802.1aq (Shortest Path Bridging) expands the VLAN limit to 16 million.

For High Availability you need to have LAG groups assigned to eliminate the 3 single place of failure presented by a LAN Cable.

Setup

What are all these VLAN's we need. We will consider we have a 4 port NIC:

• Bond 3 ports together and make it VLAN aware vmbr2 (any number that is free) for all VLAN's. The 4th port is for the cluster management.
• VLAN 10 is the management interfaces for all hypervisor servers on the LAN
• VLAN 40 is the shared storage NFS (or iSCSI if you like), make it fast 10-100G
• VLAN 100 Use the 4th as vmbr0 for cluster network on a separate subnet and VLAN 100 172.nnn.nnn.nnn/24. Cluster management is fine with a 100M NIC but it's crucial to have low latency. No other traffic on this interface.
• Use the SDN for labs
• Edit the /etc/network/interfaces set your new IP and VLAN
• Edit the /etc/hosts to reflect your IP settings
• Update your DNS

auto bond0
iface bond0 inet manual
        bond-slaves enp2s0f0 enp2s0f1 enp2s0f2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        
auto vmbr0
iface vmbr0 inet static
        bridge-ports ens1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4092

auto vmbr2.10
iface vmbr2.10 inet static
        address 10.10.10.45/24
        gateway 10.10.10.1
#MGMT Network

auto vmbr2.40
iface vmbr2.40 inet static
        address 10.10.40.45/24
        gateway 10.10.40.1
#SRV Network

127.0.0.1 localhost.localdomain localhost
10.10.10.45 sandbox.example.com sandbox

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

An other way is to make new sub NIC's like enp2s0f0.10 and assign it to a vmbr.

Networking - reload error in Proxmox

You may encounter an error while updating a nodes network settings:

eth0 : error: eth0: cmd ‘/sbin/dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0’ failed: returned 1
TASK ERROR: command ‘ifreload -a’ failed: exit code 1

To fix it, go to /etc/network/interfaces, edit the file and correct your mistake.

Firewall

You need to set up all the Firewalls to reflect the network. Proxmox Servers have 3

• Datacenter Firewall (it's off at install to not lock you out, configure and set on)
• Host Firewall
• VM Firewall

References

VLAN ^[1]