Install

Proxmox New Install – Network

How-to configure networking on a Proxmox host. Using VLAN for segregation, bonding for fault tolerance and some other advanced things.

Nalle

Nalle

— 10 min read
Proxmox New Install – Network
Photo by Kent Pilcher / Unsplash

Setting up secure and robust networking is key for any home lab. Also, networking is one of the things we want to tinker with – and brake from time to time.

Basics

You never want to manage the hypervisor host on the same network on which other clients and servers exist! If an attacker has compromised the network where a client resides, you don’t want them to have easy Layer 2 access to the management interface of your hypervisor.

⚠️ Set up a VLAN for the management Interface for increased security

If you change the IPs

  • do pvecm updatecerts --force to update Certificates
  • update your DHCP and DNS

Naming of NIC

Some examples for the most common naming patterns are:

  • eno1 — an on-board NIC
  • enp6S0 – a NIC on PCI bus 6
  • enp3s0f1 — a dual NIC on PCI bus 3, slot 0, function 1
ℹ️
Use a Linux Bond directly as a bridge port. This can be used to make the guest network fault-tolerant, increase the performance, or both.
A Linux Bond is also called NIC teaming or Link Aggregation.
💡
If your switch supports the LACP (IEEE 802.3ad) protocol, it's recommended using the bonding mode (802.3ad). Otherwise, you should always use active-backup mode.
– It's generally advised to use bonding mode and VLAN.
Corosync don't need a bond for network redundancy, can switch between networks, if one becomes unusable, without a bond.
💡
When to use a Gateway for a VLAN
No Gateway: Use this for an isolated VLAN where communication is local (like a storage VLAN).
With a Gateway: Use this when you need the VLAN to communicate with other networks, route traffic externally, or access other services through the firewall.

VLAN tagging

The beauty of the VLAN aware bridge is you can have many other VLAN configured on many other virtual machines, and each of them can communicate on the same VLAN interface.

The VLAN ID is part of the Layer 2 Ethernet frame. If the physical interface of the switch port is not configured correctly, VLAN traffic for the VLAN ID is discarded.

VMs using VLAN tags

When you create VMs, you can choose to tag the network traffic with a VLAN ID. This allows sending the VM traffic through the physical device VLAN interface to the rest of the physical network. 

auto vmbr0
iface vmbr0 inet manual
	bridge-ports ens1
	bridge-stp off
	bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4092
# VLAN Traffic


auto vmbr0.100
iface vmbr0.100 inet static
    address 10.100.110.43/24
    gateway 10.100.100.1
# Management VLAN

Corosync

  • Corosync do not need high speed /etc/network/interfaces.
  • But it is quintessential to have low latency, or it will break.
    Meaning, that you can't use a VPS or utilize a computer at a friend's place.

Setup in /etc/network/interfaces 

auto eth1
iface eth1 inet static
        address 10.10.1.151
        netmask 255.255.255.0

Setup in /etc/hosts 

127.0.0.1 localhost.localdomain localhost
192.0.2.151 one.example.com one pvelocalhost

# corosync network hosts 
10.10.1.151 one-corosync.example.com one-corosync
10.10.1.152 two-corosync.example.com two-corosync
10.10.1.153 three-corosync.example.com three-corosync

# For IPv6 capable hosts
[...]

Segregation of Traffic

Using separate NIC

    • eth0 Public Bridge for VM
    • eth1 Private Bridge for VM
    • eth3 Management *
    • eth4 Storage *
    • eth5 Corosync *
* This if only possible without HA. With HA, you need to use a bridge.

Using VLAN
bond0 (slaves enp3s0f0,enp3s0f1) bon1 (enp5s0f0,enp5s0f1)

    • vmbr1 Public Bridge for VM (bond0) VLAN 1
    • vmbr2 Private Bridge for VM (bond1) VLAN 100
      • vmbr2.10 Management
      • vmbr2.40 Storage
    • vmbr9 (eno1) Corosync

Bonding and VLAN

Consider setting up network bonding to ensure high availability and improved network performance. Edit your network configuration and use a managed switch.
Managed switches, with their basic functions, are cheap and easy to configure.
Enterprise equipment is costly and challenging to configure. LACP can be automated and provide redundancy; however, it is recommended to look for features that can be configured for speed by utilizing LAG groups.

LACP is like having multiple lanes to use in parallel. The top speed is the same, but throughput is about the double (using 2 NIC). And that is important running servers

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
#Purple - Corosync

auto enp3s0f0
iface enp3s0f0 inet manual
#Red - bond0

auto enp3s0f1
iface enp3s0f1 inet manual
#Green - bond0

auto bond0
iface bond0 inet manual
        bond-slaves enp3s0f0 enp3s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
#Bonded VLAN

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Bonded Bridge VLAN

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.10.10.43/24
        gateway 10.10.10.1
#MGMT vLAN

auto vmbr0.40
iface vmbr0.40 inet static
        address 10.10.40.43/24
#SRV vLAN

source /etc/network/interfaces.d/*

Replace the settings with your system information.

ℹ️
IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP): Creates aggregation groups that share the same speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group according to the 802.3ad specification.

Set up a Management VLAN

Set up a VLAN and remove your original IP Address and Gateway,

  • Add a new IP and GW to your Management VLAN
  • Change the IP in the /etc/hosts to the new one and ifreload -a to activate
  • Update Certificates, pvecm updatecerts --force
  • Add your new IP to your DHCP/DNS and activate
  • Create a Firewall Alias for the Management Network
  • Create an Anti Lockout rule
  • Block access to the Management VLAN from all other Networks/VLAN

Using DHCP Static Reservations

In the corporate world, DHCP is used to handle all addresses in the total environment. You need to set up static reservations on DHCP servers and update DNS servers. If you have to change the address space, all your servers will automatically do it instead of taking you days to do it. Corporate networks, use redundant setups of DNS and DHCP servers. For a home lab, you can do this.

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet dhcp
    bridge-ports enp5s0
    bridge-stp off
bridge-fd 0

What is DHCP snooping?

The DHCP snooping mechanism is a way to monitor DHCP messages across the network, and the environment in general. When a client device asks a DHCP server for an IP address, the reply should come from a source you choose. In a home lab environment, this is usually your local DHCP server.

DHCP snooping is useful, so turn it on.

It's important to make sure that only servers that have permission can be a DHCP server. If a bad server is added to the network, either accidentally or intentionally, DHCP snooping will stop it from disrupting the network and prevent attackers from harming the whole network.

Where is DHCP snooping used

  • It is imperative to safeguard Corporate Networks from unauthorized DHCP servers.
  • Home Labs has the same needs as Corporate Networks.
    Even more so, as we don't have an IT Security organization working 24/7.
  • Campus Networks, Educational institutions often have large and complex networks. Here, DHCP snooping can be used to segment different parts of the network, like administrative offices, classrooms, and student dorms. It helps to make sure the DHCP server comes from an authorized source provided by the educational institution IT department.
  • Public Wi-Fi Networks protect against rogue DHCP servers into the network on public Wi-Fi. This is important as Wi-Fi networks are often attacked using these types of attacks in internet cafés and public areas like malls and shops.
  • Data Centers are hosting multiple clients on shared infrastructure, it can be used to prevent one client’s DHCP server from serving IP addresses to another client’s network. Isolation helps make sure client security is protected.
  • ISPs use snooping to manage its customers, by to prevent malicious users from setting up unauthorized DHCP servers.

How DHCP snooping works

The network switch monitors DHCP traffic. It can find and block malicious DHCP messages if they come from untrusted sources. This is done by setting up a DHCP snooping binding table. This table records each connected device’s MAC address, IP address, lease time, and associated VLAN. It will let you know that only authorized DHCP servers are handling DHCP.

You can define a trusted port where DHCP messages can enter the network. All other ports are seen as not trusted. If DHCP offers come from ports that are not trusted, you can block them.

Terms used

  • Rogue DHCP Servers, a rogue DHCP server can mess up network operations by giving out the wrong IP addresses. By turning on DHCP snooping, these servers can't get into the network. The DHCP snooping binding database checks the authenticity of DHCP messages.
  • Enabling DHCP Snooping, most switches have a command similar to ip dhcp snooping command (familiar in Cisco IOS)
  • Trusted Ports are marked so for authorized DHCP servers.
  • Untrusted Ports all other ports are left marked as untrusted,
    and are usually the ports marked towards end-user devices.
  • Snooping for VLAN can be configured for all VLAN you are running on, or for a specific VLAN in the network.
  • DHCP Snooping Database On Cisco devices, it's possible to access the DHCP snooping database by executing the command show ip dhcp snooping. You can see current DHCP snooping database and it's details about leased IP addresses and their MAC addresses.
  • IP Source Guard provides additional security by ensuring that IP traffic is received from the legitimate source.
  • Dynamic ARP Inspection (DAI) prevents malicious ARP spoofing attacks, further fortifying network security.

Configure DHCP snooping

Since all switches do this differently, I only show a generic skeleton.

Connect to the switch and enter global configuration mode.

  • Turn on DHCP Snooping Globally
  • Allow DHCP Snooping on VLAN
  • Select Trusted Interfaces
  • Configurations that are optional
    • Rate Limiting the DHCP Traffic. Limiting the number of DHCP packets on untrusted ports can prevent DHCP flooding attacks.
  • Save the config

Example setup

auto lo
iface lo inet loopback

iface eno1 inet manual
#Purple

iface enp6s0 inet manual
#Grey 100M

auto enp3s0f0
iface enp3s0f0 inet manual
#Red

auto enp3s0f1
iface enp3s0f1 inet manual
#Green

auto bond0
iface bond0 inet manual
        bond-slaves enp3s0f0 enp3s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 192.0.2.47/24
        gateway 192.0.2.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094


auto vmbr0.10
iface vmbr0.10 inet static
        address 10.10.10.47/24
#MGMT network 

auto vmbr0.40
iface vmbr0.40 inet static
        address 10.10.40.47/24
#SRV Network

auto vmbr1
iface vmbr1 inet static
        address 10.10.90.47/24
        bridge-ports enp6s0
        bridge-stp off
        bridge-fd 0
#Cluster 100M

auto vmbr2
iface vmbr2 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#VLAN bonded

source /etc/network/interfaces.d/*

SDN Software Defined Networking in Proxmox

For network segmentation, you can use SDN. To facilitate communication between the cluster hosts while maintaining network segmentation through the utilization of the firewall. I found that the best way to accomplish this was to configure software defined networking in Proxmox to create VXLANs for the DMZ and LAN.

Adding VXLANs to the Lab Network

Configuring a software defined network in Proxmox consists of three steps for each network. I need to configure a Zone, a Vnet, and if I want to assign an IP range, a Subnet for each VXLAN.

Adding a VXLAN zone

To add a VXLAN zone, you select that zone type in the SDN interface.

Add: VXLAN Configuration

  • Add an ID: LAN
  • Peer IP address list: 10.100.100.45, 100.100.10.46, 10.100.100.47
  • MTU leave as Auto or set to: 1450
  • Ipam: leave at pve
💡
You need to set the MTU to a lower value than the standard 1500.
A VXLAN is a simulated layer 2 network on top of the existing network. An extra 50 bytes for the VXLAN header need to be added to each packet. Use MTU Auto or 1450 (1500-50).

Adding Vnets and Subnets

After creating the VXLAN zone, the next step is to add the associated Vnets and Subnets to each zone. In my lab, each zone will have a single Vnet and Subnet. Here is an example configuration of the LABnet and LAB subnet.

Datacenter→SDN→VNets hit [Create]

  • Name: LABnet
  • Alias:
  • Zone: from the pull down LAB
  • Tag: 10000

Datacenter→SDN→Subnets hit [Create]

  • Subnet: 10.100.200.0/24

Go back to the SDN overview and apply the configuration by hitting [Apply].

💡
In general, we need to provide the desired subnet in CIDR notation (e.g., 10.11.12.0/24). Defining the IP address for the gateway is also possible. In this example, the gateway will be placed on the IP address 10.11.12.1. Important is to activate the option SNAT. SNAT (Source Network Address Translation) is a technique to modify the source IP address of outgoing network traffic to appear as though it originates from a different IP address, which is usually the IP address of the router or firewall. This method is commonly employed to allow multiple devices on a private network to access external networks.

References

Proxmox [1] [2] [3]. VXLAN [4]

  1. Proxmox Daocumentation ↩︎

  2. - Networking ↩︎

  3. - SDN, wiki ↩︎

  4. VXLAN Wikipedia ↩︎

Read more