Remove a Node

Removing a node

Production clusters are usually stable for the lifetime, but not always. Testing clusters and home labs have servers coming and going all the time.
It could be due to various reasons, including hardware failure, node upgrade or replacement, performance issues, or network connectivity problems.

Don't just reformat the machine before you follow these steps to make sure you have successfully removed the node, or you may have tasks you can't delete.

In this case, I did replace one old server to use it for LXD testing.
A new server will be installed to do the work of pve-55.

Login on another node

You need to be running from another node than the one we are to remove. 😆

Prepare the Node to leave the Cluster

Replication

Replication jobs need to be finished, and all scheduled jobs should be stopped on the node to be removed. Remove all replication jobs.

Migration

Backup to PBS and then migrate all VM you like to keep in the clusters.

Evacuate Storage

Before removing a node from the Proxmox cluster, it is essential to evacuate the storage associated with that node. This step ensures that all data stored on the node is safely transferred to other available storage devices within the cluster.

Leave the Cluster

pvecm delnode pve-55

Power off and remove the boot drive from the deleted node. You do not like it to reboot into the cluster.

Check for Success

pvecm nodes

Or check the full status of the current Cluster.

pvecm status

Remove node from the GUI

Sometimes the node is still visible in the GUI. Remove it from the GUI.
Log in to an active node, for example (I usually go to the main one).

All nodes have its own directory, the directory /etc/pve/nodes/ is synced between all cluster nodes. The removed node is still visible in GUI until the node directory do not exist in the directory /etc/pve/nodes/.

Remove the ghost node from the GUI

Delete the node directory rm -rf /etc/pve/nodes/pve-55 (use your node name).

Warning ⚠️

⚠️ Don’t delete nodes unless you understand the process in dept.
The node will be in a state where it's not easy to re-add it to any cluster.
⚠️ Note that the containers on the node do not start up automatically, even though their config file says they are supposed to.
🆗 Reinstall the OS on the node.

Rejoin the Node to the Cluster

After the Node has been removed from the Cluster, make sure that you power it off. Do not power it on with network access.
If you’d like to add the Node back to the Cluster, please reinstall the Proxmox system and rejoin the Node to the Cluster.

Automation Script

You can use my script on the node to be removed

wget https://github.com/nallej/MyJourney/raw/main/StopDelCorosync.sh

Ghost of old node can be removed from /etc/pve/corosync.conf

References

Proxmox Wiki ^[1] Documentation ^[2]