XCP New Generation

The acronym XCP-ng denotes Xen Cloud Platform — the next generation.
It is the successor to XCP, which was originally created as an open source version of Citrix XenServer in 2010. XenServer was at that time closed source, with XCP being the open source version containing a subset of features.

As XenServer was open sourced in 2013, the XCP project was halted. Several years later, when Citrix stopped delivering XenServer for free and via open source, the project was revived as XCP-ng.

XEN (pronounced /ˈzɛn/) is a free and open-source type-1 hypervisor, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. It was originally developed by the University of Cambridge Computer Laboratory and is now being developed by the Linux Foundation with support from Intel, Citrix, Arm Ltd, Huawei, AWS, Alibaba Cloud, AMD, Bitdefender and epam. Version 1.0 released October 2, 2003.

The Xen Project community develops and maintains Xen Project as free and open-source software, subject to the requirements of the GNU General Public License (GPL), version 2. Xen Project is currently available for the IA-32, x86-64 and ARM instruction sets.

Scalability in AMD64 Architectures

Link to the full 4.18 support-matrix. The hypervisor can:

• scale up to 4,095 physical CPUs with 16Tb of RAM.
  • hosts with up to 12 TiB (Hosts with more memory are not security supported.)
• Using Para Virtualization (PV),
  • it supports a maximum of 512 vCPU with > 1 TB RAM per guest.
• Using Hardware Virtualization (HVM),
  • it supports a maximum of 128 vCPU with 1 TB RAM per guest.
• Guest Memory Limits up to 8 TiB (up to 16 TiB are supported, but not security supported.)

Moving from VMware to XCP to Proxmox

At work, I was exposed to XEN and VMware over the years. Moving my home lab into virtualization, my first choice was VMware, didn't really like ESXI.
Then, tried XCP and liked it a lot. Used it for some years. In December 2017, Citrix announced that they would remove important features of the Xen Server Free edition and make them only available on paid tiers.
A natural response happened on March 31, 2018, XCP-ng was announced.

After a period of running services on XCP-ng, others on TrueNAS, and others on Proxmox, my home lab switched to Proxmox during 2019. I'm really happy with the move to Proxmox.
I have worked on XOA & XCP-ng for some weeks, and I like it.

Virtual XEN

There is a guide in the documentation about running XCP-ng in a VM.
I have tested running XCP-ng in a VM, briefly, and yes it works fine for testing.

# Install the QGA
yum install qemu-guest-agent --enablerepo=base,updates
systemctl enable --now qemu-guest-agent

# Cleanup 
yum clean all
rm -rf /var/cache/yum
yum update

The architecture is different

XCP-ng is based on Citrix Hypervisor 8.2 with CentOS in dom0: 7.5. It's a Xen Project: Xen 4.13.1 + patches, Linux Kernel 4.19 + patches. New experimental storage drivers: ZFS, glusterfs and cephfs. XCP-ng can use:

• Up to 5 TB of RAM
• Up to 16 physical NICs
• Up to 288 logical processors per host, depending on CPU characteristics

See documentation if you have a Dell server or your CPU is an AMD Ryzen.

Proxmox is a Debian OS and Linux KVM hypervisor and running the latest Linux Kernel. On top there is the Application layer with all the tools from Proxmox, even a dedicated Kernel. QEMU can present to the guest operating system paravirtualized devices, where the guest OS recognizes it is running inside QEMU and cooperates with the hypervisor. Proxmox support for ZFS is superior and brings it many benefits. As normal, special care if your GPU is Nvidia.

Similarities On the other hand the specs are very similar, both have things like clustering, OVS, SDN. Proxmox support for ZFS is superior and brings it many benefits. Neither ZFS nor Ceph are compatible with a hardware RAID controller.

Proxmox

Proxmox is a complete package, including the GUI for node or cluster. It offers a fully integrated backup solution, Proxmox Backup Server, supporting different backup modes for VMs and containers, scheduled backups, and a live-restore option. It allows setting bandwidth limits for backups, specifying file exclusions, and defining global configurations. The SDN implementation is good.

Additionally, Proxmox provides backup compression, encryption, and retention options. Unique features like protected backups, adding notes to backups, and hook scripts for custom actions are also supported.

VMware vs. XCP-ng

Where XCP used to be oriented towards the paid versions and the enterprise market and the support for home lab users was limited, they are now mowing in the same direction as Proxmox has done all the time. Today the XCP tools are good. XCP needs the Xen Orchestra (must be installed by building it from source or buy a license) on a separate VM or a totally separate machine. That said, there is also the XOA and some more paid tools. ZFS in XCP-ng is hard to work with.

VMware has abandoned the FOSS market.

When Proxmox is one consolidated megalith of FOSS tools, VMware and Vates VMS are not. The consists of several independent pats, mostly closed software.

Xen Orchestra virtual Appliance

The Xen Orchestra virtual Appliance (XOA) is the pre-installed agentless solution VM. It's a web interface, accessible from any device. XOA can do VM creation, management, metrics and statistics, backup reports directly on Slack. The backup is an essential component for the security of your infrastructure. With Xen Orchestra, select the backup mode that suits you best and protect your VMs and your business.

Xen Orchestra

Xen Orchestra (XO) Web UI is a full-blown orchestration tool for your nodes, like what Proxmox has as default. You need to get a license or build it from source.

For private use, you can build the XO from GitHub sources, but it doesn't have QA and there's no stable version. It's great for a home lab or to make tests, but it is not for production.

Xen Orchestra Lite

Xen Orchestra Lite is a lightweight version of the Xen Orchestra meant for single-host administration, running directly from your browser without having to deploy anything, hosted on XCP-ng directly.
XO Lite is not an orchestrator, it's just a local management console. If you want to orchestrate your VMs (load balancing, backup, warm migration and so on), you must use Xen Orchestra!

XO Lite is still a work in progress! However, it's meant to cover all basic actions you need to boostrap your infrastructure or just do basic operation on your VMs.

XCP-ng Install

Download the installation ISO from link. Setup on a USB or Ventoy Disk.

Run the installation and set your preferences, takes 5–15 min.

Xen Orchestra Install

XOA is much better but need a license. First you need to create a VM and then download the source from GitHub and make. If you have space on your machine, do it in KVM/QEMU or VirtualBox. Another way is to use a free trial version of XOA to set up a VM and then create the XO in the VM.

You require 2 vCPU, 4 GB RAM and 10 GB disk for the VM and you need to have git and OpenSSL installed on it before starting the XO installation.

Security: Please don't expose XO to the WEB, it can be very dangerous. If somebody gets in on your XO, you can and probably will lose your data and for sure the data integrity of your network is shot to pieces.

Well, I have used the licensed version over the web for two weeks, but the security team made a lot of stuff for it to be safe. It did run through a revers proxy and used 2FA as a starter.

sudo apt update && sudo apt upgrade
sudo apt install git openssl #qemu-guest-agent nala exa 

sudo mkdir /opt/xo

git clone https://github.com/ronivay/XenOrchestraInstallerUpdater.git
cd XenOrchestraInstallerUpdater
cp sample.xo-install.cfg xo-install.cfg

Starting QGA sudo systemctl enable qemu-guest-agent.service --now

Edit the config-file nano xo-install.cfg

 21 PORT:"443"                            # We prefere HTTPS, 443
 
 80 PATH_TO_HTTPS_CERT=$INSTALLDIR/xo.crt # We will create the cert
 81 PATH_TO_HTTPS_KEY=$INSTALLDIR/xo.key  # We will create the key

Create the self sign certificate if you use HTTPS, recommended

sudo openssl req -newkey rsa:4096 \
            -x509 \
            -sha256 \
            -days 3650 \
            -nodes \
            -out /opt/xo/xo.crt  \
            -keyout /opt/xo/xo.key

Answer the questions for the certificate.

Run the installation script ./xo-install.sh, it will take some time.

Wait for the end message

You can now open XO at https://<Server IP or FQDN> credentials are [email protected] / admin

Update XO

By re-running the installation script ./xo-install.sh and choosing option 2.

XO Docker install

Use any Docker install and create the file below in a directory and run docker compose up -d to start it. Change port 8080 to what works for you.

version: "3.3"
services:
  xen-orchestra:
    container_name: xo
    stdin_open: true
    tty: true
    ports:
      - 8080:80
    volumes:
      - ./data/xo-server:/var/lib/xo-server
      - ./data/redis:/var/lib/redis
    image: ronivay/xen-orchestra
networks: {}

See this link for more information.

Conclusion

Will I switch back to Xen — No, but perhaps one or two servers? Why?

• For a home lab, Proxmox is more mature dealing with USB and other stuff in the GUI, as XCP-ng is doing much more in the CLI.
• Networking is easier to use in Proxmox.
• Proxmox Backup Server — it is excellent
• Proxmox Documentation and Wiki support is fantastic, and the Forums are active with staff members participating.
• The GUI of XCP-ng is more calm and professional than the bling of the Proxmox GUI.
• XCP-ng features a clear separation between storage and running VMs.
• Red Hat has not a good track record with FOSS

Both are Open-source with active community support. And they have Commercial support available.

For really large installations, Xen can be seen a better choice than KVM.

Clusters can be, in theory, 64 nodes, but close to or over 48 nodes will be challenging.

From a pure hypervisor perspective, I’ve always preferred Proxmox due to its ease of use. And I still do so. But XCP-ng roadmap indicates they do like to move closer.

Proxmox Containers are created and managed using the Proxmox Container Toolkit (pct). They also target system virtualization and use LXC as the basis of the container offering. The Proxmox Container Toolkit (pct) is tightly coupled with PVE. This means that it is aware of cluster setups, and it can use the same network and storage resources as QEMU VMs. You can even use the PVE firewall, create and restore backups, or manage containers using the HA framework. Everything can be controlled over the network using the PVE API. The security issues are the same as on LXC.

"Priviledged containers, they're not safe at all and should only be used in environments where unprivileged containers aren't available and where you would trust your container's user with root access to the host.
...
As privileged containers are considered unsafe, we typically will not consider new container escape exploits to be security issues worthy of a CVE and quick fix. We will however try to mitigate those issues so that accidental damage to the host is prevented." - Linuxcontainers.org

XCP-ng is an enterprise virtualization platform and needs more skills to run.

Migrating VMs between Proxmox and XCP-ng is possible using Xen Orchestra or various other methods like exporting/importing VM images. However, some features and configurations might not transfer correctly.

References

XEN ^[1] XCP-ng ^[2] XO ^[3] Vates: XCP-ng, Xen Orchestra tools and XOSTOR ^[4]