Proxmox VE Cluster Stability Improvement
Budget: $30 – $250 USD
We run a Proxmox VE 7.4 cluster with approximately 10 production nodes hosting client VMs.
The Problem:
Two nodes have been intermittently disconnecting and reconnecting from the cluster (flapping). Each time a node rejoins, it resets the global cluster MTU to 1397 (instead of 1500), which triggers a cascade of failures:
cpg_send_message failures
Token timeouts (Token has not been received in ~30,000ms)
pmxcfs leaving the CPG group
Full cluster becoming unresponsive (UI freezes, VMs unreachable from management plane)
We have isolated the two problematic nodes as a temporary fix, but we need a permanent solution.
Root cause (suspected): Network instability on bond0 interface causing a network loop, confirmed by:
vmbr0: received packet on bond0 with own address as source address
What we need:
A Proxmox/Corosync configuration that prevents one flapping node from destabilizing the entire cluster
Recommendations on corosync.conf tuning (token, MTU handling, etc.)
Best practices for bond0 configuration to prevent network loops
Ideally a monitoring/alerting script that detects MTU changes and auto-isolates the offending node
Important notes:
-No direct server access will be provided
-Full logs will be shared (journalctl, dmesg, corosync logs)
-We can run any diagnostic commands you specify and share output
-We can test configuration changes during a maintenance window
Skills needed: Proxmox VE, Corosync, Linux networking, bonding/bridging, cluster administration
Budget: Open to offers from experienced Proxmox admins only.
Key observation (important for diagnosis):
When we manually stop and mask corosync on the affected nodes:
bashsystemctl stop corosync
systemctl mask corosync
The entire cluster recovers immediately — UI returns, all other nodes go green, and cpg_send_message errors stop completely.
This confirms the issue is isolated to these two nodes and their network behavior, not a general cluster misconfiguration. The moment they are removed from the cluster ring, everything stabilizes.
The #1 goal:
We need a solution where if any single node becomes unstable or starts flapping, it gets automatically isolated, and go down — WITHOUT affecting the rest of the cluster.
Currently, one bad node can bring down all 10 nodes. This is unacceptable in a production environment.
The Problem:
Two nodes have been intermittently disconnecting and reconnecting from the cluster (flapping). Each time a node rejoins, it resets the global cluster MTU to 1397 (instead of 1500), which triggers a cascade of failures:
cpg_send_message failures
Token timeouts (Token has not been received in ~30,000ms)
pmxcfs leaving the CPG group
Full cluster becoming unresponsive (UI freezes, VMs unreachable from management plane)
We have isolated the two problematic nodes as a temporary fix, but we need a permanent solution.
Root cause (suspected): Network instability on bond0 interface causing a network loop, confirmed by:
vmbr0: received packet on bond0 with own address as source address
What we need:
A Proxmox/Corosync configuration that prevents one flapping node from destabilizing the entire cluster
Recommendations on corosync.conf tuning (token, MTU handling, etc.)
Best practices for bond0 configuration to prevent network loops
Ideally a monitoring/alerting script that detects MTU changes and auto-isolates the offending node
Important notes:
-No direct server access will be provided
-Full logs will be shared (journalctl, dmesg, corosync logs)
-We can run any diagnostic commands you specify and share output
-We can test configuration changes during a maintenance window
Skills needed: Proxmox VE, Corosync, Linux networking, bonding/bridging, cluster administration
Budget: Open to offers from experienced Proxmox admins only.
Key observation (important for diagnosis):
When we manually stop and mask corosync on the affected nodes:
bashsystemctl stop corosync
systemctl mask corosync
The entire cluster recovers immediately — UI returns, all other nodes go green, and cpg_send_message errors stop completely.
This confirms the issue is isolated to these two nodes and their network behavior, not a general cluster misconfiguration. The moment they are removed from the cluster ring, everything stabilizes.
The #1 goal:
We need a solution where if any single node becomes unstable or starts flapping, it gets automatically isolated, and go down — WITHOUT affecting the rest of the cluster.
Currently, one bad node can bring down all 10 nodes. This is unacceptable in a production environment.
Related categories:
System Admin
Linux
Education
Ubuntu
Network Administration
Virtualization
Automation