Maximize bandwidth with Dell 25G network, Debian, TrueNAS, VMware and NFS
Budget: $30 – $250 USD
I have got 2x R740xd computers. They both have 12x Dell 4TB NVMe disks in them.
Both are running VMware ESXi 8.0. The NVMe disks and also the NICs are passthrough to the VMs.
I have got a Dell_switch_S5048F–ON and Dell "BCM57414 NetXtreme-E 10Gb/25Gb" cards in both servers.
2 ports are LAGGed together everywhere (but also tried without LAGG).
Using NFS4 with TrueNAS Scale (debian) on one server and debian testing on the other server.
Under TrueNAS there is 6-6 NVMe in 2 vdevs but I can change this if needed (raidz2).
I am measuring speed with rsync and 44GB file, after making sure nothing is cached and the best I can't get better numbers than 150-200 MB/sec, which is not good.
I have tried tweaking whatever I found on the net (MTU on all devices including switch, sysctl.conf stuff etc), no help.
If we can reach 1000+ MB/sec, I would be much happier and I would say that the project is a success, but that is still lousy so the faster the better.
During the project you will not be able to reach the computers, but I will do as you say and give you screenshots etc.
If minimum 1000 MB/sec is not reached for both write and read, the project is unsuccessful.
Both are running VMware ESXi 8.0. The NVMe disks and also the NICs are passthrough to the VMs.
I have got a Dell_switch_S5048F–ON and Dell "BCM57414 NetXtreme-E 10Gb/25Gb" cards in both servers.
2 ports are LAGGed together everywhere (but also tried without LAGG).
Using NFS4 with TrueNAS Scale (debian) on one server and debian testing on the other server.
Under TrueNAS there is 6-6 NVMe in 2 vdevs but I can change this if needed (raidz2).
I am measuring speed with rsync and 44GB file, after making sure nothing is cached and the best I can't get better numbers than 150-200 MB/sec, which is not good.
I have tried tweaking whatever I found on the net (MTU on all devices including switch, sysctl.conf stuff etc), no help.
If we can reach 1000+ MB/sec, I would be much happier and I would say that the project is a success, but that is still lousy so the faster the better.
During the project you will not be able to reach the computers, but I will do as you say and give you screenshots etc.
If minimum 1000 MB/sec is not reached for both write and read, the project is unsuccessful.