The Issue: File Copies within a mount point are fast on Ubuntu 14 and slow on Ubuntu 20x Case Setup:
2 Different VCenters located in different physical locations using their own esxi hosts and datastores.
Installed 4 virtual servers using the exact same options including CPU, MEM, HD ext4 Config, SCSI, etc.
2 of these servers are installed in VCenter1. Server1 Ubuntu 14, Server2 Ubuntu 23
2 other servers installed in VCenter2. Server3 Ubuntu 14, Server4 Ubuntu 24
During installation, I chose only RemoteSSH as an extra option. After, I ran standard package updates for existing packages, installed latest vm-tools.
In both locations, the Ubuntu 14 servers were significantly faster than Ubuntu 23 when performing a simple file copy test:
Sample output:
Server1 Ubuntu 14:/d01/user# time for ((i = 0 ; i < 10 ; i++ )) ; do cp ubuntu-20.04.6-live-server-amd64.iso test.iso ; rm test.iso ; done
real 0m24.171s user 0m0.068s sys 0m11.012s
Server2 Ubuntu 23:/d01/user# time for ((i = 0 ; i < 10 ; i++ )) ; do cp ubuntu-20.04.6-live-server-amd64.iso test.iso ; rm test.iso ; done
real 0m47.093s user 0m0.011s sys 0m38.125s
The same results for Server3 and Server4 on the 2nd VCenter located in another datacenter.
Server1: hdparm -Tt /dev/sdb /dev/sdb: Timing cached reads: 16160 MB in 2.00 seconds = 8085.73 MB/sec Timing buffered disk reads: 498 MB in 3.01 seconds = 165.26 MB/sec
Server2: hdparm -Tt /dev/sdb /dev/sdb: Timing cached reads: 10482 MB in 1.99 seconds = 5274.00 MB/sec Timing buffered disk reads: 776 MB in 3.00 seconds = 258.65 MB/sec
IOStat Snippet during File Copy:
Server1: iostat -dx 2 | grep -E "Device|sdb" | tee /tmp/Server1_iostat_sdb_log.txt Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.35 5.73 13.11 1093.82 12694.15 1463.88 1.70 90.00 6.28 126.60 2.12 4.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.50 0.00 49.00 0.00 33792.00 1379.27 3.27 37.96 0.00 37.96 1.02 5.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.00 514.43 0.00 526774.13 2048.00 41.65 83.71 0.00 83.71 1.66 85.37 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.54 465.41 2.16 458862.70 1969.61 106.03 206.17 268.00 206.10 1.94 90.59
Server2: iostat -dx 2 | grep -E "Device|sdb" | tee /tmp/Server2_iostat_sdb_log.txt Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 347.00 403456.00 0.00 0.00 101.41 1162.70 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 35.19 66.00 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 40.50 47104.00 0.00 0.00 40.65 1163.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.65 7.20 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 300.00 329822.00 20.00 6.25 41.83 1099.41 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.55 55.80 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 225.50 262144.00 0.00 0.00 172.07 1162.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 38.80 54.60 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 173.50 188462.00 12.00 6.47 61.69 1086.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10.70 40.00 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 342.50 398288.00 0.00 0.00 68.18 1162.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23.35 61.40
I have attempted several changes on Server2 to which include: modify dirty kernel parameters, disable snapd, various mount options, different scheduler options(none, noop, etc), verified virtio is installed, grub GRUB_CMDLINE_LINUX_DEFAULT changes, sysctl parameters, hugepages, e2fsck, tune2fs, sync drop_caches, ntp stop, and a few more OS level changes.
I removed journaling from sdb1 ext4 on Server2. No file copy performance improvements.
Here are a few more details for consideration as to why Ubuntu20x is slower than Ubuntu14 within our VCenter:
ESXi Host version that both Server1 and Server2 reside:
Hypervisor: VMware ESXi, 6.7.0, 17499825
Datastore that both Server1 and Server2 vmdks reside: Type: NFS 3
If anyone knows where the smoking gun resides or has any advised direction, I truly appreciate your assistance.
2 Different VCenters located in different physical locations using their own esxi hosts and datastores.
Installed 4 virtual servers using the exact same options including CPU, MEM, HD ext4 Config, SCSI, etc.
2 of these servers are installed in VCenter1. Server1 Ubuntu 14, Server2 Ubuntu 23
2 other servers installed in VCenter2. Server3 Ubuntu 14, Server4 Ubuntu 24
During installation, I chose only RemoteSSH as an extra option. After, I ran standard package updates for existing packages, installed latest vm-tools.
In both locations, the Ubuntu 14 servers were significantly faster than Ubuntu 23 when performing a simple file copy test:
Sample output:
Server1 Ubuntu 14:/d01/user# time for ((i = 0 ; i < 10 ; i++ )) ; do cp ubuntu-20.04.6-live-server-amd64.iso test.iso ; rm test.iso ; done
real 0m24.171s user 0m0.068s sys 0m11.012s
Server2 Ubuntu 23:/d01/user# time for ((i = 0 ; i < 10 ; i++ )) ; do cp ubuntu-20.04.6-live-server-amd64.iso test.iso ; rm test.iso ; done
real 0m47.093s user 0m0.011s sys 0m38.125s
The same results for Server3 and Server4 on the 2nd VCenter located in another datacenter.
Server1: hdparm -Tt /dev/sdb /dev/sdb: Timing cached reads: 16160 MB in 2.00 seconds = 8085.73 MB/sec Timing buffered disk reads: 498 MB in 3.01 seconds = 165.26 MB/sec
Server2: hdparm -Tt /dev/sdb /dev/sdb: Timing cached reads: 10482 MB in 1.99 seconds = 5274.00 MB/sec Timing buffered disk reads: 776 MB in 3.00 seconds = 258.65 MB/sec
IOStat Snippet during File Copy:
Server1: iostat -dx 2 | grep -E "Device|sdb" | tee /tmp/Server1_iostat_sdb_log.txt Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.35 5.73 13.11 1093.82 12694.15 1463.88 1.70 90.00 6.28 126.60 2.12 4.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.50 0.00 49.00 0.00 33792.00 1379.27 3.27 37.96 0.00 37.96 1.02 5.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.00 514.43 0.00 526774.13 2048.00 41.65 83.71 0.00 83.71 1.66 85.37 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.54 465.41 2.16 458862.70 1969.61 106.03 206.17 268.00 206.10 1.94 90.59
Server2: iostat -dx 2 | grep -E "Device|sdb" | tee /tmp/Server2_iostat_sdb_log.txt Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 347.00 403456.00 0.00 0.00 101.41 1162.70 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 35.19 66.00 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 40.50 47104.00 0.00 0.00 40.65 1163.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.65 7.20 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 300.00 329822.00 20.00 6.25 41.83 1099.41 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.55 55.80 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 225.50 262144.00 0.00 0.00 172.07 1162.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 38.80 54.60 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 173.50 188462.00 12.00 6.47 61.69 1086.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10.70 40.00 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 342.50 398288.00 0.00 0.00 68.18 1162.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23.35 61.40
I have attempted several changes on Server2 to which include: modify dirty kernel parameters, disable snapd, various mount options, different scheduler options(none, noop, etc), verified virtio is installed, grub GRUB_CMDLINE_LINUX_DEFAULT changes, sysctl parameters, hugepages, e2fsck, tune2fs, sync drop_caches, ntp stop, and a few more OS level changes.
I removed journaling from sdb1 ext4 on Server2. No file copy performance improvements.
Here are a few more details for consideration as to why Ubuntu20x is slower than Ubuntu14 within our VCenter:
ESXi Host version that both Server1 and Server2 reside:
Hypervisor: VMware ESXi, 6.7.0, 17499825
Datastore that both Server1 and Server2 vmdks reside: Type: NFS 3
If anyone knows where the smoking gun resides or has any advised direction, I truly appreciate your assistance.