Best practices for testing 100GbE network performance on QNAP NAS?

Applicable Products

QNAP 100GbE Networking with QNAP NAS

Best Practice

Hardware Requirements

• Clients:
  • PCIe Gen3 x16 or Gen4 x8 slot
  • High-performance CPU with strong single-core frequency.
  • Adequate system memory to handle high throughput.
  • High-speed storage solution or RAM disk for accurate throughput testing.
• NAS:
  • PCIe Gen3 x16 or Gen4 x8 slot for NIC
  • Validate disk model performance
  • Cards, Cable and the GBIC:
  • 100GbE card and the GBIC are in the compatibility list for the NAS model.
  • QSFP cables (4-lane combined) compatible with 100Gb networks.

Environment Setup

• Install Correct Network Drivers both PC and NAS:
  • PC Side Configuration
  • Set MTU to 9000.
  • Disable Interrupt Moderation.(particularly on Windows). 
  • Check disk I/O capability.

• NAS Side Configuration
  • Set MTU to 9000.
  • Use ifconfig to check TX/RX errors.
  • Ensure SAMBA advanced setting > KSMBD and AIO are enabled
  • Ensure that snapshots and encrypted volumes/shares are disabled.
  • Without SED pool
• CPU Performance Monitoring
  • During performance tests, closely monitor CPU utilization using tools such as top or htop.
  • If any single CPU core exhibits sustained high usage (approaching 100%), it indicates a CPU bottleneck. This means the CPU is insufficiently powerful for 100Gb network testing.

Networking test with iperf3

• Multi-process Test Methodology
  • On NAS (server-side): 
    • Open multiple iperf3 server SSH consoles with each port:

iperf3 -s -f M -p 50001 & iperf3 -s -f M -p 50002 & iperf3 -s -f M -p 50003 & iperf3 -s -f M -p 50004 &

• After testing complete

killall iperf3

• On Client Side:
  • Conduct tests with multiple parallel processes to fully utilize the 100Gb bandwidth
    • Linux example:

iperf3 -c <IP> -f M -i 1 -P 4 -t 60 -p 50001 & iperf3 -c <IP> -f M -i 1 -P 4 -t 60 -p 50002 & iperf3 -c <IP> -f M -i 1 -P 4 -t 60 -p 50003 & iperf3 -c <IP> -f M -i 1 -P 4 -t 60 -p 50004 &

• Windows batch script:

@echo off
start "iperf3-50001" cmd /k "iperf3.exe -c <ip> -f M -i 1 -t 60 -P 1 -p 50001"
start "iperf3-50002" cmd /k "iperf3.exe -c <ip> -f M -i 1 -t 60 -P 1 -p 50002"
start "iperf3-50003" cmd /k "iperf3.exe -c <ip> -f M -i 1 -t 60 -P 1 -p 50003"
start "iperf3-50004" cmd /k "iperf3.exe -c <ip> -f M -i 1 -t 60 -P 1 -p 50004"

echo press any key to close
pause >nul

• iperf3 example:
  • Write speed=2692+3300+3572+2333=11897 MB/s

• Read speed = 2966+3130+3844+1320 = 11260 MB/s

File System Testing

• FIO for Linux：

fio --filename=Test1 --direct=1 --rw=write --bs=1M --ramp_time=10 --runtime=60 --name=test-write --ioengine=libaio --iodepth=32 --numjobs=32 --size=100G
fio --filename=Test2 --direct=1 --rw=read --bs=1M --ramp_time=10 --runtime=60 --name=test-read --ioengine=libaio --iodepth=32 --numjobs=32 --size=100G

FIO for Windows

fio --filename=fio.dat --rw=write --bs=1M --ramp_time=10 --runtime=60 --name=test-write --ioengine=windowsaio --iodepth=32 --numjobs=32 --size=100G
fio --filename=fio.dat --rw=read --bs=1M --ramp_time=10 --runtime=60 --name=test-read --ioengine=windowsaio --iodepth=32 --numjobs=32 --size=100G --fallocate=none

• Windows File Explorer:
  • File Browser = single-thread
  • Write performance improves with multi-process copying.
  • Reading limited by CPU core
• Robocopy: 
  • Write Performance
  • PowerShellPowerShell

    robocopy A:\10G I:\10G /MIR /IS /NP /NDL /NFL /MT:32

  • Write Performance = 7819298321 Bytes/sec = 7453 MB/s
  • Robocopy has limitation in read performance.

Checklist by layers:

1. Hardware connection and Interface checking.
   1. Checked Hardware supported 100Gb
   2. Checked 100Gb Adapter, Fibre module/DAC cable are supported by compatibility list.
   3. Confirmed connection rate and  Duplex mode
2. System resource and interrupt (Linux client)
   1. Checked CPU core loading and interrupt

      #check every cpu core usage
      top
      htop 
      # Check CPU cores handling ethX interrupts.
      cat /proc/interrupts |grep ethX  
      # Debug IRQ distribution across CPU cores.
      irqbalance –debug

      
      
   2. Ethernet adapter to set cpu core affinity (RPS / XPS)
      ShellShell

      #list the cpu core
      lscpu | grep '^CPU(s):'
      #Set which core to handle the threads
      echo <core mask> > /sys/class/net/ethX/queues/rx-0/rps_cpus

      
      
   3. Check Ethernet Adapter driver & Firmware version 
      ShellShell

      ethtool –i ethX

      
      
3. Network stack and performance parameters (Linux client)
   1. Managed Switch must be set up to MTU 9000
   2. MTU setting Jumbo Frame to 9000hell

      ifconfig ethX mtu 9000 up

      ffer limit
   3. Check TCP buffer limitShell
      
      
      ShellShell

      sysctl net.core.rmem_max
      sysctl net.core.wmem_max

      1. Suggestion value

         sysctl -w net.core.rmem_max=16777216
         sysctl -w net.core.wmem_max=16777216

   4. Enabled TSO / GSO / SG

      ethtool –K ethX tso on gso on sg on

      
      
4. Alternative Actual Performance Test
   1. Single machine test
   2. External test
   3. Use RAM Disk avoid disk bottleneck
      1. Linux

         mount -t tmpfs -o size=16G tmpfs /mnt/ramdisk

         
         
      2. Windows: Search for ramdisk tool.
   4. Make transfer test (SMB / iSCSI / NFS)
      1. Test source file store to RAM Disk
      2. Test with other tool to check, example: dd / robocopy / rsync / fio