QNAP, Inc. - Network Attached Storage (NAS)

Language

Support

Using iSCSI Extensions For RDMA (iSER), to improve the performance of QNAP NAS with VMware® ESXi

1. Challenges

In a virtualization environment, the drives, storage configuration, network, hypervisor and applications are all critical factors in overall performance and stability. IT personnel for Small and Medium-sized Businesses (SMB) are often required to optimize performance and identify bottle-necks.

As a storage provider that is committed to the SMB market, QNAP recognizes these challenges and understands that the key to increasing storage performance not only lies not only in the storage system but also in the storage network. QNAP now supports iSCSI Extensions for RDMA (iSER), a solution which has already been adopted by enterprises for years, on all the major series of QNAP SMB NAS products. Our goal is to provide IT professionals with an indispensable solution that can greatly improve performance in virtualized environments, and allow their business to increase overall IT efficiency.

Enabling iSER increases the performance of a QNAP Hybrid Storage NAS with Qtier™ by more than 50% in QNAP Lab tests. Refer to Chapter 5 for details.

2. Introduction to iSCSI Extensions for RDMA (iSER)

In QTS version 4.3.3 and later, iSER can be enabled on iSCSI connections between VMware virtualization servers and Intel or AMD-based QNAP NAS. The only requirement is that each server and NAS has an iSER-compatible network card.

The iSER protocol differs from traditional iSCSI, as it allows data being transferred to bypass the network driver and socket layers, and enter the memory buffers of the ESXi server or NAS directly. This gives better data transfer performance, and also reduces both latency and the CPU load on the NAS.

QNAP found that the performance of virtual machines can be increased by 60% to 100%* with iSER. Also, the latency of random access can be decreased by 50% or more. These test results show that iSER is especially suitable for businesses that use QNAP NAS as VMware ESXi datastores, hosting business applications or virtual desktop infrastructure (VDI). 

The following chapters list the system requirements and the steps required to enable iSER in a virtualization environment. The document also provides a basic demonstration of how iSER can improve the performance of a hybrid storage NAS using Qtier(™), and an all flash NAS.

iSER (Right) bypasses the TCP/IP, Network Driver and Socket Library to transfer data directly to application or storage memory.

*Refer to chapter 5 for the test configuration and results. Performance improvements will vary, depending on your system configuration.

3. System Requirements

Enabling iSER in your QNAP and VMware environment requires the following:

  • One or more QNAP NAS with an Intel or AMD CPU, running QTS 4.3.3 or later.
  • One or more VMware ESXi servers (currently QTS only supports iSER with VMware ESXi).
  • An iSER-compatible network card for each QNAP NAS and VMware ESXi server.
  • The iSER compatibility driver** MLNX-OFED-ESX-1.9.10.5 needs be installed in VMware ESXi 5.5 and 6.0.  The driver is available at http://www.mellanox.com/page/products_dyn?product_family=29
    Note: VMware ESXi 6.5 is still not support the iSER protocol currently.
  • The NAS and VMware ESXi servers must be connected either with a Mellanox Switch that support iSER***, or connected directly without a switch.
    Note: Other switches which support PFC (Priority Flow Control) can also be used, but in that case the amount of performance increase from iSER cannot be guaranteed.

*The current network cards that can be used with iSER are listed below. To purchase expansion cards, visit the QNAP website or contact an authorized distributer.

Model Spec Product
LAN-10G2SF-MLX Dual-port 10 GbE Adapter: Mellanox Connect-3 Pro EN Connector: SFP+
LAN-40G2SF-MLX Dual-port 40 GbE Adapter: Mellanox ConnectX-3 Pro EN Connector: QSFP

** This driver is specifically designed to enable iSER. Installing it may affect regular iSCSI connection performance.

*** Below is a list of Mellanox Ethernet cards and switches recommended for use with iSER. The Mellanox Spectrum switches deliver consistently ultra-low latency and non-blocking switching at line rate without packet loss. Combined with optimal congestion management, the Mellanox switches allow the maximal iSER performance. For more information on iSER-compatible network cards and switches, refer to the Mellanox official website or contact an authorized distributer.

Network Card Spec
MCX311A-XCCT ConnectX®-3 Pro EN network interface card, 10GbE, single-port SFP+, PCIe3.0 x8 8GT/s, tall bracket, RoHS R6
MCX312B-XCCT ConnectX®-3 Pro EN network interface card, 10GbE, dual-port SFP+, PCIe3.0 x8 8GT/s, tall bracket, RoHS R6
MCX312C-XCCT ConnectX®-3 Pro EN network interface card, 10GbE, dual-port SFP+, PCIe3.0 x8 8GT/s, tall bracket, RoHS R6
MCX313A-BCCT ConnectX®-3 Pro EN network interface card, 40/56GbE, single-port QSFP, PCIe3.0 x8 8GT/s, tall bracket, RoHS R6
MCX314A-BCCT ConnectX®-3 Pro EN network interface card, 40/56GbE, dual-port QSFP, PCIe3.0 x8 8GT/s, tall bracket, RoHS R6
MCX342A-XCPN ConnectX®-3 Pro EN network interface card for OCP, 10GbE dual-port SFP+, PCIe3.0 x8, no bracket, RoHS R6
MCX342A-XCQN ConnectX®-3 Pro EN network interface card for OCP with IPMI and NC-SI, 10GbE dual-port SFP+, PCIe3.0 x8, no bracket, RoHS R6
MCX345A-BCPN ConnectX®-3 Pro EN network interface card for OCP, 40GbE single-port QSFP, PCIe3.0 x8, no bracket, RoHS R6
MCX345A-BCQN ConnectX®-3 Pro EN network interface card for OCP with IPMI and NC-SI, 40GbE single-port QSFP, PCIe3.0 x8, no bracket, RoHS R6
MCX346A-BCPN ConnectX®-3 Pro EN network interface card for OCP, 40GbE dual-port QSFP, PCIe3.0 x8, no bracket, RoHS R6
MCX346A-BCQN ConnectX®-3 Pro EN network interface card for OCP with IPMI and NC-SI, 40GbE dual-port QSFP, PCIe3.0 x8, no bracket, RoHS R6
 
Model Spec
MSN2700-BS2F Spectrum™ based 40GbE 1U Open Ethernet Switch with MLNX-OS, 32 QSFP 28 ports
MSN2700-BS2R Spectrum™  based 40GbE, 1U Open Ethernet Switch with MLNX-OS, 32 QSFP 28 ports
MSN2410-BB2F Spectrum™ based 10GbE/100GbE 1U Open Ethernet switch with MLNX-OS, 48 SFP 28 ports
MSN2100-BB2F Spectrum™ based 40GbE 1U Open Ethernet Switch with MLNX-OS, 16 QSFP 28 ports
MSN2100-BB2R Spectrum™ based 40GbE 1U Open Ethernet Switch with MLNX-OS, 16 QSFP28 ports

4. Deploying iSER in an Existing VMware Virtualization Environment

This chapter contains instructions on how to enable iSER on an iSCSI connection between one QNAP NAS and one VMware ESXi server. If your environment contains multiple NAS or multiple VMware servers, follow the applicable steps for each NAS and server.

4-1. Configuring network settings on the NAS

Before changing any settings, ensure that the QNAP NAS is an Intel or AMD based model, and is already running QTS 4.3.3 or later.

  1. Shut down the NAS and the VMware ESXi server, and install an iSER-compatible network card in each device.
  2. Connect the NAS and the VMware ESXi server, either via an iSER-compatible switch, or by connecting them directly.
  3. Turn on the NAS and ESXi server.
  4. Log into QTS on the NAS and go to “Control Panel” > “Network & Virtual Switch” > “Interfaces”. Find the interfaces that belong to the iSER-compatible network card, and assign them each a static IP address. Also setting “Jumbo Frame” to 9,000 is recommended, to maximize performance improvements when using iSER.
Edit the newly added network card’s adapter in “Network & Virtual Switch”. The Jumbo Frame settings can be found by clicking the “Configure” icon.

4-2. Installing the Mellanox iSER driver (Not applicable for VMware ESXi 6.5 or later)

The Mellanox iSER driver must be installed on each ESXi server. Note: VMware ESXi 6.5 and later natively supports iSER and does not require this driver.

  1. Download the Mellanox iSER driver MLNX-OFED-ESX-1.9.10.5 from http://www.mellanox.com/page/products_dyn?product_family=29
  2. Upload the driver to the ESXi server. In this tutorial, we saved the driver to the NAS directly, and then use Putty and SSH to upload the driver to the ESXi server.
Use the command #scp to upload the driver to ESXi server’s root directory
  1. (Optional) Remove any previous versions of the driver.
    The v1.9.10.5 driver cannot be used at the same time as any other version of the same driver. If a version of the driver other than MLNX-OFED-ESX-1.9.10.5 is already installed, stop all operations and virtual machines on the ESXi server, and then remove the existing driver.

    Run the command esxcli software vib list | grep mlx to confirm the currently installed driver version and name. Below is the command for removing a previous version of the driver, which has three parts: nmlx4-core, nmlx4-en and nmlx4-rdma.
    esxcli software vib remove -n (nmlx4-core)
    esxcli software vib remove -n (nmlx4-en)
    esxcli software vib remove -n (nmlx4-rdma)
  2. Run the following command to install the new driver:
    esxcli software vib install -d /MLNX-OFED-ESX-1.9.10.5-10EM-600.0.0.2494585.zip
  3. After installation is complete, please restart the VMware ESXi and run the command esxcli software vib list | grep mlx to verify that the net-mlx4-core, net-mlx4-en and net-mlx4-ib drivers are installed, and all are version 1.9.10.5-10EM-600.0.0.2494585.
Putty can be used to install the driver on an ESXi server.

4-3. Configuring network settings on the ESXi server

This section guides you through setting up the iSER connection on an ESXi server.

  1. Log into VMware vCenter.
  2. Select the VMware ESXi server in the left panel.
  3. Go to “Configuration” Tab > “Networking” and click “Add Networking” to open the “Add Network Wizard.
Add a new virtual switch for the ESXi server from vCenter.
  1. If you wish to let the Guest OS can access the network interface, select “Virtual Machine” as the “Connection Type” and click “Next” first, otherwise, go to step 6.
  2. On the “Network Access” page, select the newly installed iSER-compatible network card. Click “Next” and “Next” leaving all settings as default, and then click “Finish” to complete the “Add Network” wizard.
Select the newly added network card to continue.
  1. Click “Add Networking” again. Select “VMkernel” as the “Connection Type”, ensure the newly installed iSER-compatible network card is selected and following the wizard to completed the settings. Leave the “Port Group” properties as default.
  2. On the “IP Settings” page, specify a static IP address for the network adapter. Then click “Next” and “Finish” to close the wizard.
Continue to add the VMkernel type and specify a static IP for the network card
  1. Ensure that you see the new virtual switch for the iSER network card on the “Configuration” > “Networking” screen.
The new virtual switch contains the VM network (Optional) and VMkernel.

4-4. Changing the network connection from iSCSI to iSER

  1. Log into VMware vCenter.
  2. Select the VMware ESXi server in the left panel.
  3. Go to “Configuration” Tab > “Storage Adapters”. On this page, you should be able to see the iSER displayed as “Mellanox iSCSI over RDMA (iSER) Adapter” in addition to the existing “iSCSI Software Adapter”.
    Note: If you are setting up a new virtualization environment, you need to create the “iSCSI software adapter” first, then restart the ESXi server to see the “Mellanox iSCSI over RDMA (iSER)” adapter.
    The Mellanox iSCSI over RDMA (iSER) Adapter appears after the NAS and VMware ESXi network settings are configured in section 4-3
  4. Select the “iSCSI Software Adapter” device and click “Properties”.
  5. Remove all existing port bindings from the list on the “Network Configurations” tab. Also remove all iSCSI targets from the lists on the “Dynamic Discovery” and “Static Discovery” tabs. This ensures that connections will only use the iSER protocol.
    Clear all items from the lists in “Network Configuration”, “Dynamic Discovery” and “Static Discovery”
  6. Close the properties for the iSCSI software adapter. On the “Configuration” > “Storage Adapters” page, select the device under “Mellanox iSCSI over RDMA (iSER) Adapter” and click “Properties”.
  7. On the “Network Configuration” tab, click “Add” to create a new VMKernel port binding. Select the VMKernel created for the iSER connection in section 4-3.
    Edit the Mellanox iSCSI over RDMA (iSER) Adapter device properties and add the iSER virtual switch to rescan iSCSI target.
  8. Repeat steps 5 and 6 for each device under “Mellanox iSCSI over RDMA (iSER) Adapter” that requires an iSER connection.

    The QNAP iSCSI target and LUN will now be automatically connected using the new iSER interface.
    The iSCSI LUN on the QNAP NAS is connected using the iSER adapter.

5. Performance tests and results

The following equipment was used by the QNAP testing team to measure the performance benefits of iSER compared to regular iSCSI:

QNAP NAS

  • TVS-EC2480-SAS-RP: hybrid storage array
  • TES-EC3085U: all flash storage array

VMware

  • VMware ESXi 6.0 cluster of five Dell PowerEdge R420 servers. Each server had an Intel Xeon E5-2450 CPU and 128GB of RAM.
  • Virtual machine OS: Windows 2012 R2 64bit

Networking

  • Mellanox 40GbE LAN-40G2SF-MLX Network Card, installed in the NAS and in each PowerEdge server.
  • Mellanox 40GbE Spectrum SN2410 Switch, connecting the VMware ESXi servers and NAS.

Note: To show performance gains, only one 40GbE cable was used to connect each NAS to the Mellanox 40GbE Switch. However, in normal conditions it is possible to configure multiple 10GbE and 40GbE connections and multiple switches to further increase performance and storage availability with iSER.

Iometer was installed on each VMware ESXi server and configured with the following settings.

Iometer (iSCSI Throughput)
Outstanding I/Os→ 1 per target Write Throughput → 2M Sequential 100%
Write 100%
Workers → 20 Read Throughput → 2M Sequential 100%
Read 100%
Ramp up time →30 seconds Transferring Size → 16GB file
Iometer (iSCSI IOPS)
Outstanding I/Os → 1 per target Write IOPS → 4K Sequential 100%
Write 100%
Workers → 20 Read IOPS → 4K Sequential 100%
Read 100%
Ramp up time →30 seconds Transferring Size → 16GB file

Note: This test used common Iometer settings. Real-world access patterns will vary depending on the application.

5-1. One VM with Qtier™: Sequential Read/Write performance increased 100% with iSER to 4000 MB/s

In this test, we tested the performance of one virtual machine connected to the hybrid storage NAS. This kind of configuration is common for business applications such as mail servers, which require both good performance and high storage capacity.

NAS Specifications
Model QNAP TVS-EC2480-SAS-RP
CPU Intel Xeon E3-1246
Memory 32GB
Storage Configuration Seagate SAS SSD x12, RAID 10
Seagate SATA HDD x12, RAID 6
1 Storage Pool, 1 Block-based iSCSI LUN
Network Card 40GbE Network Card LAN-40G2SF-MLX

In the test results, we see an increase of over 100% in sequential reads and write speeds, reaching a sequential read speed of over 4,000MB/s. This result shows that iSER can greatly improve the sequential read and write performance of high demand applications.



5-2. Five VMs with Qtier™: Random Read/Write performance increased 80% with iSER to 180,000 IOPS

This second test uses the same hybrid storage NAS configuration as the previous test. However, this time we increased the number of VMware virtual machines to five. This test case is representative of a small to medium business that wants to run multiple applications requiring random access, such as a web or log server database, on a single QNAP NAS.

The test results combine the performance of all virtual machines to show the total read and write performance. The results show that the random read and write performance increased over 80% when using iSER. Random read/writes reached a total of 180,000 IOPS with only 12 SSDs in a RAID 10 configuration. There was also an increase in sequential access speeds. 
Latency was reduced from 0.9 ms to 0.5 ms, improving response time when multiple applications require random access to the same NAS.



5-3. Five VMs with all flash storage: Random Read/Write performance increased 60% with iSER to 250,000 IOPS

In the previous test, each VM was able to gain about 36,000 IOPS using the hybrid storage NAS with Qtier™. For certain business applications such as SQL server, these results still need to be improved. To further explore the benefits of using iSER, we changed the NAS to a QNAP TES-3085U. This is an all flash storage NAS with 30 SSD slots. The NAS was connected to five VMware virtual machines.

This test setup does not require Qtier, so one large storage pool is no longer needed. For this test the 30 SSDs were divided into 5 storage pools, each containing a block-based iSCSI LUN. This storage configuration decreases the risk of data loss, and also increases performance, as meta-data can be retrieved independently from each storage pool.

NAS Specifications
Model QNAP TES-3085U
CPU Intel Xeon D-1548
Memory 32GB
Storage Configuration Samsung SATA SSD x30, RAID 10
5 Storage Pools
5 Block-based iSCSI LUNs
Network Card 40GbE Network Card LAN-40G2SF-MLX

During the test, we combined the performance of multiple virtual machines to observe the total read and write performance. Random read and write performance increased by more than 80% with iSER, reaching a total of 250,000 IOPS. This means each VM can get up to 50,000 IOPS. This number exceeds 30,000 IOPS, which is the general recommendation for an SQL server. Latency was reduced to 0.3 ms, compared to 1.1 ms with iSCSI.


5-4. Test Conclusions

With iSER the random read and write performance of multiple virtual machines can be increased by up to 80%, and latency can be reduced by about 50%. These results apply to both hybrid storage and an all flash NAS.

The conclusion from the QNAP test team is that iSER can effectively increase the performance of a QNAP NAS when used as a VMware ESXi datastore. This increases the performance of key applications that run on the VMware cluster. By adding iSER support to QTS, QNAP demonstrates innovation and a commitment to delivering fast performance efficient storage to its customers around the world.

QNAP believes that iSER is an excellent option for IT professionals who want to increase business application performance. In the future QNAP will continue to test iSER in different configurations and then publish the test results. 

Release date: 2017-05-25
Was it helpful?
Thank you for your feedback.
Thank you for your feedback. If you have any question, please contact support@qnap.com
100% of people think it helps.