Using ES NAS (Enterprise Storage NAS) for Microsoft Exchange Server
Overview
This document details an I/O stress test of QNAP ES1640DC enterprise storage being used to provide Microsoft Exchange Services. We prepared a Windows Server 2012 R2 virtual machine (VM), and installed Exchange Server 2013. Stress testing was performed with the Microsoft JetStress Utility to verify I/O performance.
Each Windows Server 2012 R2 VM has four iSCSI LUNs attached. These are provided by QNAP ES1640DC to store databases and logs. In addition, each Exchange Server has Multipath I/O (MPIO) set up to have failover, and load balancing capability during messaging operations.
Infrastructure Components
System Network Configuration
| Primary Site Storage Network Settings | ||
|---|---|---|
| Role | IP | Description |
| SCA Management | 192.168.200.101 | Management IP in SCB |
| SCA Ethernet 1 | 10.1.1.101 | 10GBase-T Lan port 1 in SCA |
| SCA Ethernet 2 | 10.1.2.101 | 10GBase-T Lan port 2 in SCA |
| SCB Management | 192.168.200.102 | Management IP in SCB |
| SCB Ethernet 1 | 10.1.1.102 | 10GBase-T Lan port 1 in SCB |
| SCB Ethernet 2 | 10.1.2.102 | 10GBase-T Lan port 2 in SCB |
| Primary Site ESXi Cluster Network Settings | ||
| Role | IP | Description |
| ESXi 1 Management | 192.168.200.1 | Management IP in ESXi 1 |
| ESXi 1 Ethernet1 | 10.1.1.11 | 10G Data port 1 in ESXi 1 |
| ESXi 1 Ethernet2 | 10.1.2.11 | 10G Data port 2 in ESXi 1 |
| ESXi 2 Management | 192.168.200.2 | Management IP in ESXi 2 |
| ESXi 2 Ethernet1 | 10.1.1.12 | 10G Data port 1 in ESXi 2 |
| ESXi 2 Ethernet2 | 10.1.2.12 | 10G Data port 2 in ESXi 2 |
| Secondary Site Storage Network Settings | ||
| Role | IP | Description |
| SCA Management | 192.168.201.101 | Management IP in SCB |
| SCA Ethernet 1 | 10.1.3.101 | 10GBase-T Lan port 1 in SCA |
| SCA Ethernet 2 | 10.1.4.101 | 10GBase-T Lan port 2 in SCA |
| SCB Management | 192.168.201.102 | Management IP in SCB |
| SCB Ethernet 1 | 10.1.3.102 | 10GBase-T Lan port 1 in SCB |
| SCB Ethernet 2 | 10.1.4.102 | 10GBase-T Lan port 2 in SCB |
| Secondary Site ESXi Cluster Network Settings | ||
| Role | IP | Description |
| ESXi 1 Management | 192.168.201.1 | Management IP in ESXi 1 |
| ESXi 1 Ethernet1 | 10.1.3.11 | 10G Data port 1 in ESXi 1 |
| ESXi 1 Ethernet2 | 10.1.4.11 | 10G Data port 2 in ESXi 1 |
| ESXi 2 Management | 192.168.201.2 | Management IP in ESXi 2 |
| ESXi 2 Ethernet1 | 10.1.3.12 | 10G Data port 1 in ESXi 2 |
| ESXi 2 Ethernet2 | 10.1.4.12 | 10G Data port 2 in ESXi 2 |
QNAP ES1640DC Storage Configuration
- 4*200GB 2.5” SAS SSD for Cache Acceleration
- 12*1TB SAS HDD with RAID1+0 pool
- Add 4*250GB iSCSI LUNs for database, 2*64GB iSCSI LUNs for database log and SSD Cache for each LUN enabled.
ESXi Server Configuration
- Supermicro SYS-6028U-TR4T+ with Intel Xeon Processor E5-2620v3, 256GB RAM
- VMware vSphere 6.0 Update 1b (ESXi ISO Build 3380124)
Exchange Server Configuration
- Exchange 2013 drivers (ESE version 15.00.1130.002)
- Microsoft JetStress Utility (JetStress version 15.01.0318.000).
- 4*Windows 2012 R2 Virtual machines with 4 vCPU & 16 GB RAM
- Attached 4*250GB iSCSI drives for database, 2*64GB iSCSI drives for database log and formatted with 64KB allocation unit size for each Exchange server node.
- Configure VM vNIC adapter to VMXNET 3 for optimized network performance.
Solution description
Exchange Server Configuration
- Number of DAG : 1
- Number of Server per DAG : 4 (2 for primary site, 2 for secondary site)
- Number of Exchange mailboxes : 4,000 mailboxes (1,000 mailboxes per server)
- Estimated IOPS per mailbox : 0.168 (250 messages sent and received per mailbox per day)
- Size of each Mailbox : 2GB
- Number of database per Exchange server : 4 (2 active and 2 passive)
- Mailbox copies per database : 2 (2 active and 2 passive)
Note:
The Technet Exchange Sizing Guide provided the table to compute the required IOPS per mailbox user (0.168) for 250 messages sent or received per mailbox per day.
Test results summary
Storage performance results
The JetStress tests are run in parallel across four VMs with four databases per Exchange server.
After the JetStress test are concluded, we can see that one Exchange Server's Target transaction I/O is 168 per second, and the Achieved transaction I/O is around 232 per second.
Conclusion
JetStress reported that across all the Exchange Servers, the total Achieved transaction I/O in this DAG is above 1100 IOPS serving 4000 mailbox users.
Appendix – Exchange-Server1 Performance Test Results Report
Test Summary
| Overall Test Result | Pass |
| Machine Name | JETSTRESS-TEST01 |
| Test Description | |
| Test Start Time | 1/18/2016 9:58:08 AM |
| Test End Time | 1/18/2016 12:11:09 PM |
| Collection Start Time | 1/18/2016 10:11:17 AM |
| Collection End Time | 1/18/2016 12:10:54 PM |
| Jetstress Version | 15.01.0318.000 |
| ESE Version | 15.00.1130.002 |
| Operating System | Windows Server 2012 R2 Standard (6.2.9200.0) |
| Performance Log |
C:\JetStress_Report\HDD_RAID10_MP_1000_0.168_2GB(64k+SSD)\Tuning_2016_1_18_10_3_3.blg C:\JetStress_Report\HDD_RAID10_MP_1000_0.168_2GB(64k+SSD)\Performance_2016_1_18_10_11_1.blg |
Database Sizing and Throughput
| Achieved Transactional I/O per Second | 232.184 |
| Target Transactional I/O per Second | 168 |
| Initial Database Size (bytes) | 2098108301312 |
| Final Database Size (bytes) | 2098880053248 |
| Database Files (Count) | 4 |
Jetstress System Parameters
| Thread Count | 5 |
| Minimum Database Cache | 128.0 MB |
| Maximum Database Cache | 1024.0 MB |
| Insert Operations | 40% |
| Delete Operations | 20% |
| Replace Operations | 5% |
| Read Operations | 35% |
| Lazy Commits | 70% |
| Run Background Database Maintenance | True |
| Number of Copies per Database | 2 |
Database Configuration
| Instance900.1 |
Log path: H:\log1 Database: D:\Jetstress001001.edb |
| Instance900.2 |
Log path: H:\log2 Database: E:\Jetstress002001.edb |
| Instance900.3 |
Log path: I:\log3 Database: F:\Jetstress003001.edb |
| Instance900.4 |
Log path: I:\log4 Database: G:\Jetstress004001.edb |
Transactional I/O Performance
| MSExchange Database ==> Instances | I/O Database Reads Average Latency (msec) | I/O Database Writes Average Latency (msec) | I/O Database Reads/sec | I/O Database Writes/sec | I/O Database Reads Average Bytes | I/O Database Writes Average Bytes | I/O Log Reads Average Latency (msec) | I/O Log Writes Average Latency (msec) | I/O Log Reads/sec | I/O Log Writes/sec | I/O Log Reads Average Bytes | I/O Log Writes Average Bytes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Instance900.1 | 14.404 | 2.363 | 37.194 | 20.644 | 33346.816 | 36932.486 | 0.000 | 1.063 | 0.000 | 5.325 | 0.000 | 20625.184 |
| Instance900.2 | 14.178 | 2.401 | 37.150 | 20.335 | 33388.017 | 36875.070 | 0.000 | 1.085 | 0.000 | 5.254 | 0.000 | 20607.456 |
| Instance900.3 | 14.044 | 2.423 | 37.402 | 20.907 | 33347.365 | 36822.408 | 0.000 | 1.079 | 0.000 | 5.319 | 0.000 | 20732.541 |
| Instance900.4 | 13.741 | 2.363 | 37.562 | 20.991 | 33315.952 | 36779.894 | 0.000 | 1.046 | 0.000 | 5.350 | 0.000 | 20513.353 |
Background Database Maintenance I/O Performance
| MSExchange Database ==> Instances | Database Maintenance I/O Reads/sec | Database Maintenance I/O Reads Average Bytes |
|---|---|---|
| Instance900.1 | 8.993 | 261223.158 |
| Instance900.2 | 8.990 | 261295.253 |
| Instance900.3 | 8.997 | 261166.269 |
| Instance900.4 | 9.000 | 261228.101 |
Log Replication I/O Performance
| MSExchange Database ==> Instances | I/O Log Reads/sec | I/O Log Reads Average Bytes |
|---|---|---|
| Instance900.1 | 0.468 | 179529.908 |
| Instance900.2 | 0.461 | 179571.373 |
| Instance900.3 | 0.470 | 181962.563 |
| Instance900.4 | 0.465 | 180502.970 |
Total I/O Performance
| MSExchange Database ==> Instances | I/O Database Reads Average Latency (msec) | I/O Database Writes Average Latency (msec) | I/O Database Reads/sec | I/O Database Writes/sec | I/O Database Reads Average Bytes | I/O Database Writes Average Bytes | I/O Log Reads Average Latency (msec) | I/O Log Writes Average Latency (msec) | I/O Log Reads/sec | I/O Log Writes/sec | I/O Log Reads Average Bytes | I/O Log Writes Average Bytes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Instance900.1 | 14.404 | 2.363 | 46.187 | 20.644 | 77717.222 | 36932.486 | 2.539 | 1.063 | 0.468 | 5.325 | 179529.908 | 20625.184 |
| Instance900.2 | 14.178 | 2.401 | 46.141 | 20.335 | 77794.607 | 36875.070 | 2.883 | 1.085 | 0.461 | 5.254 | 179571.373 | 20607.456 |
| Instance900.3 | 14.044 | 2.423 | 46.399 | 20.907 | 77523.545 | 36822.408 | 2.720 | 1.079 | 0.470 | 5.319 | 181962.563 | 20732.541 |
| Instance900.4 | 13.741 | 2.363 | 46.562 | 20.991 | 77368.577 | 36779.894 | 2.757 | 1.046 | 0.465 | 5.350 | 180502.970 | 20513.353 |
Host System Performance
| Counter | Average | Minimum | Maximum |
|---|---|---|---|
| % Processor Time | 1.121 | 0.572 | 3.695 |
| Available MBytes | 14252.779 | 14203.000 | 14292.000 |
| Free System Page Table Entries | 16485011.017 | 16482346.000 | 16490963.000 |
| Transition Pages RePurposed/sec | 0.000 | 0.000 | 0.000 |
| Pool Nonpaged Bytes | 117907960.518 | 93839360.000 | 127930368.000 |
| Pool Paged Bytes | 164860169.086 | 164831232.000 | 164884480.000 |
| Database Page Fault Stalls/sec | 0.000 | 0.000 | 0.000 |
Test Log 1/18/2016 9:58:08 AM -- Preparing for testing ...
1/18/2016 9:58:12 AM -- Attaching databases ...
1/18/2016 9:58:12 AM -- Preparations for testing are complete.
1/18/2016 9:58:12 AM -- Starting transaction dispatch ..
1/18/2016 9:58:12 AM -- Database cache settings: (minimum: 128.0 MB, maximum: 1.0 GB)
1/18/2016 9:58:12 AM -- Database flush thresholds: (start: 10.2 MB, stop: 20.5 MB)
1/18/2016 9:58:17 AM -- Database read latency thresholds: (average: 20 msec/read, maximum: 100 msec/read).
1/18/2016 9:58:17 AM -- Log write latency thresholds: (average: 10 msec/write, maximum: 100 msec/write).
1/18/2016 9:58:17 AM -- Attaining prerequisites:
1/18/2016 10:03:03 AM -- \MSExchange Database(JetstressWin)\Database Cache Size, Last: 968945700.0 (lower bound: 966367600.0, upper bound: none)
1/18/2016 10:03:04 AM -- Performance logging started (interval: 5000 ms).
1/18/2016 10:03:04 AM -- Automatic tuning is starting ...
1/18/2016 10:03:36 AM -- Instance900.1 has 0.00 for read latency slope.
1/18/2016 10:03:36 AM -- Instance900.2 has 0.01 for read latency slope.
1/18/2016 10:03:36 AM -- Instance900.3 has 0.02 for read latency slope.
1/18/2016 10:03:36 AM -- Instance900.4 has 0.03 for read latency slope.
1/18/2016 10:05:43 AM -- 135 batch transactions/sec and 8 sessions have 270 IOPS.
1/18/2016 10:05:43 AM -- 8 sessions have actual 270 IOPS (target IOPS: 168)
1/18/2016 10:05:43 AM -- Instance900.1 has 20.3 for I/O Database Reads Average Latency.
1/18/2016 10:05:43 AM -- Instance900.2 has 20.1 for I/O Database Reads Average Latency.
1/18/2016 10:05:43 AM -- Instance900.3 has 19.7 for I/O Database Reads Average Latency.
1/18/2016 10:05:43 AM -- Instance900.4 has 19.6 for I/O Database Reads Average Latency.
1/18/2016 10:05:43 AM -- The process has average database read latencies higher than 20.000 msec.
1/18/2016 10:05:43 AM -- Instance900.1 has 2.0 for I/O Log Writes Average Latency.
1/18/2016 10:05:43 AM -- Instance900.1 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:05:43 AM -- Instance900.2 has 2.0 for I/O Log Writes Average Latency.
1/18/2016 10:05:43 AM -- Instance900.2 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:05:43 AM -- Instance900.3 has 1.5 for I/O Log Writes Average Latency.
1/18/2016 10:05:43 AM -- Instance900.3 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:05:43 AM -- Instance900.4 has 1.4 for I/O Log Writes Average Latency.
1/18/2016 10:05:43 AM -- Instance900.4 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:05:43 AM -- Operation mix: Sessions 4, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%.
1/18/2016 10:06:15 AM -- Instance900.1 has 0.07 for read latency slope.
1/18/2016 10:06:15 AM -- Instance900.2 has 0.03 for read latency slope.
1/18/2016 10:06:15 AM -- Instance900.3 has 0.13 for read latency slope.
1/18/2016 10:06:15 AM -- Instance900.4 has 0.06 for read latency slope.
1/18/2016 10:08:22 AM -- 76 batch transactions/sec and 4 sessions have 166 IOPS.
1/18/2016 10:08:22 AM -- 4 sessions have actual 166 IOPS (target IOPS: 168)
1/18/2016 10:08:22 AM -- Instance900.1 has 15.5 for I/O Database Reads Average Latency.
1/18/2016 10:08:22 AM -- Instance900.2 has 15.3 for I/O Database Reads Average Latency.
1/18/2016 10:08:22 AM -- Instance900.3 has 15.7 for I/O Database Reads Average Latency.
1/18/2016 10:08:22 AM -- Instance900.4 has 15.1 for I/O Database Reads Average Latency.
1/18/2016 10:08:22 AM -- Instance900.1 has 1.2 for I/O Log Writes Average Latency.
1/18/2016 10:08:22 AM -- Instance900.1 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:08:22 AM -- Instance900.2 has 1.1 for I/O Log Writes Average Latency.
1/18/2016 10:08:22 AM -- Instance900.2 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:08:22 AM -- Instance900.3 has 0.8 for I/O Log Writes Average Latency.
1/18/2016 10:08:22 AM -- Instance900.3 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:08:22 AM -- Instance900.4 has 0.9 for I/O Log Writes Average Latency.
1/18/2016 10:08:22 AM -- Instance900.4 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:08:22 AM -- Operation mix: Sessions 5, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%.
1/18/2016 10:08:53 AM -- Instance900.1 has 0.08 for read latency slope.
1/18/2016 10:08:53 AM -- Instance900.2 has 0.20 for read latency slope.
1/18/2016 10:08:53 AM -- Instance900.3 has 0.04 for read latency slope.
1/18/2016 10:08:53 AM -- Instance900.4 has 0.16 for read latency slope.
1/18/2016 10:11:01 AM -- 96 batch transactions/sec and 5 sessions have 196 IOPS.
1/18/2016 10:11:01 AM -- 5 sessions have actual 196 IOPS (target IOPS: 168)
1/18/2016 10:11:01 AM -- Instance900.1 has 15.9 for I/O Database Reads Average Latency.
1/18/2016 10:11:01 AM -- Instance900.2 has 15.2 for I/O Database Reads Average Latency.
1/18/2016 10:11:01 AM -- Instance900.3 has 15.2 for I/O Database Reads Average Latency.
1/18/2016 10:11:01 AM -- Instance900.4 has 15.4 for I/O Database Reads Average Latency.
1/18/2016 10:11:01 AM -- Instance900.1 has 1.2 for I/O Log Writes Average Latency.
1/18/2016 10:11:01 AM -- Instance900.1 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:11:01 AM -- Instance900.2 has 1.2 for I/O Log Writes Average Latency.
1/18/2016 10:11:01 AM -- Instance900.2 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:11:01 AM -- Instance900.3 has 1.5 for I/O Log Writes Average Latency.
1/18/2016 10:11:01 AM -- Instance900.3 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:11:01 AM -- Instance900.4 has 1.3 for I/O Log Writes Average Latency.
1/18/2016 10:11:01 AM -- Instance900.4 has 0.0 for I/O Log Reads Average Latency.
1/18/2016 10:11:01 AM -- Performance logging has ended.
1/18/2016 10:11:01 AM -- Automatic tuning succeeded.
1/18/2016 10:11:02 AM -- Operation mix: Sessions 5, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%.
1/18/2016 10:11:02 AM -- Performance logging started (interval: 15000 ms).
1/18/2016 10:11:02 AM -- Attaining prerequisites:
1/18/2016 10:11:02 AM -- \MSExchange Database(JetstressWin)\Database Cache Size, Last: 1050984000.0 (lower bound: 966367600.0, upper bound: none)
1/18/2016 12:11:03 PM -- Performance logging has ended.
1/18/2016 12:11:03 PM -- JetInterop batch transaction stats: 22851, 22851, 22850 and 22850.
1/18/2016 12:11:03 PM -- Dispatching transactions ends.
1/18/2016 12:11:03 PM -- Shutting down databases ...
1/18/2016 12:11:09 PM -- Instance900.1 (complete), Instance900.2 (complete), Instance900.3 (complete) and Instance900.4 (complete)
1/18/2016 12:11:09 PM -- C:\JetStress_Report\HDD_RAID10_MP_1000_0.168_2GB(64k+SSD)\Performance_2016_1_18_10_11_1.blg has 479 samples.
1/18/2016 12:11:09 PM -- Creating test report ...
1/18/2016 12:11:13 PM -- Instance900.1 has 14.4 for I/O Database Reads Average Latency.
1/18/2016 12:11:13 PM -- Instance900.1 has 1.1 for I/O Log Writes Average Latency.
1/18/2016 12:11:13 PM -- Instance900.1 has 1.1 for I/O Log Reads Average Latency.
1/18/2016 12:11:13 PM -- Instance900.2 has 14.2 for I/O Database Reads Average Latency.
1/18/2016 12:11:13 PM -- Instance900.2 has 1.1 for I/O Log Writes Average Latency.
1/18/2016 12:11:13 PM -- Instance900.2 has 1.1 for I/O Log Reads Average Latency.
1/18/2016 12:11:13 PM -- Instance900.3 has 14.0 for I/O Database Reads Average Latency.
1/18/2016 12:11:13 PM -- Instance900.3 has 1.1 for I/O Log Writes Average Latency.
1/18/2016 12:11:13 PM -- Instance900.3 has 1.1 for I/O Log Reads Average Latency.
1/18/2016 12:11:13 PM -- Instance900.4 has 13.7 for I/O Database Reads Average Latency.
1/18/2016 12:11:13 PM -- Instance900.4 has 1.0 for I/O Log Writes Average Latency.
1/18/2016 12:11:13 PM -- Instance900.4 has 1.0 for I/O Log Reads Average Latency.
1/18/2016 12:11:13 PM -- Test has 0 Maximum Database Page Fault Stalls/sec.
1/18/2016 12:11:13 PM -- The test has 0 Database Page Fault Stalls/sec samples higher than 0.
1/18/2016 12:11:13 PM -- C:\JetStress_Report\HDD_RAID10_MP_1000_0.168_2GB(64k+SSD)\Performance_2016_1_18_10_11_1.xml has 478 samples queried.