How would you design the network for iSCSI storage?
Show the answer
Say this first
Two independent paths, each with its own VLAN and subnet, on separate host NICs and ideally separate switches, with multipathing (MPIO) on the host instead of LACP. If I use jumbo frames, they are set end to end: host, every switch port and the array. Initiators and targets share a layer 2 segment so storage is not routed or firewalled. Host and array ports are edge ports, and I monitor latency, drops and TCP retransmits.
Go deeper
Layout
| Path | VLAN | Subnet | Host NIC | Array ports |
|---|---|---|---|---|
| A | 101 | 10.101.0.0/24 | ens224 | Controller A port 1, controller B port 1 |
| B | 102 | 10.102.0.0/24 | ens256 | Controller A port 2, controller B port 2 |
Why MPIO and not LACP
LACP hashes one iSCSI session onto one link, so it adds no per-session throughput and fails over at the link layer without knowing the array. MPIO sees every path, balances I/O across them and fails over with awareness of the array's controllers. Most storage vendors require MPIO for iSCSI.
MTU
9000 on hosts and array, switch ports at 9216 or the platform maximum. Test each host NIC with
ping -M do -s 8972 <target>. A mismatch shows as sessions that log in fine and then stall on large reads.
Other points
- Avoid routing and firewall inspection between initiator and target. Both add latency and session drops.
- Edge ports (PortFast) on host and array ports so a topology change does not stall storage.
- Follow the array vendor on Ethernet flow control. Recommendations differ.
- Dedicated switches, or QoS when shared. Many hosts reading at once cause microbursts.
They will ask next
Why keep the two paths in different subnets?
With both NICs in one subnet, the host's routing table picks one NIC for everything and the second path carries nothing useful. Separate subnets force each session onto its own NIC and switch.
Traps
- Enabling jumbo frames on the hosts and the array but missing one switch port in the middle.