Home | Software-Defined | Storage Spaces Direct | Configure Dell S4048 switches for Storage Spaces Direct

Configure Dell S4048 switches for Storage Spaces Direct

When we deploy Storage Spaces Direct (S2D), either hyperconverged or disaggregated, we have to configure the networking part. Usually we work with Dell hardware to deploy Storage Spaces Direct and the one of the switches supported by the Dell reference architectures is the S4048 (Force 10). In this topic, we will see how to configure this switch from scratch.

This topic has been co-written with Romain Serre – SCC architect solution.

Stack or not Stack

Usually, customers know the stack feature which is common to all network vendors such as Cisco, Dell, HP and so on. This feature enables to add several identical switches in a single configuration managed by a master switch. Because all switches share the same configuration, for the network administrators, all these switches are seen like a single one. So, the administrators connect to the master switch and then edit the configuration on all switches member of the stack.

If the stacking is sexy on the paper, there is a main issue especially with storage solution such as S2D. With S4048 stack, when you run an update, all switches reload at the same time. Because S2D highly relies on the network, your storage solution will crash. This is why the Dell reference architecture for S2D recommends to deploy a VLT (Virtual Link Trunking).

With Stacking you have a single control plane (you configure all switches from a single switch) and a single data plane in a loop free topology. In a VLT configuration, you have also a single data plane in a loop free topology but several control planes, which allow you to reboot switches one by one.

For this reason, the VLT (or MLAG) technology is the preferred way for Storage Spaces Direct.

S4048 overview

A S4048 switch has 48x 10GB/s SFP+ ports, 6x 40GB/s QSFP+ ports, a management port (1GB/s) and a serial port. The management and the serial ports are located on the back. In the below diagram, there is three kinds of connection:

  • Connection for S2D (in this example from port 1 to 16, but you can connect until port 48)
  • VLTi connection
  • Core connection: the uplink to connect to core switches

In the below architecture schema, you can find both S4048 interconnected by using VLTi ports and several S2D nodes (hyperconverged or disaggregated, that doesn’t matter) connected to port 1 to 16. In this topic, we will configure these switches regarding this configuration.

Switches initial configuration

When you start the switch for the first time you have to configure the initial settings such as switch name, IP address and so on. Plug a serial cable from the switch to your computer and connect through Telnet with the following settings:

  • Baud Rate: 115200
  • No Parity
  • 8 data bits
  • 1 stop bit
  • No flow control

Then you can run the following configuration

Enable

Configure

# Configure the hostname

hostname SwitchName-01

# Set the IP address to the management ports, to connect to switch through IP

interface ManagementEthernet 1/1

ip address 192.168.1.1/24

no shutdown

exit

# Set the default gateway

ip route 0.0.0.0/0 192.168.1.254/24

# Enable SSH

ip ssh server enable

# Create a user and a password to connect to the switch

username admin password 7 MyPassword privilege 15

# Disable Telnet through IP

no ip telnet server enable

Exit

# We leave enabled Rapid Spanning Tree Protocol.

protocol spanning-tree rstp

no disable

Exit

Exit

# Write the configuration in memory

Copy running-configuration startup-configuration

After this configuration is applied, you can connect to the switch through SSH. Apply the same configuration to the other switch (excepted the name and IP address).

Configure switches for RDMA (RoCEv2)

N.B: For this part we assume that you know how RoCE v2 is working, especially DCB, PFC and ETS.

Because we implement the switches for S2D, we have to configure the switches for RDMA (RDMA over Converged Ethernet v2 implementation). Don’t forget that with RoCE v2, you have to configure DCB and PFC end to end (on servers and on switches side). In this configuration, we assume that you use the Priority ID 3 for SMB traffic.

By default the queue value is 0 for all dot1p (QoS) traffic. We enable this command globally to change this behavior.

service-class dynamic dot1p

# Data-Center-Bridging enabled. This enable to configure Lossless and latency sensitive traffic in a Priority Flow Control (PFC) queue.

dcb enable

# Provide a name to the DCB buffer threshold

dcb-buffer-threshold RDMA

priority 3 buffer-size 100 pause-threshold 50 resume-offset 35

exit

# Create a dcb map to configure the PFC and ETS rule (Enhanced Transmission Control)

dcb-map RDMA

# For priority group 0, we allocate 50% of the bandwidth and PFC is disabled

priority-group 0 bandwidth 50 pfc off

# For priority group 3, we allocate 50% of the bandwidth and PFC is enabled

priority-group 3 bandwidth 50 pfc on

# Priority group 3 contains traffic with dot1p priority 3.

priority-pgid 0 0 0 3 0 0 0 0

Exit

Exit

Copy running-configuration startup-configuration

Repeat this configuration on the other switch.

VLT domain implementation

First of all, we have to create Port Channel with two QSFP+ ports (port 1/49 and 1/50):

# Configure the port-channel 100 (make sure it is not used)

interface Port-channel 100

# Provide a description

description VLTi

# Do not apply an IP address to this port channel

no ip address

#Set the maximum MTU to 9216

mtu 9216

# Add port 1/49 and 1/50

channel-member fortyGigE 1/49,1/50

# Enable the port channel

no shutdown

Exit

Repeat this configuration on the second switch Then we have to create the VLT domain and use this port-channel. Below the configuration on the first switch:

# Configure the VLT domain 1

vlt domain 1

# Specify the port-channel number which will be used by this VLT domain

peer-link port-channel 100

# Specify the IP address of the other switch

back-up destination 192.168.1.2

# Specify the priority of each switch

primary-priority 1

# Give an used MAC address for the VLT

system-mac mac-address 00:01:02:01:02:05

# Give an ID for each switch

unit-id 0

# Wait be 10s before the configuration saved is applied

delay-restore 10

Exit

Exit

Copy Running-Configuration Startup-Configuration

On the second switch, the configuration looks like this:

# Configure the VLT domain 1

vlt domain 1

# Specify the port-channel number which will be used by this VLT domain

peer-link port-channel 100

# Specify the IP address of the other switch

back-up destination 192.168.1.1

# Specify the priority of each switch

primary-priority 2

# Give an used MAC address for the VLT

system-mac mac-address 00:01:02:01:02:05

# Give an ID for each switch

unit-id 1

# Wait be 10s before the configuration saved is applied

delay-restore 10

Exit

Exit

Copy Running-Configuration Startup-Configuration

Now the VLT is working. You don’t have to specify VLAN ID on this link. The VLT manage itself tagged and untagged traffic.

S2D port configuration

To finish the switch configuration, we have to configure ports and VLAN for S2D nodes

Interface range Ten 1/1-1/16

# No IP address assigned to these ports

no ip address

# Enable the maximum MTU to 9216

mtu 9216

# Enable the management of untagged and tagged traffic

portmode hybrid

# Enable switchport Level 2 and this port is added to default VLAN to send untagged traffic.

Switchport

# Configure the port to Edge-Port

spanning-tree 0 portfast

# Enable BPDU guard on these port

spanning-tree rstp edge-port bpduguard

#Apply the DCB policy to these port

dcb-policy buffer-threshold RDMA

# Apply the DCB map to this port

dcb-map RDMA

# Enable port

no shutdown

Exit

Exit

Copy Running-Configuration Startup-Configuration

You can copy this configuration to the other switch. Now just VLAN are missing. To create VLAN and assign to port you can run the following configuration:

Interface VLAN 10

Description “Management”

Name “VLAN-10”

Untagged TenGigabitEthernet 1/1-1/16

Exit

Interface VLAN 20

Description “SMB”

Name “VLAN-20”

tagged TenGigabitEthernet 1/1-1/16

Exit

[etc.]

Once you have finished, copy this configuration on the second switch.

Et Voilà !

About Frederic Stefani

Frederic Stefani works as a Solutions Architect at Dell EMC. Frederic is focused on Microsoft Technologies, especially Microsoft Cloud Solutions based on Microsoft System Center, Microsoft Virtualization and Microsoft Azure. This includes Microsoft Hyper-V, Windows Server, Storage, Networking and Azure Pack as well as Service Management Automation. Frederic works closely with Customers and Microsoft to promote Microsoft Technology as speaker for Dell EMC, Microsoft and other Technical events such as Microsoft Experiences or Dell events. Frederic also works ahead of sales with our customers to promote Dell EMC services Specialities: Cloud, Datacenter, Microsoft Azure, Microsoft Azure Stack, Windows Server, Hyper-V, Windows Clients, Disaster Recovery, Powershell, Windows Deployment Services, Windows Unattended Installations, Microsoft Hosting Solutions, Powershell Development, SCVMM, System Center, Datacenter Management, Storage, Networking, Software Defined Datacenter. P-Seller (Partner Seller) within the competences: Core Infrastructure and Azure for Microsoft France

Leave a Reply

%d bloggers like this: