RoCE/RDMA/DCB what is it and how to configure it

You have probably heard these acronyms somewhere, so what are these and are they the same. In short yes and no

RoCE stands for RDMA over Converged Ethernet, the RDMA part is Remote Direct Memory Access.

RDMA allows for network data(TCP packets) to be offloaded on the Network cards and put directly in to the memory, bypassing the hosts CPU. Allowing for the host to have all the access to the CPU. In normal TCP offload all the network traffic goes trough the CPU and with higher speeds will take more CPU. On a 10gbit network it would take about 100% cpu on a 12 core Intel Xeon V4 CPU.

Mellanox has a good explanation for RDMA here.

DCB stands for Data Center Bridging

What it contains are enhancements to Ethernet communication protocol. Ethernet is a best-effort network that may experience packet loss when network devices are busy, creating re transmission. DCB allows for selected traffic to have zero packet loss. It eliminates loss due to queue overflow and to be able to allocate bandwidth on links. DCB allows for different priorities of packets being sent over the network.

 

In this post i will cover how to enable RDMA and DCB in Windows for SMB and on different switches. I will update with more switches as i read trough different vendors configuration. As the setup varies a lot from vendor to vendor.

Switches and Vendors that is covered in this post

Dell

N4000 series
Force 10 S4810p, S6000, S6000-on(FTOS)

Cisco

Nexus NX-OS

Mellanox

SN2100

HPE

FlexFabric 5700

Quanta

LB8

 

How to configure Windows Server 2012, 2012R2 and 2016 with RDMA and DCB

For SMB you will need to install WindowsFeauture Data-Center-Bridging

Reboot the server and let’s configure the DCB settings. SMB always use Priority 3, you can use any other, but best practice is 3.

After the QOS part is done, let’s configure a network team or a switch. For S2D one uses a setswitch with Embededteaming

Let’s create some network cards and enable RDMA on them. Once RDMA is enabled DCB will also be enabled for SMB.

To check if RDMA is enabled you can run this command

Now DCB and RDMA is configured in Windows, let’s move to the switch setup.

 

This is where the hard part is, figuring out the correct setup for your switch. Most switch vendors support this.

Dell N4000 series

Turn off flowcontrol on all interfaces.

What you set here is that we have sett traffic class 3 into group 0, and we have set max and min bandwith on the groups. The groups are 0,1,2. This gives max bandwith for group 0 and 1 50% each.  Then we enable the DCB config on the interfaces with mode on. And with priority 3 no-drop we enable the no packet drop on the traffic class 3.

Dell Force 10 S4810p

Turn off flowcontrol on all interfaces.

Dell Force 10 S6000, S6000-On(FTOS)

Turn off flowcontrol on all interfaces.

Cisco Nexus NX-OS

By default PFC(Priority Flow Control) is enabled on Cisco Nexus switches. To hard enable it do the following.

Mellanox SN2100

 

HPE FlexFabric 5700

Quanta

This is the basic how to enable, not had the chance to test this out my self yet. So this will be updated as the manual is not straight forward.

 

2 thoughts on “RoCE/RDMA/DCB what is it and how to configure it

  • June 29, 2017 at 6:04 pm
    Permalink

    Just a note, I believe your Dell Force 10 S4810 config is slightly off. You are marking Priority 4 not Priority 3 with the current command of “priority-pgid 1 1 1 1 0 1 1 1”

    Reply
    • June 30, 2017 at 10:29 am
      Permalink

      Thanks you for pointing that out, you are absolutely correct 🙂 I write mistake from my side.

      Thanks
      JT

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *