VxRail 7.0.13x – Networking Flexibility

One of the biggest changes we’ve added to the VxRail 7.0 code release is around the networking. In previous VxRail releases, we could only guarantee the Network Daughter Card (NDC) would be installed in the system. That lead us to only configuring the NDC for “System Traffic” (Management, vSAN, vMotion) and then the PCIe NICs that were added could be used VM traffic, if necessary. On top of that, VxRail would create a dedicated DVS for each cluster (regardless of vCenter deployment architecture). Some of these deployment options didn’t fully align with how some customers preferred their network layout. Based upon that feedback and some changes that were made to the VxRail Manager architecture starting with 7.0.010, we were able to meet most options now. Here’s a quick summary of the changes to the network layout that have been added. I will discuss each change below.

For each of these changes, you can view the setup requirements in the VxRail Networking Guide.

Network Changes:

• Customer supplied VDS (leverage an existing VDS in a customer’s environment)

• NDC+PCIe NIC (ability to deploy the cluster using a port from the NDC and PCIe NICs for system traffic)

• PCIe NIC Only (ability to deploy the cluster with only PCIe NICs for system traffic)

• LAG (ability to support LAG with VxRail for vSAN traffic)

• 2 VDS (ability to deploy with 2 VDSs, providing for physical and logical traffic isolation)

Customer supplied VDS This was something added into VxRail version 7.0.010 and started the foundation for the network flexibility in the subsequent releases. We have had some customers ask that they deploy the VxRail not only against their existing vCenter, but also against an existing VDS in their environment. Prior to this change, our larger customers were limited to 128 VxRail clusters per vCenter instance due to the max number of VDSs per vCenter being 128. This feature addition allows those customers to increase the number of VxRail nodes per cluster. It also allows them to have more than a single cluster to a single VDS. One note here though is that this does require you leverage your existing vCenter.

NDC+PCIe NIC As mentioned above, VxRail used to only leverage the NDC ports for the VxRail system traffic. This lead to some concerns around NIC level redundancy and being able to leverage ports from multiple NICs to achieve this. Starting with 7.0.100, we added in the ability to leverage a combination of the NDC and PCIe NICs for the VxRail system traffic. This setting can be accomplished during the deployment of the system, or after you upgrade to VxRail 7.0.1xx, assuming you have the PCIe NICs in the system. When doing this, it’s important that you cable the system correctly. Similar to this picture:

The cabling in the example above will now allow you to take the system traffic and spread it across both the NDC and PCIe NIC. Here’s an example of how the system could be configured:

As you can see above, we now have vMotion and vSAN utilizing one port from the NDC and PCIe NIC as well as the Management and vCenter Server networks doing the same. If you wanted, you could also isolate vSAN to a dedicated pair of NIC ports and move vMotion over the same ports as Management and vCenter Server.

PCIe NIC Only After we were able to achieve spreading the networks across both the PCIe and NDC NICs, the next step was to remove the NDC from the deployment requirements completely. This was brought to market in 7.0.13x release. For those customers that want to use only PCIe NICs for their deployment, we can do that now. With this option, you can choose to isolate the traffic to only the PCIe NICs and not even configured the NDC ports for system traffic.

This option may be useful for customers that require a specific NIC that’s available only as a PCIe option and not on the NDC. One area this would help would be with 10GbE cards as we only offer 4x10GbE on the NDC, but there are 2x10GbE options for the PCIe NICs. This would allow the same port density, but utilize 2 NICs instead of just a single NIC, helping to provide the NIC level redundancy.

You also have the option of moving all traffic to PCIe NICs post deployment (or upgrade) to VxRail 7.0.130.

Please note, at this time, we have not validated using only 100GbE PCIe NICs for deployment of VxRail.

LAG Support Support for link aggregation is a request that seems to be asked from time to time. I don’t hear of it often, but it is something our Product Management team has been tracking for a while. There were some instances where this was being requested per customer standards and others where customers believed their workload could saturate the vSAN network. Starting with 7.0.130, we have added this support. If you look at the diagram below, we are using LAG for the vMotion and vSAN networks. Currently, we only support LAG for non-management VDS portgroups, meaning we can’t use LAG on the Management, vCenter and VxRail private portgroups. To leverage the LAG functionality, it is required that we have 4 physical network ports available.

There are some considerations with regards to LAG with VxRail and deploying the cluster. We can deploy a cluster with LAG enabled during the deployment if we are using a customer supplied VDS as described in the first feature description. In that scenario, the VDS must already exist and have LAG configured on the existing customer supplied vCenter. You must also make sure that the physical switch are configured with LAG support.

For environments where the VxRail deployed VDS is in use, the support for LAG is a Day 2 (post deployment) operation. This configuration change is supported on systems that were deployed at VxRail 7.0.13x or systems that have been upgraded to 7.0.13x. If you have upgraded your system but only have 2 physical network ports, it is possible to upgrade your node with additional NICs to achieve the requirement of 4 ports.

With VxRail support for LAG, it does support both static and dynamic LAG. More information on static and dynamic LAG support are in the VxRail Networking Guide.

2 VDS Deployment The last change that we’ve made is for the VxRail to deploy a pair of VDSs to help with both physical and logical isolation of the network traffic. There are a couple ways this may be implemented. First, in the picture below, we show the ability to take the NDC and PCIe NICs along with 2 VDSs. This will help isolate both the management and non-management traffic to dedicated VDSs as well as top of rack (TOR) switches. Using 2 pairs or TORs has been something customers have been doing with a single VDS for a while to help with the physical separation. That would allow them to keep what some refer to as “front end” and “back end” traffic separate. One thing that could be slightly different below is the “Guest VM” port group may reside on either of the 2 VDSs below.

We can take the concept above even further by adding in additional NIC ports to the mix. This could be met by either using 4 port cards or by adding in additional 2 port cards to the PCIe slots. VxRail has always supported the ability to leverage any unused NIC ports for VM level traffic. The challenge in the past would be around the NDC ports and that VxRail would always deploy/configure only the NDC ports. In the diagram below, you can see a full build out of the complete flexibility you now have with VxRail during the build (or after an upgrade to 7.0.13x)

In the above diagram, the management and non-management VDS would be deployed and configured during the install. After the deployment is completed, you could then create the production VDS to be used for the VM level traffic. There would also be the option to leverage LAG on the non-management VDS, if you choose. The ability to deploy a VxRail with 2 VDSs is achieved via API today, along with the VxRail configuration portal. Cliff Cahill recently wrote a post about this, summarizing the steps required.

I’m very excited for all the networking changes that we’ve added in the VxRail 7.0.x release. One of the requests I most often is asked about is what flexibility can we have with the VDS and network configuration. These latest changes allow us to provide our customers with the flexibility required to meet all the requests I have heard over the years with VxRail.

Read more from Jeremy on his blog: Inside The (Vx)Rails