Intel Optane Gen 2 in VxRail 15G

When Dell Technologies releases a new Generation of PowerEdge Servers, there is an overlap in the availability of the previous generation. As of the 1st half of 2021, Dell has released its 15th Generation of PowerEdge Servers which have introduced a number of new and exciting bells and whistles, but the one in particular that we will be covering today is the introduction of Intel Optane Gen 2, or Series 200 as it may be referred to in some context.

If you have not read my first blog on Intel Optane in VxRail, you should start here for the foundational information on what Optane is and how its leveraged on VxRail. That will help you understand more of what I am discussing in this post which is an overview of Intel Optane Gen 2 on VxRail 15G Nodes. I will wait here until you're done reading then we will continue!

OK! Let's get into it.

Upfront I want to mention the prerequisites to leverage Intel Optane Series 200 (PMem and SSDs) on VxRail.

1. VxRail 15G Nodes (E660/F, P670F, V670F) are required.

2. Both CPU sockets must be populated.

3. vSphere 7.0U2 or later versions (VxRail 7.0.210<) required.

There are some massive gains in density with the new 15G line up leveraging the horsepower of the Intel Sky Lake chipset that we are really excited about! Here is a comparison of the Optane PMem 100 Series and the 200 Series:

There is a 25% increase in PMem capacity and a 32% increase in bandwidth with Series 200! On top of that, if you use Optane PMem you can squeeze out an additional 50% memory capacity to total 6TB per socket (12TB per host)!!!

RABBIT TRAIL ALERT!!!!! I'd like to call something out here worth mentioning. I also want to tell you that there is no shame whatsoever that come along with the statements I am about to make, this is for awareness and me getting my own observations out there. Also, if you've worked with me you know the genuine care that I take with all customers I work with. Most customers are pretty conservative with their approach to oversubscription. Maybe for some, they are skeptical on performance impacts if they load a box up. Or some have a methodical approach to scale and planning so they lock in virtual to physical ratios. Some are memory bound. Perhaps a good bit of those customers just don’t really think about it. Regardless of which scenario you find yourself in, typically there is a lot of resource left on the table. This could be an extra node(s) worth of resources that you are paying for, but is completely untapped. Dollars have been locked up and we throw away the keys. On average, we see customers (most, but not all) running no more than about 50 VMs per host and are anywhere between 2-3:1 vCPU to pCore. CPU never touches above 30% peak and you've backed yourself in a corner with how you've populated memory so in order to scale you have to add yet another node…sound familiar? You're not alone! pCPU cores and memory density has exploded in its density capabilities and if done correctly, the network can support it. What I am trying to say here is that there is no reason we shouldn’t consider increasing the number of VMs per host. There is no reason that a large database workloads shouldn’t be converted from physical to virtual. There is no reason we should be paying valuable dollars for resources that are sitting there doing nothing for you. *END RANT*

System Board and Memory Layout

With Ice Lake there are inherent changes in the system board layouts which are worth noting and being aware of. Understanding this will help you plan for the present and the future.

Just like every other Intel chipset architecture in the past you will have some single socket systems and some dual socket systems. The Dell Technologies PowerEdge portfolio is ripe with multiple formfactor options, but we do not offer all of them in our VxRail lineup. VxRail nodes are only offered with dual socket system boards. You may know that we offer a subset of VxRail node configurations that support single socket population. There are pros and cons to populating a single socket , but if you are configuring Optane Memory, or Optane SSDs, you MUST have BOTH sockets populated. This allows the system to leverage the fullest capacity and performance capability.

Here is a look at the system board in our 15G node offerings leveraging Intel Sky Lake CPUs. What you will notice is that there are 32 DIMM slots between both sockets which is 8 more memory DIMM slots than in the previous 14G nodes based on Cascade Lake. Still, half (16) of the DIMMS are allocated to CPU1 and the other half to CPU2, thus both sockets must be populated with a CPU to leverage all 32 memory DIMMs.

Another difference in the Ice Lake based board layout is that there are a total of 8 memory channels per CPU, each with 2 DPC (DIMMs per channel). In Cascade Lake, there were 6 channels per socket, each with 2 DPC. Here is a table showing the slot to channel to processor alignment.

This is important because there are guidelines in the use of Intel Optane Pmem that need to be considered when architecting the systems to use this technology. You can refer to the Intel Persistent Memory 200 Series (BPS) Installation Guidelines for a complete list, but I will only cover some of them in this post.

The one I will call out first and foremost is that just like in Cascade Lake, Ice Lake still limits 1 DPC of Optane PMem. At most, you can have half of your total DIMM slots populated with PMem (16 DIMMs total). However, the size of the Optane PMem modules have doubled in size going from 256GB per stick to 512GB. Fully popped PMem slots will total the 4TB per socket, 8TB per dual socket system. WOW! Then, you still have 16 total DIMM slots available for DRAM which offers the max DIMM size of 256GB using LRDIMMs which gives you 50% more memory to total 12TB per host. INSANE, right?! I said it in my last Optane post and I will say it again:

"The 2 are converging - Memory and Storage. I think eventually we will be leveraging hosts that have nothing but Memory DIMMs for storage capacity!"

I know we aren't there yet and we are still a ways away from hitting over 100TB of capacity per node in memory alone but we will get there!

Optane Series 200 PMem DIMMs come in 3 sizes: 128GB, 256GB and 512GB.

Optane PMem Modes

You still have the 2 modes to choose from, which I covered in my first Optane post. Here is another look at what those modes are.

If you're looking for a lower cost per/GB than DRAM with more density and higher performance than SSDs, Memory Mode is your mode. If you are looking for maximum application performance and in-memory storage for high performance analytical workloads, App Direct Mode is your mode. Maybe you have use for both, which is great!

NOTE: You cannot mix the 2 modes on a host, and furthermore it would be a best practice to keep the operating modes consistent within the cluster to avoid issues that could present themselves during planned or unplanned node outages, etc.

Something to be excited about on VMware's awareness of Persistent Memory is this new feature that is new in vSphere 7.0U3:

vSphere Memory Monitoring and Remediation (vMMR)

Memory metrics now available on vCenter UI!

• Provides insights and troubleshooting on host and VM-level statistics for both PMem and DRAM in the Advanced Memory section for Performance on the UI

• Create custom alerts that can trigger alarms for desired threshold levels.

• Supported for Intel Cascade Lake and Ice Lake processors

This is such a great feature and I am happy to see that there will be more memory metrics visible in vCenter.

Optane SSDs Now, to talk a bit about the Series 200 SSDs. There are MASSIVE gains in performance and durability with the Optane P5800X. Had to do that in large font size to really get the point across - haha! Seriously though….let's take a look.

Here are the specs of the P5800X:

If we compare this against the Series 100 Optane SSDs (D4800X) we are seeing a 3x increase in 4K Random Read IOPS and between 2-3X 4K Random Write IOPS, over 2x 100% Sequential Read and Write bandwidth, and the endurance rating is 2-5X depending on the drive capacity of the Gen 1 SSDs.

In VxRail 15G, the Cache drive options include the Optane Series 200 P5800X 400 and 800GB options. You can configure up to (2) Optane Cache drives in the 1U E660/F and up to (4) in the 2U P670F and V670F.

Intel Optane Series 200 coupled with PCIe 4.0 presents a huge performance uplift that not only caters itself to those workloads that are high bandwidth, extremely high IO, but also increasing the life of your hardware and doing more with less.