Clustering – Exit | the | Fast

Hyper-V Design for NUMA Architecture and Alignment

I’m publishing this design topic separately for both Hyper-V and vSphere, assuming that the audience will differ between the two. Some of this content will be repeated between posts but arranged to focus on the hypervisor each prospective reader will likely care most about.
Be sure to also check out: vSphere Design for NUMA Architecture and Alignment (coming soon)
Non-Uniform Memory Access (NUMA) has been with us for awhile now and was created to overcome the scalability limits of the Symmetric Multi-Processing (SMP) CPU architecture. In SMP, all memory access was tied to a singular shared physical bus. There are obvious limitations in this design, especially with higher CPU counts and increased traffic on the bus.

NUMA was created to limit the number of CPUs tied to a single memory bus and as a result defines what is construed as a NUMA node. A high performing NUMA node is defined by the memory directly attached to a physical CPU socket accessed directly by consuming applications and services.

NUMA spanning occurs when memory is accessed remote to the NUMA node an application, service or VM is connected to. NUMA spanning means your application or service won’t run out of memory or CPU cores within its NUMA node as it can access the remote DIMMs or cores of an adjacent CPU. The problem with NUMA spanning is that your application or service has to cross the lower performing interconnects to store its pages in the remote DIMMs. This dialog in Hyper-V Settings explains the situation fairly well:

What is NUMA alignment?

In the case of virtualization, VMs created on a physical server receive their virtual CPU (vCPU) allotments from the physical CPU cores of their hosts (obviously). There are many tricks used by hypervisors to oversubscribe CPU and run as many VMs on a host as possible while ensuring good performance. Better utilization of physical resources = better return on investment and use of the company dollar. To design your virtual environment to be NUMA aligned means ensuring that your VMs receive vCPUs and RAM tied to a single physical CPU (pCPU), thus ensuring the memory and pCPU cores they access are directly connected and not accessed via traversal of a CPU interconnect (QPI). As fast as interconnects are these days, they are still not as fast as the connection between a DIMM and pCPU.
Running many smaller 1 or 2 vCPU VMs with lower amounts of RAM, such as virtual desktops, aren’t as big of a concern and we generally see very good oversubscription at high performance levels. In the testing my team does at Dell for VDI, we see somewhere between 9-12 vCPUs per pCPU core depending on the workload and VM profile. For larger server-based VMs, such as those used for RDSH sessions, this ratio is smaller and more impactful because there are fewer overall VMs consuming a greater amount of resources. In short, NUMA alignment is much more important when you have VMs with larger vCPU and RAM allocations.
To achieve sustained high performance for a dense server workload, it is important to consider:

The number of vCPUs assigned to each VM within a NUMA node to avoid contention
The amount RAM assigned to each VM to ensure that all memory access remains within a physical NUMA node.

The following is an example of a balanced configuration based on a dual 12-core, 256GB server used for RDSH. The server hosts 6 high performance VMs that are configured within 2 NUMA node boundaries. Given the number of vCPUs and RAM assigned to each VM, each NUMA node is oversubscribed 2x from a pCPU core perspective and does not exceed the 128GB of RAM available to each pCPU. This ensures that NUMA spanning is not required and minimizes potential contention amongst the VMs. Your oversubscription ratio may vary depending on workload or SLAs.

NUMA Alignment in Hyper-V

Let’s walk through a couple of examples in the lab to illustrate this. My Windows cluster consists of:

2 x Dell PowerEdge R610’s with dual X5670 6-core CPUs and 96GB RAM running Server 2012 R2 Datacenter in a Hyper-V Failover Cluster. 24 total logical processors seen by Windows (hyperthreaded).
3 x 2012 R2 VMs named X1, X2, X3 each configured with 12 x vCPUs, 32GB RAM

Looking at the vCPU NUMA configuration of my X2 VM, notice that the hardware defaults to a 2x oversubscription scheme based on the physical server hardware. A NUMA node means 1 pCPU socket so default max vCPUs is 12 in my test server (6-cores hyperthreaded = 12 logical cores). These values can be adjusted on a VM by VM basis and Windows will allow you to assign up to the max logical observed processor count in the server. Here you can see my X2 VM is configured according to a 2x maximum for 12 vCPUs total aligning to the NUMA boundary of a single pCPU. Based on the capability of the pCPUs in my server, the maximum value I can assign to a single VM is 24 vCPUs. Also note that the maximum amount of RAM that can be assigned to a single VM matches that of a single 48GB NUMA node on this server.

Hyper-V automatically assigns VMs to NUMA nodes and balances the load between them. Changing this same VM to an allotment of 13 vCPUs, 1 vCPU beyond the configured NUMA boundary, has immediate affects. Now this VM is spanning across 2 NUMA nodes/ pCPU sockets for its vCPU entitlement. CPU contention could occur at this point with other VMs as load increases but memory pages will not become remote unless the memory available to a local NUMA node is exceeded. The same thing happens here if you assign a memory allocation larger than a single NUMA node.

To illustrate what is happening here, consider the following drawing. The X3 VM having 13 vCPUs is exceeding the configured maximum of processors for its NUMA topology and must now span 2 physical sockets. Notice that the assigned virtual memory in this configuration is still local to each VM’s primary NUMA node.

For a VM to actually be able to store memory pages in a remote NUMA node, the key change that must be made is called out on the main processor tab for the VM spanning NUMA: Dynamic memory must be disabled to allow vNUMA. If dynamic memory is enabled on a VM, no NUMA topology is exposed to the VM, regardless of what settings you change! Note here also that there is no way to set CPU affinity for a VM in Hyper-V and generally you should never need to. If you want to guarantee a certain percentage of CPU for a VM, set its “Virtual Machine Reserve %” accordingly.

NOTE – Remember that if you use Dynamic Memory, you need to carefully consider the values for: Startup, min, max and buffer. What you configure for Startup here is what your VM will boot with and consume as a minimum! If you set your startup to 32GB, your VM will consume 32GB with no workload, reducing what is available to other VMs. Configure your max RAM value as the top end amount you want your VM to expand to.

To illustrate the effects of an imbalanced configuration, consider the following PowerShell output. My X2 VM has been assigned 32GB RAM with Dynamic Memory enabled. My X3 VM has also been assigned 32GB RAM but with Dynamic Memory disabled thus forcing NUMA spanning. The “remote physical pages” counters indicate NUMA spanning in action with the X3 VM currently storing 6% of its memory pages in the remote NUMA node. This is with absolutely no workload running mind you. Lesson: In most cases, you want to leave Dynamic Memory enabled to achieve the best performance and balance of resources! Commands used below:

Get-VM
Get-VMProcessor
Get-counter “\hyper-v VM VID Partition(*)\*”

Here is a visual of what’s happening with X3 spanning NUMA. A percentage of its memory pages are being stored in the remote NUMA node 0 which requires crossing the QPI links to access, thus reducing performance:

Look what happens when I re-enable Dynamic Memory on X3. You want to always see the values for “remote physical pages” as 0:

The risk of this happening is greater with VMs with heavy resource requirements. There may be some instances where the performance tradeoff is worth it, simply because you have a VM that absolutely requires more resources. Look at what happens when I bring my X1 VM online configured with 64GB static RAM on the same server:

It is immediately punted to the second server in my cluster that has enough resources to power it on. If the cluster had too few resources available on any node, the VM would fail to power on.

It’s worth noting that if you have your VMs set to save state when the Hyper-V host shuts down, be prepared to consume a potentially enormous amount of disk. When this is enabled Hyper-V will write a BIN file equal to the amount of RAM in use by the VM. If you disabled Dynamic Memory then the BIN file will be the full amount of assigned RAM.

32GB on disk for my X3 VM with Dynamic Memory disabled:

Key Takeaways

Designing your virtual infrastructure to work in concert with the physical NUMA boundaries of your servers will net optimal performance and minimize contention, especially for VMs with large resource assignments.
A single VM consuming more vCPUs than a single NUMA node contains, will be scheduled across multiple physical NUMA nodes causing possible CPU contention as loads increase.
A single or multiple VMs consuming more RAM than a single NUMA node contains will span NUMA and store a percentage of their memory pages in the the remote NUMA node causing reduced performance.
In Hyper-V, in most cases, you want to leave Dynamic Memory enabled so the server can adjust using a balloon driver as required.
CPU affinity does not exist in Hyper-V. If you want to assign a VM a guaranteed slice of CPU, set its “virtual machine reserve %” in its CPU settings.
Virtual memory assignments are cumulative, more impactful when not using dynamic memory.
Virtual CPU assignments are non-cumulative and will not force a VM to span NUMA as long as a vCPU assignment falls within the limits of a defined NUMA node. A vCPU assignment that exceeds the boundaries of a configured NUMA node will cause spanning.
Be careful what you set your “automatic stop action” to in Hyper-V manager or SCVMM. If set to save state, large BIN files will be written to disk equaling the memory in use by each VM.

Resources:

http://lse.sourceforge.net/numa/faq/
Hyper-V Virtual NUMA Overview
Hyper-V 2012 Checklist

Dell XC Series 2.0: Product Architectures

Following our launch of the new 13G-based XC series platform, I present our product architectures for the VDI-specific use cases. Of the platforms available, this use case is focused on the extremely powerful 1U XC630 with Haswell CPUs and 2133MHz RAM. We offer these appliances on both Server 2012 R2 Hyper-V and vSphere 5.5 U2 with Citrix XenDesktop, VMware Horizon View, or Dell vWorkspace. All platform architectures have been optimized, configured for best performance and documented.

Platforms

We have three platforms to choose from optimized around cost and performance, all being ultimately flexible should specific parameters need to change. The A5 model is the most cost effective leveraging 8-core CPUs, 256GB RAM 2 x 200GB SSDs for performance and 4 x 1TB HDDs for capacity. For POCs, small deployments or light application virtualization, this platform is well suited. The B5 model steps up the performance by adding four cores per socket, increasing the RAM density to 384GB and doubling the performance tier to 2 x 400GB SSDs. This platform will provide the best bang for the buck on medium density deployments of light or medium level workloads. The B7 is the top of the line offering 16-core CPUs and a higher capacity tier of 6 x 1TB HDDs. For deployments requiring maximum density of knowledge or power user workloads, this is the platform for you.

At 1U with dual CPUs, 24 DIMM slots and 10 drive bays…loads of potential and flexibility!

Solution Architecture

Utilizing 3 platform hardware configurations, we are offering 3 VDI solutions on 2 hypervisors. Lots of flexibility and many options. 3 node cluster minimum is required with every node containing a Nutanix Controller VM (CVM) to handle all IO. The SATADOM is present for boot responsibilities to host the hypervisor as well as initial setup of the Nutanix Home area. The SSDs and NL SAS disks are passed through directly to each CVM which straddle the hypervisor and hardware. Every CVM contributes its directly-attached disks to the storage pool which is stretched across all nodes in the Nutanix Distributed File System (NDFS) cluster. NDFS is not dependent on the hypervisor for HA or clustering. Hyper-V cluster storage pools will present SMB version 3 to the hosts and vSphere clusters will be presented with NFS. Containers can be created within the storage pool to logically separate VMs based on function. These containers also provide isolation of configurable storage characteristics in the form of dedupe and compression. In other words, you can enable compression on your management VMs within their dedicated container, but not on your VDI desktops, also within their own container. The namespace is presented to the cluster in the form of \\NDFS_Cluster_name\container_name.
The first solution I’ll cover is Dell’s Wyse vWorkspace which supports either 2012 R2 Hyper-V or vSphere 5.5. For small deployments or POCs we offer this solution in a “floating mgmt” architecture which combines the vWorkspace infrastructure management roles and VDI desktops or shared session VMs. vWorkspace and Hyper-V enables a special technology for non-persistent/ shared image desktops called Hyper-V Catalyst which includes 2 components: HyperCache and HyperDeploy. Hyper-V Catalyst provides some incredible performance boosts and requires that the vWorkspace infrastructure components communicate directly with the hyper-V hypervisor. This also means that vWorkspace does not require SCVMM to perform provisioning tasks for non-persistent desktops!

HyperCache – Provides virtual desktop performance enhancement by caching parent VHDs in host RAM. Read requests are satisfied from cache including requests from all subsequent child VMs.
HyperDeploy – Provides instant cloning of parent VHDs massively diminishing virtual desktop pool deployment times.

You’ll notice the HyperCache components included on the Hyper-V architectures below. 3 to 6 hosts in the floating management model, depicted below with management, desktops and RDSH VMs logically separated only from a storage container perspective by function. The recommendation of 3-7 RDSH VMs is based our work optimizing around NUMA boundaries. I’ll dive deeper into that in an upcoming post. The B7 platform is used in the architectures below.

Above ~1000 users we recommend the traditional distributed management architecture to enable more predictable scaling and performance of both the compute and management hosts. The same basic architecture is the same and scales to the full extent supported by the hypervisor, is this case Hyper-V which supports up to 64 hosts. NDFS does not have a scaling limitation so several hypervisor clusters can be built within a single contiguous NDFS namespace. Our recommendation is to then build independent Failover Clusters for compute and management discretely so they can scale up or out independently.
The architecture below depicts a B7 build on Hyper-V applicable to Citrix XenDesktop or Wyse vWorkspace.

This architecture is relatively similar for Wyse vWorkspace or VMware Horizon View on vSphere 5.5 U2 but fewer total compute hosts per HA cluster, 32 total. For vWorkspace, Hyper-V Catalyst is not present in this scenario so vCenter is required to perform desktop provisioning tasks.

For the storage containers, the best practice of less is more still stands. If you don’t need a particular feature don’t enable it, as it will consume additional resources. Deduplication is always recommended on the performance tier since the primary OpLog lives on SSD and will always benefit. Dedupe or compression on the capacity tier is not recommended, of course you absolutely need it. And if you do prepare to increase each CVM RAM allocation to 32GB.

Container	Purpose	Replication Factor	Perf Tier Deduplication	Capacity Tier Deduplication	Compression
Ds_compute	Desktop VMs	2	Enabled	Disabled	Disabled
Ds_mgmt	Mgmt Infra VMs	2	Enabled	Disabled	Disabled
Ds_rdsh	RDSH Server VMs	2	Enabled	Disabled	Disabled

Network Architecture

As a hyperconverged appliance, the network architecture leverages the converged model. A pair of 10Gb NICs minimum in each node handle all traffic for the hypervisor, guests and storage operations between CVMs. Remember that the storage of all VMs is kept local to the host to which the VM resides, so the only traffic that will traverse the network is LAN and replication. There is no need to isolate storage protocol traffic when using Nutanix.
Hyper-V and vSphere are functionally similar. For Hyper-V there are 2 vSwitches per host, 1 external that aggregates all of the services of the host management OS as well as the vNICs for the connected VMs. The 10Gb NICs are connected to a LBFO team configured in Dynamic Mode. The CVM alone connects to a private internal vSwitch so it can communicate with the hypervisor.

In vSphere it’s the same story but with the concept of Port Groups and vMotion.

We have tested the various configurations per our standard processes and documented the performance results which can be found in the link below. These docs will be updated as we validate additional configurations.

Product Architectures for 13G XC launch:

http://bit.ly/DellXCLib

Resources:

About Wyse vWorkspace HyperCache
About Wyse vWorkspace HyperDeploy
SJbWUYl4LRCwFNcatHuG

Dell XC Series Web-scale Converged Appliance 2.0

I am pleased to present the Dell XC 2.0 series of web-scale appliances based on the award-winning 13G PowerEdge server line from Dell. There’s lots more in store for the XC so just focusing on this part of the launch we are introducing the XC630 and the XC730xd appliances.

Flexibility and performance are key tenets of this launch providing not only a choice in 1U or 2U form factors, but an assortment of CPU and disk options. From a solution perspective, specifically around VDI, we are releasing three optimized and validated platforms with accompanied Product Architectures to help you plan and size your Dell XC deployments.

The basic architecture of the Dell XC powered by Nutanix remains the same. Every node is outfitted with a Nutanix Controller VM (CVM) that connects to a mix of SSD and HDD to contribute to a distributed storage pool that has no inherent scaling limitation. Three nodes minimum required and either VMware vSphere or Microsoft Windows Server 2012 R2 Hyper-V are supported hypervisors. Let’s take a look at the new models.

XC630

The XC630 is a 1U dual-socket platform that supports 6-core to 16-core CPUs and up to 24 x 2133MHz 16GB RDIMMs or 32GB LRDIMMs. The XC630 can be configured using all flash or using two tiers of storage which can consist of 2 to 4 x SSDs (200GB, 400GB or 800GB) and 4 to 8 x 1TB HDDs (2TB HDDs coming soon). Flexible! All flash nodes must have a minimum of 6 x SSDs while nodes with two storage tiers must have a minimum of two SSDs and four HDDs. All nodes have a minimum of 2 x 1Gb plus 2 x 10Gb SFP+ or BaseT Ethernet that can be augmented via an additional card.

New to the XC 2.0 series is a 64GB SATADOM that is used to boot each node. Each node is also outfitted with a 16GB SD card used for the purposes of initial deployment and recovery. The SSDs and HDDs that comprise the Nutanix Distributed File System (NDFS) storage pool are presented to each CVM via an on-board 1GB PERC H730 set in pass-through mode. Simple, powerful, flexible.

XC730xd

For deployments requiring a greater amount of cold tier data capacity, the XC730xd can provide up to 32TB raw per node. The XC730xd is a 2U dual-socket platform that supports 6-core to 16-core CPUs and up to 24 x 2133MHz 16GB RDIMMs or 32GB LRDIMMs. The XC730xd is provided with two chassis options: 24 x 2.5” disks or 12 x 3.5” disks. The 24-drive model requires the use of two tiers of storage which can consist of 2 to 4 x SSDs (200GB, 400GB or 800GB) and 4 to 22 x 1TB HDDs .The 12-drive model also requires two tiers of storage consisting of 2 to 4 x SSDs (200, 400GB or 800GB) and up to 10 x 4TB HDDs. All nodes have a minimum of 2 x 1Gb plus 2 x 10Gb SFP+ or BaseT Ethernet that can be augmented via an additional card.

The XC730xd platforms are also outfitted with a 64GB SATADOM that is used to boot the nodes. The 16GB SD card used for the purposes of initial deployment and recovery is present on these models as well. The SSDs and HDDs that comprise the Nutanix Distributed File System (NDFS) storage pool are presented to each CVM via an on-board 1GB PERC H730 set in pass-through mode. Simple, powerful, flexible.

12 drive option, hopefully the overlaps in the image below make sense:

24 drive option:

Nutanix Software Editions

All editions of the Nutanix software platform are available with variable lengths for support and maintenance.

This is just the beginning. Keep an eye out for additional platforms and offerings from the Dell + Nutanix partnership! Next up is the VDI product architectures based on the XC630. Stay tuned!!

http://www.dell.com/us/business/p/dell-xc-series/pd

Proper Shut Down of Nutanix 2012 R2 Hyper-V Cluster

Happy 2015!! I’m seriously backed up and delinquent on blog posts so prepare for a deluge of interesting topics soon.

As painful and hopefully unlikely as this scenario may be, better to have a planned and controlled outage vs unplanned with potential data loss, or worse. This post covers the cluster shutdown process for Nutanix with Hyper-V. In case you didn’t know, Nutanix is not dependent on Windows or vSphere for clustering, as it handles its own HA mechanism. This does require, however, that you consider the independent Nutanix cluster services to ensure a graceful shutdown.

In the references at the bottom, you’ll notice that for vSphere clusters, the method described there is to shut down the Nutanix cluster first, then the vSphere cluster. For Hyper-V the procedure is reversed, requiring that the Hyper-V cluster be shut down first, then the Nutanix cluster. If you shut down your Nutanix cluster services first, you won’t be able to connect to your Failover Cluster to properly shut it down.

Shut Down Failover Cluster

Gracefully shut down all VMs (Not Nutanix CVMs, yet!!)
Open Failover Cluster Manager (cluadmin.msc) and connect to the Failover Cluster
Right-click the cluster name, select “more actions” –> “shut down cluster”, select “yes” to confirm

Alternatively, in PowerShell enter: “Stop-Cluster”

Once the Hyper-V cluster is shut down, you can begin to shut down the Nutanix cluster.

Shut Down Nutanix Cluster

SSH or connect to any CVM console, log in
Enter “cluster stop” (root not required)
Once completed successfully, you will see confirmation that all services have stopped on all CVMs:
Shut down each CVM on all hosts:
Lastly, shut down each physical Hyper-V node – This can be done of number of ways, onesy twosy, PowerShell, or via the Remote Shutdown UI (shutdown /i) for bulk operation:

Startup is the reverse, ensuring that your Nutanix cluster is up and running properly before starting your failover cluster.

References:

http://blogs.msdn.com/b/clustering/archive/2013/08/23/10443912.aspx

http://port5480.blogspot.com/2014/10/shut-down-nutanix-cluster.html

Clustering Server 2012 R2 with iSCSI Storage

Yay, last post of 2014! Haven’t invested in the hyperconverged Software Defined Storage model yet? No problem, there’s still time. In the meanwhile, here is how to cluster Server 2012 R2 using tried and true EqualLogic iSCSI shared storage.

EQL Group Manager

First, prepare your storage array(s), by logging into EQL Group Manager. This post assumes that your basic array IP, access and security settings are in place. Set up your local CHAP account to be used later. Your organization’s security access policies or requirements might dictate a different standard here.

Create and assign an Access Policy to the VDS/VSS in Group Manager otherwise this volume will not be accessible. This will make subsequent steps easier when it’s time to configure ASM.

Create some volumes in Group Manager now so you can connect your initiators easily in the next step. It’s a good idea to create your cluster quorum LUN now as well.

Host Network Configuration

First configure the interfaces you intend to use for iSCSI on your cluster nodes. Best practice says that you should limit your iSCSI traffic to a private Layer2 segment, not routed and only connecting to the devices that will participate in the fabric. This is no different from Fiber Channel in that regard, unless you are using a converged methodology and sharing your higher bandwidth NICs. If using Broadcom NICs you can choose Jumbo Frames or hardware offload, the larger frames will likely net a greater performance impact. Each host NIC used to access your storage targets should have a unique IP address able to access the network of those targets within the same private Layer2 segment. While these NICs can technically be teamed using the native Windows LBFO mechanism, best practice says that you shouldn’t, especially if you plan to use MPIO to load balance traffic. If your NICs will be shared (not dedicated to iSCSI alone) then LBFO teaming is supported in that configuration. To keep things clean and simple I’ll be using 4 NICs, 2 dedicated to LAN, 2 dedicated to iSCSI SAN. Both LAN and SAN connections are physically separated to their own switching fabrics as well, this is also a best practice.

MPIO – the manual method

First, start the MS iSCSI service, which you will be prompted to do, and check its status in PowerShell using get-service –name msiscsi.

Next, install MPIO using Install-WindowsFeature Multipath-IO

Once installed and your server has been rebooted, you can set additional options in PowerShell or via the MPIO dialog under File and Storage Services—> Tools.

Open the MPIO settings and tick “add support for iSCSI devices” under Discover Multi-Paths. Reboot again. Any change you make here will ask you to reboot. Make all changes once so you only have to do this one time.

The easier way to do this from the onset is using the EqualLogic Host Integration Tools (HIT Kit) on your hosts. If you don’t want to use HIT for some reason, you can skip from here down to the “Connect to iSCSI Storage” section.

Install EQL HIT Kit (The Easier Method)

The EqualLogic HIT Kit will make it much easier to connect to your storage array as well as configure the MPIO DSM for the EQL arrays. Better integration, easier to optimize performance, better analytics. If there is a HIT Kit available for your chosen OS, you should absolutely install and use it. Fortunately there is indeed a HIT Kit available for Server 2012 R2.

Configure MPIO and PS group access via the links in the resulting dialog.

In ASM (launched via the “configure…” links above), add the PS group and configure its access. Connect to the VSS volume using the CHAP account and password specified previously. If the VDS/VSS volume is not accessible on your EQL array, this step will fail!

Connect to iSCSI targets

Once your server is back up from the last reboot, launch the iSCSI Initiator tool and you should see any discovered targets, assuming they are configured and online. If you used the HIT Kit you will already be connected to the VSS control volume and will see the Dell EQL MPIO tab.

Choose an inactive target in the discovered targets list and click connect, be sure to enable multi-path in the pop-up that follows, then click Advanced.

Enable CHAP log on, specify the user/pw set up previously:

If your configuration is good the status of your target will change to Connected immediately. Once your targets are connected, the raw disks will be visible in Disk Manager and can be brought online by Windows.

When you create new volumes on these disks, save yourself some pain down the road and give them the same label as what you assigned in Group Manager! The following information can be pulled out of the ASM tool for each volume:

Failover Clustering

With all the storage pre-requisites in place you can now build your cluster. Setting up a Failover Cluster has never been easier, assuming all your ducks are in a row. Create your new cluster using the Failover Cluster Manager tool and let it run all compatibility checks.

Make sure your patches and software levels are identical between cluster nodes or you’ll likely fail the clustering pre-check with differing DSM versions:

Once the cluster is built, you can manipulate your cluster disks and bring any online as required. Cluster disks will not be able to be brought online until all nodes in the cluster can access the disk.

Next add your cluster disks to Cluster Shared Volumes to enable multi-host read/write and HA.

The new status will be reflected once this change is made.

Configure your Quorum to use the disk witness volume you created earlier. This disk does not need to be a CSV.

Check your cluster networks and make sure that iSCSI is set to not allow cluster network communication. Make sure that your cluster network is setup to allow cluster network communication as well as allowing client connections. This can of course be further segregated if desired using additional NICs to separate cluster and client communication.

Now your cluster is complete and you can begin adding HA VMs, if using Hyper-V, SQL, File or other roles as required.

References:

http://blogs.technet.com/b/keithmayer/archive/2013/03/12/speaking-iscsi-with-windows-server-2012-and-hyper-v.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/03/18/is-nic-teaming-in-windows-server-2012-supported-for-iscsi-or-not-supported-for-iscsi-that-is-the-question.aspx

Dell XC Series – Product Architectures

Hyperconverged Web-scale Software Defined Storage (SDS) solutions are white hot right now and Nutanix is leading the pack with their ability to support all major hypervisors (vSphere, Hyper-V and KVM) while providing nearly unlimited scale. Dell partnering with Nutanix was an obvious mutually beneficial choice for the reasons above plus supplying a much more robust server platform. Dell also provides a global reach for services and support as well as solving other challenges such as hypervisors installed in the factory.

Nutanix operates below the hypervisor layer so as a result requires a lot of tightly coupled interaction with the hardware directly. Many competing platforms in this space sit above the hypervisor so require vSphere, for example, to provide access to storage and HA but they are also limited by the hypervisor’s limitations (scale). Nutanix uses its own algorithm for clustering and doesn’t rely on a common transactional database which can cause additional challenges when building solutions that span multiple sites. Because of this the Nutanix Distributed Filesystem (NDFS) supports no known limits of scale. There are current Nutanix installations in the thousands of nodes across a contiguous namespace and now you can build them on Dell hardware.

Along with the Dell XC720xd appliances, we have released a number of complementary workload Product Architectures to help customers and partners build solutions using these new platforms. I’ll discuss the primary architectural elements below.

Wyse Datacenter Appliance Architecture for Citrix

Wyse Datacenter Appliance Architecture for VMware

Wyse Datacenter Appliance Architecture for vWorkspace

Nutanix Architecture

Three nodes minimum are required for NDFS to achieve quorum so that is the minimum solution buy in, then storage and compute capacity can be increased incrementally beyond by adding one or more nodes to an existing cluster. The Nutanix architecture uses a Controller VM (CVM) on each host which participates in the NDFS cluster and manages the hard disks local to its own host. Each host requires two tiers of storage: high performance/ SSD and capacity/ HDD. The CVM manages the reads/writes on each host and automatically tiers the IO across these disks. A key value proposition of the Nutanix model is data locality which means that the data for a given VM running on a given host is stored locally on that host as apposed to having reads and writes crossing the network. This model scales indefinitely in a linear block manner where you simply buy and add capacity as you need it. Nutanix creates a storage pool that is distributed across all hosts in the cluster and presents this pool back to the hypervisor as NFS or SMB.

You can see from the figure below that the CVM engages directly with the SCSI controller through which it accesses the disks local to the host it resides. Since Nutanix sits below the hypervisor and handles its own clustering and data HA, it is not dependent upon the hypervisor to provide any features nor is it limited by any related limitations.

From a storage management and feature perspective, Nutanix provides two tiers of optional deduplication performed locally on each host (SSD and HDD individually), compression, tunable replication (number of copies of each write spread across disparate nodes in the cluster) and data locality (keeps data local to the node the VM lives on). Within a storage pool, containers are created to logically group VMs stored within the namespace and enable specific storage features such as replication factor and dedupe. Best practice says that a single storage pool spread across all disks is sufficient but multiple containers can be used. The image below shows an example large scale XC-based cluster with a single storage pool and multiple containers.

While the Nutanix architecture can theoretically scale indefinitely, practicality might dictate that you design your clusters around the boundaries of the hypervisors, 32 nodes for vSphere, 64 nodes for Hyper-v. The decision to do this will be more financially impactful if you separate your resources along the lines of compute and management in distinct SDS clusters. You could also, optionally, install many maximum node hypervisor clusters within a single very large, contiguous Nutanix namespace, which is fully supported. I’ll discuss the former option below as part of our recommended pod architecture.

Dell XC720xd platforms

For our phase 1 launch we have five platforms to offer that vary in CPU, RAM and size/ quantity of disks. Each appliance is 2U, based on the 3.5” 12-gen PowerEdge R720XD and supports from 5 to 12 total disks, each a mix of SSD and HDD. The A5 platform is the smallest with a pair of 6-core CPUs, 200GB SSDs and a recommended 256GB RAM. The B5 and B7 models are almost identical except for the 8-core CPU on the B5 and the 10-core CPU on the B7. The C5 and C7 boast a slightly higher clocked 10-core CPU with doubled SSD densities and 4-5x more in the capacity tier. The suggested workloads are specific with the first three targeted at VDI customers. If greater capacity is required, the C5 and C7 models work very well for this purpose too.

For workload to platform sizing guidance, we make the following recommendations:

Platform	Workload	Special Considerations
A5	Basic/ light task users, app virt	Be mindful of limited CPU cores and RAM densities
B5	Medium knowledge workers	Additional 4 cores and greater RAM to host more VMs or sessions
B7	Heavy power users	20 cores per node + a recommended 512GB RAM to minimize oversubscription
C5	Heavy power users	Higher density SSDs + 20TB in the capacity tier for large VMs or amount of user data
C7	Heavy power users	Increased number of SSDs with larger capacity for greater amount of T1 performance

Here is a view of the 12G-based platform representing the A5-B7 models. The C5 and C7 would add additional disks in the second disk bay. The two disks in the rear flexbay are 160GB SSDs configured in RAID1 via PERC used to host the hypervisor and CVM, these disks do not participate in the storage pool. The six disks in front are controlled by the CVM directly via the LSI controller and contribute to the distributed storage pool across all nodes.

Dell XC Pod Architecture

This being a 10Gb hyperconverged architecture, the leaf/ spine network model is recommended. We do recommend a 1Gb switch stack for iDRAC/ IPMI traffic and build the leaf layer from 10Gb Force10 parts. The S4810 is shown in the graphic below which is recommended for SFP+ based platforms or the S4820T can be used for 10GBase-T.

In our XC series product architecture, the compute, management and storage layers, typically all separated, are combined here into a single appliance. For solutions based on vWorkspace under 10 nodes, we recommend a “floating management” architecture which allows the server infrastructure VMs to move between hosts also being used for desktop VMs or RDSH sessions. You’ll notice in the graphics below that compute and management are combined into a single hypervisor cluster which hosts both of these functions.

Hyper-V is shown below which means the CVMs present the SMBv3 protocol to the storage pool. We recommend three basic containers to separate infrastructure mgmt, desktop VMs and RDSH VMs. We recommend the following feature attributes based on these three containers (It is not supported to enable compression and deduplication on the same container):

Container	Purpose	Replication Factor	Perf Tier Deduplication	Capacity Tier Deduplication	Compression
Ds_compute	Desktop VMs	2	Enabled	Enabled	Disabled
Ds_mgmt	Mgmt Infra VMs	2	Enabled	Disabled	Disabled
Ds_rdsh	RDSH Server VMs	2	Enabled	Enabled	Disabled

You’ll notice that I’ve included the resource requirements for the Nutanix CVMs (8 x vCPUs, 32GB vRAM). The vRAM allocation can vary depending on the features you enable within your SDS cluster. 32GB is required, for example, if you intend to enable both SSD and HDD deduplication. If you only require SSD deduplication and leave the HDD tier turned off, you can reduce your CVM vRAM allocation to 16GB. We highly recommend that you disable any features that you do not need or do not intend to use!

For vWorkspace solutions over 1000 users or solutions based on VMware Horizon or Citrix XenDesktop, we recommend separating the infrastructure management in all cases. This allows management infrastructure to run in its own dedicated hypervisor cluster while providing very clear and predictable compute capacity for the compute cluster. The graphic below depicts a 1000-6000 user architecture based on vWorkspace on Hyper-V. Notice that the SMB namespace is stretched across both of the discrete compute and management infrastructure clusters, each scaling independently. You could optionally build dedicated SDS clusters for compute and management if you desire, but remember the three node minimum, which would raise your minimum build to 6 nodes in this scenario.

XenDesktop on vSphere, up to 32 nodes max per cluster, supporting around 2500 users in this architecture:

Horizon View on vSphere, up to 32 nodes max per cluster. supporting around 1700 users in this architecture:

Network Architecture

Following the leaf/ spine model, each node should connect 2 x 10Gb ports to a leaf switch which are then fully mesh connected to an upstream set of spine switches.

On each host there are two virtual switches: one for external access to the LAN and internode communication and one private internal vSwitch used for the CVM alone. On Hyper-V the two NICs are configured in a LBFO team on each host with all management OS vNICs connected to it.

vSphere follows the same basic model except for port groups configured for the VM type and VMKernel ports configured for host functions:

Performance results

The tables below summarize the user densities observed for each platform during our testing. Please refer to the product architectures linked at the beginning of this post for the detailed performance results for each solution.

Resources:

http://en.community.dell.com/dell-blogs/direct2dell/b/direct2dell/archive/2014/11/05/dell-world-two-questions-cloud-client-computing

http://blogs.citrix.com/2014/11/07/dell-launches-new-appliance-solution-for-desktop-virtualization/

http://blogs.citrix.com/2014/11/10/xendesktop-technologies-introduced-as-a-new-dell-wyse-datacenter-appliance-architecture/

http://blogs.vmware.com/euc/2014/11/vmware-horizon-6-dell-xc-delivering-new-economics-simplicity-desktop-application-virtualization.html

http://stevenpoitras.com/the-nutanix-bible/

http://www.dell.com/us/business/p/dell-xc-series/pd

Dell XC Web Scale Converged Appliance

That’s a mouthful! Here’s a quick taste of the new Dell + Nutanix appliance we’ve been working on. Our full solution offering will be released soon, stay tuned. In the meantime, the Dell marketing folks put together a very sexy video:

vSphere 5.5: Navigating the Web Client

As I called out in my vSphere 5.5 upgrade post, the vSphere Client is now deprecated in 5.5 so in preparation of an inevitable future, I’m forcing myself to use the Web Client to gain familiarity. Turns out there was way more moving around than I initially thought so I’m selfishly documenting a few pertinent items that seemed less than intuitive to me my first time through. Some things are just easier to do in the legacy vSphere Client, or maybe I’m just too accustomed after 3 generations of ESX/i. In any case, I encourage you to use the web client as well and hopefully these tips will help.

Topics covered in this post:

How to configure iSCSI software adapters
How to add datastores
How to manage multipathing
How to rename an ESXi host
Cool changes in Recent Tasks pane
Traffic Shaping
Deploying vCenter Operations Manager (vCOps)

How to configure iSCSI software adapters:

This assumes that the preliminary steps of setting up your storage array and requisite physical networking have already been properly completed. The best and easiest way to do this is via dedicated switches and server NICs for iSCSI in a Layer2 switch segment. Use whatever IP scheme you like, this should be a closed fabric and there is no reason to route this traffic.

First things first, if you don’t have a software iSCSI adapter created on your hosts, create one in the Storage Adapters section of Storage Management for a particular ESXi host. Once created, it will appear in the list below. A quick note on software vs hardware iSCSI initiators. Physical initiators can generally do iSCSI offload OR jumbo frames. not both. We have seen the use of jumbo frames to be more impactful to performance than iSCSI offload, so software initiators with jumbo frames enabled is the preferred way to go here.

Click over to the Networking tab and create a new vSwitch with a VMkernel Network Adapter for iSCSI.

Choose the physical adapters to be used in this vSwitch, create a useful network Port Group label such as iSCSI-1 and assign an IP address that can reach the storage targets. Repeat this process and add a second VMkernel adapter to the same vSwitch. Configure your VMK ports to use apposing physical NICs. This is done by editing the port group settings and changing the Failover order. This allows you to cleanly share 2 physical NICs for 2 iSCSI connections within a single vSwitch.

In my case VMK2 is active on vmnic3 and VMK3 is active on vmnic1 providing physical path redundancy to the storage array.

When all is said and done, your vSwitch configuration should look something like this:

Next under the iSCSI software adapter, add the target IP to your storage (group IP for EqualLogic). Authentication needs and requirements will vary between organizations. Choose and configure this scheme appropriately for your environment. For my lab, I scope connections based on subnet alone which defines the physical Layer2 boundary of my iSCSI fabrics.

Next configure the network port binding to ensure that the port groups you defined earlier get bound to the iSCSI software adapter using the proper physical interfaces.

At this point, if you have any volumes created on your array and presented to your host, a quick rescan should reveal the devices presented to your host as LUNs.

You should also see 2 paths per LUN (device) per host based on 2 physical adapters connecting to your array. EqualLogic is an active/passive array so only connections to the active controller will be seen here.

If you run into trouble making this work after these steps, jump over to the vSphere Client which does make this process a bit easier. Also keep in mind that all pathing will be set to Fixed by default. See my How to manage multipathing topic below for guidance on changing this.

iSCSI works very well with jumbo frames which is an end-to-end Layer2 technology, so makes sure a MTU of 9000 is set on all ESXi iSCSI vSwitches, VMK ports, as well as the NICs on the storage array. Your switches must be capable of supporting jumbo frames as well. This will increase the performance of your iSCSI network and front-end storage operation speeds.

How to add datastores:

Once your new datastore has been provisioned from your storage platform and presented to your ESXi hosts, from the Hosts and Clusters view, navigate to Related Objects then datastores. From here click the Create a new Datastore button.

Choose the host or cluster to add the datastore to, choose whether it is NFS or VMFS, name the datastore and choose a host that can see it. You should see the raw LUN in the dialog below.

Choose the VMFS version and any partition options you want to implement. Confirm and deploy.

If presenting to multiple hosts, once the VMFS datastore is created and initialized on one, they all should see it assuming the raw device is present via a previous adapter rescan.

How to manage multipathing:

From the Hosts and clusters view, click the Storage tab, choose the datastore you want to manage, click Manage in the middle pane then click Connectivity and Multipathing under Settings.

Alternatively, from the Hosts and Clusters view (from any level item), navigate to Related Objects, then Datastores. Either click the volume you want to edit or choose Settings from the dropdown. Either method will get you to the same place.

From the datastore Settings page, click Manage and under Settings (once again) click Connectivity and Multipathing. In the middle of the screen you should see all hosts attached to whatever datastore you selected. Clicking on each host will reveal the current Path Selection Policy below, “Fixed” by VMware default along with the number of paths present per host.

To change this to Round Robin, click Edit Multipathing, change the Path Selection Policy, repeat for each host connected to the datastore.

How to rename an ESXi host:

Renaming hosts is one area that the Web Client has made significantly easier (once you figure out where to go)! Select a host from the Hosts and Clusters view, click Manage, click Networking, then TCP/IP Configuration below.

From the DNS Configuration menu, select “Enter settings manually”, put whatever hostname you would like here.

VMware recommends putting a host in maintenance mode and disconnecting it from vCenter before doing this. I did this hot with my host active in an HA cluster with zero ill affects. I did it a few times just to make sure. The other way to do this is via the CLI. Connect to your ESXi host via SSH, vMA or vCLI and run:

esxcli system hostname set –host=hostname

Cool changes in Recent Tasks pane:

Not only is the Recent Tasks pane off to the right now, which I really like, it breaks out tasks by All, Running and Failed individually for easier viewing, including the ability to view your own tasks for environments with many admins. Previously these tasks were all lumped together and longer running tasks would get buried in the task stream.

The Recent Tasks pane also provides a new and better method to deal with pop-up configuration dialogs. Ever start configuring something using the old vSphere Client, get 4-5 clicks deep in the pop-up configuration, then realize you need some other piece of information requiring you to cancel out so you can go back to some other area in vCenter? This problem is now resolved in the web client with a cool new section of the Tasks pane called Work in Progress. It doesn’t matter what you’re doing or how far along you are in the dialog. If you need to break away for any reason, you can simply minimize the pop up and come back to it later. These minimized pop-ups will show in the Work in Progress pane below recent tasks.

The example here shows 3 concurrent activities in various states: a vMotion operation, a VM edit settings and a clone operation of the same VM even. Any activity that generates a pop-up dialog can be set aside and picked up again later. This is a huge improvement over the legacy vSphere Client. Very cool!!

Traffic Shaping:

It appears that in the web client you can only apply traffic shaping at the vSwitch level, not at an individual port group or VMK level. Here you can see shaping available for the standard vSwitch:

These settings, while viewable in the VMK policies summary, are not changeable (that I can see).

To override the vSwitch shaping policy and apply one to an individual port group or VMK, you have to use the legacy vSphere Client. Not sure if this is an oversight on VMware’s part or yet another sign of things to come requiring dvSwitching to assign shaping policies below the vSwitch level.

Deploying vCenter Operations Manager (vCOps):

Made extremely easy in vSphere 5.5 via the web client is the deployment of the incredible vCOps vApp for advanced monitoring of your environment. VMware has made detailed performance monitoring of your vSphere world incredibly simply and intuitive through this easy to set up and use vApp. Really impressive. From the home page, click vCenter Operations Management.

On the Getting Started screen, click Deploy vCOps. If you have a valid vmware.com login, entire it here to download the OVF and related files for deployment. You can alternatively point to the file locally if you have it already.

Accept the EULAs and choose all the placement and sizing options for the VM.

A word of caution, do not expect DRS to make a host placement decision for you here during the deployment. The wizard will allow you to select your cluster as a resource destination but the deployment will ultimately fail. Choose a specific host to deploy the VM to instead.

The requisite files will be downloaded from VMware directly and deployed to your environment. Off to the races!

Once deployed, you’ll see 2 new VMs running under the vCOps vApp object in your datacenter.

Once the VMs are powered on and the vApp has been started, you should see new options under vCenter Operations Manager.

First, click the Configure link to open the admin site in a web page. The default login for the admin account is admin/ admin, for root the password is vmware. Configure the initial setup to point to vCenter and the analytics VM which it should detect. Install the certificates as prompted and continue through the registration process.

Once complete, return to the vCOps page in vCenter and click Open, a new web page will launch for you to consume the vCOps goodness. After a short while performance stats should start pouring in for everything in your vSphere environment. Usage patterns and workload profiles can be identified so appropriate adjustments can be made. What you do from here with the data collected is entirely up to you. 🙂

A couple more screens just to show you the capability of vCOps, since I like it so much. Storage at the datastore view:

VM performance view:

Citrix XenDesktop on Dell VRTX

Dell Desktop Virtualization Solutions/ Cloud Client Computing is proud to announce support for the Dell VRTX on DVS Enterprise for Citrix XenDesktop. Dell VRTX is the new consolidated platform that combines blade servers, storage and networking into a single 5U chassis that we are enabling for the Remote Office/ Branch Office (ROBO) use case. There was tremendous interest in this platform at both the Citrix Synergy and VMworld shows this year!

Initially this solution offering will be available running on vSphere with Hyper-V following later, ultimately following a similar architecture to what I did in the Server 2012 RDS solution. The solution can be configured in either 2 or 4 M620 blades, Shared Tier 1 (cluster in a box), with 15 or 25 local 2.5” SAS disks. 10 x 15K SAS disks are assigned to 2 blades for Tier 1 or 20 disks for 4 blades. Tier 2 is handled via 5 x 900GB 10K SAS. A high-performance shared PERC is behind the scenes accelerating IO and enabling all disk volumes to be shared amongst all hosts.
Targeted sizing for this solution is 250-500 XenDesktop (MCS) users with HA options available for networking. Since all blades will be clustered, all VDI sessions and mgmt components will float across all blades to provide redundancy. The internal 8-port 1Gb switch can connect up to existing infrastructure or a Force10 switch can be added ToR. Otherwise there is nothing externally required for the VRTX VDI solution, other than the end user connection device. We of course recommend Dell Wyse Cloud Clients to suit those needs.

For the price point this one is tough to beat. Everything you need for up to 500 users in a single box! Pretty cool.
Check out the Reference Architecture for Dell VRTX and Citrix XenDesktop: Link

Server 2012 Native NIC Teaming and Hyper-V

One of the cool new features introduced in Server 2012 is the ability to team your NICs natively which is fully supported by Microsoft, without using any vendor-specific drivers or software. OEM driver-based teaming solutions have been around for much longer but have never been supported by Microsoft and are usually the first thing asked to be disabled if you ever call for PSS support. Server 2012 teaming is easy to configure and can be used for simple to very complex networking scenarios, converged or non. In this post I’ll cover the basics of teaming with convergence options having a focus around Hyper-V scenarios. A quick note on vernacular: Hyper-V host, parent, parent partition, or management OS all refer to essentially the same thing. Microsoft likes to refer to the network connections utilized by the hyper-V host as the “management OS” in the system dialogs.

100% of the teaming configuration can be handled via PowerShell with certain items specifically requiring it. Basic functionality can be configured via the GUI which is most easily accessed from the Local Server page in Server Manager:

The NIC Teaming app isn’t technically part of Server Manager but it sure looks and feels like it. Click New Team under the tasks drop down of the Teams section:

Name the team, select the adapters you want to include (up to 32) then set the additional properties. Switch independent teaming mode should suit the majority of use cases but can be changed to static or LACP mode if required. Load balancing mode options consists of address hash or Hyper-V port. If you intend to use this team within a Hyper-V switch, make sure the latter is selected. Specifying all adapters as active should also serve the majority of use cases but a standby adapter can be specified if required.

Click ok and your new team will appear within the info boxes in the lower portion of the screen. Additional adapters can be added to an existing team at any time.

Every team and vNIC you create will be named and enabled via the Microsoft Network Adapter Multiplexor Driver. This is the device name that Hyper-V will see so name your teams intuitively and take note of which driver number is assigned to which team (multiplexor driver #2, etc).

In the Hyper-V Virtual Switch Manager, under the external network connection type you will see every physical NIC in the system as well as a multiplexor driver for every NIC team. Checking the box below the selected interface does exactly what it suggests: shares that connection with the management OS (parent).

VLANs (non Hyper-V)

VLANs can be configured a number of different ways depending on your requirements. The highest level of methods can be configured in the NIC team itself. From within the Adapters and Interfaces dialog, click the Team Interfaces tab, then right click the team you wish to configure.

Entering a specific VLAN will limit all traffic on this team accordingly. If you intend to use this team with a Hyper-V switch DO NOT DO THIS! It will likely cause confusion and problems later. Leave any team to be used in Hyper-V in default mode with a single team interface and do your filtering within Hyper-V.

Team Interfaces can also be configured from this dialog which will create vNICs tied to a specific VLAN. This can be useful for specific services that need to communicate on a dedicated VLAN not part of Hyper-V. First select the team on the left, then right and the Add Interface drop down item will become active.

Name the interface and set the intended VLAN. Once created these will also appear as multiplexor driver devices in the network connections list. They can then be assigned IP addresses the same as any other NIC. vNICs created in this manner are not intended for use in Hyper-V switches!

VLANS (Hyper-V)

For Hyper-V VLAN assignments, the preferred method is to let the Hyper-V switch perform all filtering. This varies a bit depending on if you are assigning management VLANs to the Hyper-V parent or to guest VMs. For guest VMs, VLAN IDs should be specified within vNICs connected to a Hyper-V port. If multiple VLANs need to be assigned, add multiple network adapters to the VM and assign VLAN IDs as appropriate. This can also be accomplished in PowerShell using the Set-VMNetworkAdapterVlan command.

NIC teaming can also be enabled within a guest VM by configuring the advanced feature item within the VM’s network adapter settings. This is an either/ or scenario, guest vNICs can be teamed if there is no teaming in the Management OS.

Assigning VLANs to interfaces used by the Hyper-V host can be done a couple of different ways. Most basically, if your Hyper-V host is to share the team or NIC of a virtual switch this can be specified in the Virtual Switch Manager for one of your external vSwitches. Optionally a different VLAN can be specified for the management OS.

Converged vs Non-converged

Before I go much further on carving VLANs for the management OS out of Hyper-V switches, let’s look at a few scenarios and identify why we may or may not want to do this. Thinking through some of these scenarios can be a bit tricky conceptually, especially if you’re used to working with vSphere. In ESXi all traffic sources and consumers go through a vSwitch, in all cases. ESXi management, cluster communications, vMotion, guest VM traffic…everything. In Hyper-V you can do it this way, but you don’t have to, and depending on how or what you’re deploying you may not want to.

First let’s look at a simple convergence model. This use case applies to a clustered VDI infrastructure with all desktops and management components running on the same set of hosts. 2 10Gb NICs configured in a team, all network traffic for both guest VMs and parent management will traverse the same interfaces, storage traffic split out via NPAR to MPIO drivers. The NICs are teamed for redundancy, guest VMs attach to the vSwitch and the management OS receives weighted vNICs from the same vSwitch. Once a team is assigned to a Hyper-V switch, it cannot be used by any other vSwitch.

Assigning vNICs to be consumed by the management OS is done via PowerShell using the Add-VMNetworkAdapter command. There is no way to do this via the GUI. The vNIC is assigned to the parent via the ManagementOS designator, named whatever you like and assigned to the virtual switch (as named in Virtual Switch Manager).

Once the commands are successfully issued, you will see the new vNICs created in the network connections list that can be assigned IP addresses and configured like any other interface.

You can also see the lay of the land in PowerShell (elevated) by issuing the Get-VMNetworkAdapter command.

Going a step further, assign VLANs and weighted priorities for the management OS vNICs:

Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName “Cluster” -Access -VlanId 25
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName “Live Migration” -Access -VlanId 50
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName “Management” -Access -VlanId 10

Set-VMNetworkAdapter -ManagementOS -Name “Cluster” -MinimumBandwidthWeight 40
Set-VMNetworkAdapter -ManagementOS -Name “Live Migration” -MinimumBandwidthWeight 20
Set-VMNetworkAdapter -ManagementOS -Name “Management” -MinimumBandwidthWeight 5

For our second scenario let’s consider a non-converged model with 6 NICs. 2 for iSCSI storage, 2 for the Hyper-V switch and 2 for the management OS. The NICs used to access any Ethernet based storage protocol traffic should not be teamed, let MPIO take care of this. The NIC team used by the Hyper-V switch will not share with any other host function, the same goes for the management team.

This is a perfectly acceptable method of utilizing more NICs on a Hyper-V host if you haven’t bought into the converged network models. Not everything must go through a vSwitch if you don’t want it to. Flexibility is good but this is the point that may confuse folks. vNICs dedicated to the management OS attached to the Mgmt NIC team can be carved out via PowerShell or via the Teaming GUI as Team Interfaces. Since these interfaces will exist and function outside of Hyper-V, it is perfectly acceptable to configure them in this manner. Storage traffic should be configured using MPIO and not NIC teaming.

The native Server 2012 NIC teaming offering is powerful and a much welcomed addition to the Server software stack. There are certainly opportunities for Microsoft to enhance and simply this feature further by adding best practices gates to prevent misconfiguration. The bandwidth weighting calculations could also be made simpler and more in line with WYSIWYG. Converged or non, there are a number of ways to configure teaming in Server 2012 to fit any use case or solution model with an enhanced focus paid to Hyper-V.