SAP HANA TDI on Cisco UCS and VMware vSphere - Part 2

ESXi Host

UCS Service Profile

The ESXi host provides the platform where virtual machines are running on. The service profile contains the configuration of the hardware in a Cisco UCS environment. Service Profile Templates or vSphere Auto Deploy can be used to ease the ESXi deployment process. In this example, a standalone service profile creation is shown.

For each vSwitch, it is recommended to configure two uplink interfaces with MTU 9000 as trunk. The VLAN assignment takes place in the port group configuration of the vSwitch.

In order to get the best performance for virtualization, certain BIOS features should be enabled. The c-states can be controlled by the hypervisor and do not necessarily have to be disabled. It depends on the performance needs vs. power saving aspects how balanced this configuration should be.

VMware vSphere screenshot

Although the use of VM-FEX is optional, it is recommended to enable all Intel Direct IO features.

NUMA Locality

It is beneficial for HANA when a VM with 2 vSockets get scheduled on the same physical socket rather than being spread out to remote sockets. While distributing workload of the same VM on multiple sockets to get the full computing power of the core might be beneficial for other workloads, HANA benefits largely from L3 caching effects  and therefore the VM should stick to its sockets as much as possible. Using sched.vcpu.affinity per VM is a quite harsh way to configure NUMA locality and also limits VM operational tasks, so let's look at another more gentle approach with quite the same benefits.

VMware KB 2003582 describes the two parameters numa.vcpu.preferHT and numa.preferHT, one for a single VM, the other for the entire ESXi host. Using one of these parameters solves the problem of vCPUs getting scheduled to distant NUMA nodes and therefore leverages L3 caching effects for the VM, too.

Network

SAP HANA has different types of network communication channels to support the different SAP HANA scenarios and setups.

Source: SAP SE

On the basis of the listed network requirements, every server must be equipped with two 1 or 10 Gigabit Ethernet (10 Gigabit Ethernet is recommended) interfaces for scale-up systems to establish communication with the application servers and users (client zone). If the storage for SAP HANA is external and accessed through the network, two additional 10 Gigabit Ethernet or 8-Gbps Fibre Channel interfaces are required (storage zone). Scale-out systems need a 10 Gigabit Ethernet link for internode communication (internal zone). When using multiple ESXi hosts in a vSphere Cluster with enabled DRS, at least one additional 10 Gigabit Ethernet link is required for vMotion traffic.

For the internal zone, storage zone and the vMotion network, it is recommended to configure jumbo frames end-to-end.

The network traffic can be consoliated by using the same vSwitch with several load-balanced uplinks.

Regarding the network adapter placement, the devices might show up in an unexpected order. In order to align vNIC and vmnic enumeration, apply vNIC Placement Policy according to the UCS Manager Server Management Guide and follow VMware KB 2091560 and KB 2019871.

Storage

The storage system must be certified as part of an appliance, independent of the appliance vendor, or must be certified as SAP HANA TDI Storage in the Certified SAP HANA Hardware Directory.

It is recommended to physically separate the the origin (VMFS LUN or NFS export) of the datastores providing data and log for performance reasons. Additional performance classes can be distinguished:

               

Category	Read Performance	Write Performance
OS boot disk	medium	low
/hana/shared	medium	low
/hana/data	very high	high
/hana/log	high	very high
backup	low	medium

Source: scn.sap.com