Share IT!: [lunar.lab] Configuring NSX Advanced Load Balancer for Tanzu Kubernetes Grid (TKG) on VMware vSphere

Disclaimer:

This is for lunar.lab only and not intended as a guide for production environment.
All deployed on vSphere environment.
I use nested environment.
I use vSphere 7, NSX ALB 20.1.6, TKGm 1.5.1
The official documentation is here: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-mgmt-clusters-install-nsx-adv-lb.html

To support my learning as part of VMware SEAK (South East Asia & Korea) Tanzu Take-12 Program, I try to build my own lab. I don't have the luxury of physical lab on my own, so I build on nested environment my company provided. Here is the first of my documentation.

I started with a base pod which provide me a working vSphere cluster. The networking setup is one distributed switch with following portgroup:

ESXi: management network vmkernel - 192.168.110.0/24
vMotion: vmotion vmkernel - 10.10.30.0/24
storage: storage vmkernel - 10.10.20.0/24
VM: VM management network - 192.168.110.0/24
tkg-vip-network: load balancer/ingress virtual IP - 192.168.120.0/24
tkg-network: management/workload cluster nodes - 192.168.100.0/24
avi-internal: placeholder network for ALB Service Engine

All subnet are routable.

For TKG deployment, I would need the following networking services:

L4 load balancer for the control plane, both for management and workload cluster
Kubernetes L4 load balancer service
Kubernetes ingress service
IPAM for automated service engine provisioning
DNS for Kubernetes ingress service

Here in lunar.lab, those requirements will be fulfilled by NSX ALB (was AVI).

Step 1 - Deploy NSX ALB Controller

Deploy NSX ALB Controller from OVF Template

Fill in all details (ovf file, vm name, compute/storage/network deployment destination)
Customize template

You’ll only need to configure the first three parameters. These are networking settings for ALB Controller management network. ALB Controller management interface should be able to reach vCenter Server.

Step 2 - Initial Setup

Once deployed, power On the controller, it might take a while until fully booted up. From a browser, open the IP Address of NSX ALB Controller. You’ll be asked to configure admin password. Perform the initial configuration.

Fill in the passphrase, this is mandatory.
Optional – Configure DNS Resolvers and search domain
I don’t have SMTP server on my lab, select None.
Keep default option for Multi Tenant settings
Select Setup Cloud After
Select Save

Step 3 - Configure Cloud Endpoint

Select vCenter Server as cloud endpoint and provide login information for your vCenter Server, select Write for acces permission. Ignore other settings for now and select Save.

Step 4 – Network Profile Preparation

Navigate to Infrastructure\Network. Configure subnet/static IP Pool for following network:

Management Network for Service Engine
TKG Nodes
VIP for LB and Ingress service

Sample of creating IP Pool 192.168.120.101 to 192.168.120.200 for VIP.

Step 5 – Create IPAM Profile

Navigate to Templates\IPAM/DNS Profiles, create IPAM Profile. For Usable Network, select port group you would like your load balancer VIP resides.

Step 6 – Create DNS Profile

Navigate to Templates\IPAM/DNS Profiles, create DNS Profile. Domain Name is the subdomain which you want NSX ALB to handle. In this case tkg.corp.local.

Step 7 – Edit Cloud Endpoint

Navigate to Infrastructure\Clouds, edit the Default Cloud. Select IPAM/DNS profile you created on previous steps on the IPAM/DNS section.

At Data Center tab, NSX ALB Controller will automatically retrieve inventory from the vCenter Server you configured. Select the correct Data Center and select DHCP Enabled. Leave other option as default. Select Network tab.

On Management Network field, select port group which you want your Service Engine management network to connect to. On IP Address Management for Management Network section, make sure both check box unselected and select the appropriate IP Subnet and Static IP Pool for the Management Network you selected above. Key in the correct Default Gateway.

Step 8 – Enable Basic Authentication

Navigate to Administration\Settings\Access Settings, click the pencil icon to edit settings, enable Allow Basic Authentication.

Step 9 – Create System DNS Virtual Service

Navigate to Applications\Virtual Services, select Create Virtual Service, select Advanced Setup.

Provide Virtual Service name, select System-DNS as Application Profile. Provide IP Address and Application Domain Name for this Virtual Service. On Advanced tab, select corresponding portgroup for the Placement Network, which is the Subnet you choose for this Virtual Service's IP Address. Leave other settings as default.

Please note that Application Domain Name automatically entered with the domain name configured in Step 5.

Step 10 – Confirm Service Engine and Virtual Service Creation

First Virtual Service creation will also deploy Service Engine. Navigate to Infrastructure\Service Engine. Wait until two Service Engine up and running.

Navigate to Application\Virtual Services. Wait until System DNS Virtual Services you create up and running.

Step 11 – Create DNS Delegation

In my lab, parent domain corp.local manage by Microsoft DNS. For the purpose of TKG, I need to create DNS delegation for subdomain tkg.corp.local on ALB.

Go to DNS Manager and create new delegation in the target parent domain. Key in the subdomain name and Name Server.

Complete

As a last note, a specific supported NSX ALB version is required for TKG deployment. Find the compatibility in TKG release notes. Below is the release notes for TKG 1.5.1 which used in this lab.