[How To] Prevent Container with Privileged Mode to Run on Kubernetes Cluster

My Kubernetes exploration brought me to the topic of privileged pods. Privileged pod, or container running in privileged mode is a configuration option of K8s deployment which can be useful, but can also be dangerous. 

This is an excerpt of deployment specification where privileged mode defined.

kind: Deployment
...
spec:
  template:
    spec:
      containers:
      - name: ...
        image: nginx:1.14.2
        securityContext:
          privileged: true

Found this article:

https://www.cncf.io/blog/2020/10/16/hack-my-mis-configured-kubernetes-privileged-pods/

which explains about what's the true intent of running privileged pod and the security risk it caused, including how to exploit privileged pod to do malicious intent.

Now the question is, how can we prevent container with privileged mode to run on our Kubernetes cluster?

[lunar.lab] Cannot Resolve ".local" Domain from TKGm Workload Cluster

Problem Statement

• Kubernetes Pod Status ImagePullBackOff 
• Describe pod show error message:

dial tcp: lookup harbor-01a.corp.local: Temporary failure in name resolution

• Container image pulled from local container registry with ".local" domain suffix

[lunar.lab] Allow TKGm Workload Cluster to Pull Image from Harbor Configured with Self-signed Certificate

Disclaimer

• This method is kind of a hack and hence ** Unsupported **.
• I do this only within my lab or PoC with controlled environment.

Problem Statement

TKGm Workload Cluster do not allow pulling image from Container Registry configured with Self-signed Certificate.

Doing such thing will throw error message as follows:

x509: certificate signed by unknown authority

[How To] Enhance Online Boutique App to Use Persistent Volume

Online Boutique (https://github.com/GoogleCloudPlatform/microservices-demo) is a web-based e-commerce microservices demo app built by folks at Google. I use this as demo app to deploy on top of Tanzu Kubernetes platform. One of the demo scenario I do is how to consume vSphere datastore as persistent storage for Kubernetes app, in easy, on-demand, fully automated, and scalable fashion. This can be done by a feature called Cloud Native Storage (CNS). Read more about CNS here:

https://blogs.vmware.com/virtualblocks/2019/08/14/introducing-cloud-native-storage-for-vsphere/

One of Online Boutique service is redis-cart. This is the service in charge for Shopping Cart. If any item added to Shopping Cart, the record will be handled by this service. With default configuration, the data volume used by redis-cart do not use persistent volume. If redis-cart is failed, Shopping Cart data will be lost. This article explains how to alter this and use vSphere datastore to provide persistent storage for redis-cart service.

[How To] Avoid Hitting Docker Pull Rate Limit by Authenticate Pull Request

When demoing kubernetes platform, I definitely need sample application to deploy. There are some great reference here: https://williamlam.com/2020/06/interesting-kubernetes-application-demos.html, where most of the source container images are coming from Docker registry. If you try to deploy the app manifests, you might hit error like the following:

429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit.

[lunar.lab] Deploy TKG Management Cluster on vSphere

Now all the preparation completed, I finally able to deploy TKG management cluster. The recommended (and easiest) way to do this for the first time is using the installer interface. From the bootstrap machine prepared earlier (https://dy.si/TAg1M72), I type this:

tanzu management-cluster create --ui --browser none --bind 192.168.110.101:8081

• 192.168.110.101 is my bootstrap machine IP Address
• I set the installer interface to be accessible on port 8081
• --browser none to tell the installer not to open browser locally as my bootstrap machine is without GUI
• Open http://192.168.110.101:8081 on a browser with network access to bootstrap machine
• As always, this is the official documentation that I refer to: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-mgmt-clusters-deploy-ui.html

Step 1 - IaaS Provider

[lunar.lab] Prepare to Deploy TKG Management Clusters to vSphere

• Created a base image template that matches the management cluster’s Kubernetes version.
• Official documentation: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-mgmt-clusters-vsphere.html#import-a-base-image-template-into-vsphere-4
• Import latest available version and -1 to test version upgrade
• In my case for TKGm 1.5.1, I import version Kubernetes v1.21.8 and v1.22.5
• I found out that Ubuntu template is not recognized when trying to deploy workload cluster from TMC, so I also import Photon image.

[lunar.lab] Install Harbor Container Registry as Docker Containers

This is an installation note of Harbor container registry for lunar.lab. Keeping in mind to minimize footprint due to resource constraint, I decided to install Harbor service in bootstrap machine VM which I already deployed (See here for the article: https://dy.si/TAg1M72).

• Official documentation followed for this purpose can be found here: https://goharbor.io/docs/2.5.0/install-config/
• List of software version:
• Harbor 2.5.0
• Docker Engine 20.10.16
• Docker Compose 2.5.0
• For a more proper way to deploy Harbor in TKG, you may want to check this official documentation:
• TKGm: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-packages-harbor-registry.html
• TKGs: https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-2B3F2498-7BE3-4179-9F92-C83902061D42.html  

[lunar.lab] Prepare Bootstrap Machine for TKGm Deployment

Having a bootstrap machine is one of the step required for deploying TKGm to vSphere as stated here:

https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-mgmt-clusters-vsphere.html

This article will run through the step I took to configure that bootstrap machine which involves these following 4 steps.

Step 1 - Starting Point: Ubuntu VM

I create Ubuntu VM from scratch with the following configuration:

• Virtual Hardware specification
• 8 vCPU
• 8GB RAM
• 40GB disk
• Ubuntu 20.04
• Minimum install + OpenSSH
• Configure static IP
• Configure Internet access (using proxy)

Once VM created on vSphere, boot from Ubuntu 20.04 installation image, then all the above settings can be configured easily through the installation wizard. Pretty straightforward.

Help! Where is my tanzu cluster Plugin?

So you have lost your tanzu cluster Plugin after upgrading Tanzu CLI?

I tried to upgrade my TKG from version 1.4.2 to 1.5.1. One of the first step is to upgrade Tanzu CLI. Once upgraded, I realize that tanzu cluster plugin is missing! 😱 How can I manage my TKG cluster then? This did not happen when I did the upgrade from 1.3.1 to 1.4.2! 

Help, I Cannot Pass IaaS Provider Step when Deploying TKG 1.4.2 Management Cluster to vSphere!

In TKG 1.4, if you deploy management clusters to vSphere with the installer interface, as a first step of configuring vCenter Server as IaaS Provider, you’ll need to fill in your vCenter Server IP Address or FQDN, username/password then hit Connect. If your vCenter use the default certificate, you’ll found this error:

Failed to connect to the specified vCenter Server. Post "https://IP_or_FQDN/sdk": x509: cannot validate certificate for IP_or_FQDN because it doesn't contain any IP SANs

[lunar.lab] Configuring NSX Advanced Load Balancer for Tanzu Kubernetes Grid (TKG) on VMware vSphere

Disclaimer:

• This is for lunar.lab only and not intended as a guide for production environment. 
• All deployed on vSphere environment. 
• I use nested environment.
• I use vSphere 7, NSX ALB 20.1.6, TKGm 1.5.1 
• The official documentation is here: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-mgmt-clusters-install-nsx-adv-lb.html

To support my learning as part of VMware SEAK (South East Asia & Korea) Tanzu Take-12 Program, I try to build my own lab. I don't have the luxury of physical lab on my own, so I build on nested environment my company provided. Here is the first of my documentation.

I started with a base pod which provide me a working vSphere cluster. The networking setup is one distributed switch with following portgroup:

• ESXi: management network vmkernel - 192.168.110.0/24
• vMotion: vmotion vmkernel - 10.10.30.0/24
• storage: storage vmkernel - 10.10.20.0/24
• VM: VM management network - 192.168.110.0/24
• tkg-vip-network: load balancer/ingress virtual IP - 192.168.120.0/24
• tkg-network: management/workload cluster nodes - 192.168.100.0/24
• avi-internal: placeholder network for ALB Service Engine

All subnet are routable.