Automate the Deployment of Azure Kubernetes Services Cluster + Application Gateway Ingress Controller

This post will demonstrate how to deploy a AKS cluster using Advanced Networking. We will then deploy an Application Gateway Ingress Controller. Essentially this will install a dedicated ingress POD that fully manages the Application gateway.

This means all entries in the Application gateway are 100% managed by AKS. If you manually add an entry to the AG, it will be removed by AKS Ingress Controller.

Overview

Considerations

  • Decided to dedicate an entire /16 IP range to the AKS cluster for simplicity e.g. 10.69.0.0/16.
  • Leverage AKS with Advanced Networking (CNI).
  • CNI provided the use of Application Gateway with WAFv2.
  • SSL offloading is configured. The actual private key (PEM – Base64 encoded) is stored in the default namespace in AKS. Whenever you deploy a new application, just –export (Deprecated) the key to the new namespace. The AG Ingress Controller will automatically be configured with the SSL certificate.
  • We will apply RBAC rules so AKS can manage the application gateway and VMSS scaleset.
  • RBAC to access container registry.

By using an Application Gateway, we can leverage additional benefits such as Web Application Firewall (V2), OWASP 3.0 firewall detection/prevention rules. Microsoft have totally refactors the AG WAF2 technology stack. It is much faster to provision and can deal with much larger amounts of traffic now.

By combining Load Balancing with WAF, we get the best of both worlds. If you have heavy traffic, it might be good to first do a performance test before making a final decision on AG + AKS stack.

Environment Setup + Tools

We are using AKS VMSS preview feature. Azure Virtual Machine Scale Sets have been around for a long time, and are in fact used by Microsoft Service Fabric. It makes total sense that this auto-scaling architecture is leveraged by AKS.

Due to the preview status of Container Services and VMSS+AKS, we will choose Azure CLI.

You can use Ubuntu Windows Shell or a Linux Ubuntu Shell.

Run the following code to setup your bash environment.

#!/bin/bash
echo "Updating system..."
sudo apt-get update
sudo apt-get upgrade

echo "Installing AzureCLI"
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

echo "Installing helm for AKS Admin"
curl -LO https://git.io/get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
helm init --service-account tiller --history-max 200

Helm is a client side tool to provide configuration settings to AKS. Tiller is a server side setting that runs on AKS that applies configuration settings that are applied from a helm client.

Create a config folder with 2 files.

Replace THECERTIFICATECHAIN with the contents of your base64 encoded .cer certificate chain. The script will replace <THEPRIVATEKEY> when you paste your private key. Future namespace or apps, will be able to find this in the default namespace. Thus a 1 time operations

e.g. (kubectl get secret rangerrom-tls –namespace=default ….).

apiVersion: v1
kind: Secret
metadata:
  name: rangerrom-tls
type: kubernetes.io/tls
data:
  tls.crt: THECERTIFICATECHAIN
  tls.key: THEPRIVATEKEY

rangerrom-tls.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

tiller-rbac.yml

Azure Prerequisites

Ensure you have
* VNET in a resounce group – ${env}-network-rg
* Subnet with a name ${env}-aks-cluster-subnet matching the IP rules

Install AKS into existing VNET

#!/bin/bash
aksversion='1.13.7'
while ! [[ "$env" =~ ^(sb|dv|ut|pd)$ ]] 
do
  echo "Please specifiy environment [sb, dv,ut,pd]?"
  read -r env
done 

case $env in

  dv)
    servicecidr="10.66.64.0/18"
    dnsserver="10.66.64.10"
    az account set --subscription 'RangerRom DEV'
    subscriptionid=$(az account show --subscription 'RangerRom DEV' --query id | sed  's/\"//g')
    ;;

  sb)
    servicecidr="10.69.64.0/18"
    dnsserver="10.69.64.10"
    az account set --subscription 'RangerRom SANDBOX'
    subscriptionid=$(az account show --subscription 'RangerRom SANDBOX' --query id | sed  's/\"//g')
    ;;

  ut)
    servicecidr="10.70.64.0/18"
    dnsserver="10.70.64.10"
    az account set --subscription 'RangerRom TEST'
    subscriptionid=$(az account show --subscription 'RangerRom TEST' --query id | sed  's/\"//g')
    ;;

  pd)
    servicecidr="10.68.64.0/18"
    dnsserver="10.68.64.10"
    az account set --subscription 'RangerRom PROD'
    subscriptionid=$(az account show --subscription 'RangerRom PROD' --query id | sed  's/\"//g')
    ;;  
  *)
    echo "environment not found"
    exit
    ;;
esac

env="rrau${env}"
location="australiaeast"

az group create --location $location --name "${env}-aks-rg"
sleep 5

az feature register -n VMSSPreview --namespace Microsoft.ContainerService
az provider register -n Microsoft.ContainerService

az aks create \
    --resource-group "${env}-aks-rg" \
    --name "${env}-aks-cluster" \
    --enable-vmss \
    --node-count 2 \
    --kubernetes-version $aksversion \
    --generate-ssh-keys \
    --network-plugin azure \
    --service-cidr $servicecidr \
    --dns-service-ip $dnsserver \
    --vnet-subnet-id "/subscriptions/${subscriptionid}/resourceGroups/${env}-network-rg/providers/Microsoft.Network/virtualNetworks/${env}-network/subnets/${env}-aks-cluster-subnet"



clusterprincipalid=$(az ad sp list --display-name ${env}-aks-cluster --query [0].objectId)
resourceGroupid=$(az group show --name ${env}-network-rg --query 'id')
echo "Configuring cluster to owner ${resourceGroupid}"
cmd="az role assignment create --role Contributor --assignee $clusterprincipalid --scope $resourceGroupid"
eval $cmd


echo "Configuring AKS Cluster with Tiller"
az aks get-credentials --resource-group "${env}-aks-rg" --name "${env}-aks-cluster" --overwrite-existing
kubectl apply -f ./config/tiller-rbac.yml
helm init --service-account tiller

while ! [[ ${#privatekey} -gt 2000 ]]
do
  echo "Please provide TLS Private Key - BASE64 Encoded PEM?"
  read -r privatekey
done

kubectl create namespace scpi-${env}
cat ./config/rangerrom-tls.yml  | sed "s/THEPRIVATEKEY/$privatekey/" > temptls.yml
kubectl apply -f temptls.yml -n default
#The Flag --export is going to be deprecated - Below is workaround.
kubectl get secret rangerrom-tls --namespace=default -o yaml | \
   sed '/^.*creationTimestamp:/d' |\
   sed '/^.*namespace:/d' |\
   sed '/^.*resourceVersion:/d' |\
   sed '/^.*selfLink:/d' |\
   sed '/^.*uid:/d' |\
   kubectl apply --namespace=scpi-${env} -f -

rm -f ./temptls.yml
privatekey=""

echo "Setup container registry permissions"
az acr create -n "${env}containerregistry" -g "${env}-common-rg" --sku Premium
containerid=$(az acr show -n ${env}containerregistry --query id)
principalidaks=$(az ad sp list --all --query "([?contains(to_string(displayName),'"${env}-aks-cluster"')].objectId)[0]")
cmd1="az role assignment create --role acrpull --assignee $principalidaks --scope $containerid"
eval $cmd1

echo "Setup container registry permissions - Centralised"
containerid=$(az acr show --subscription 'RangerRom PROD' -n rraupdcontainerregistry --query id)
cmd2="az role assignment create --role acrpull --assignee $principalidaks --scope $containerid"
eval $cmd2

Install and Configure AKS to control the Ingress

#!/bin/bash
while ! [[ "$env" =~ ^(sb|dv|ut|pd)$ ]] 
do
  echo "Ensure you have owner permissions on the subscription before you continue."
  echo "Please specifiy environment [sb, dv,ut,pd]?"
  read -r env
done 

case $env in

  dv)
    az account set --subscription 'RangerRom DEV'
    ;;

  sb)
    az account set --subscription 'RangerRom SANDBOX'
    ;;

  ut)
    az account set --subscription 'RangerRom TEST'
    ;;

  pd)
    az account set --subscription 'RangerRom PROD'
    ;;  
  *)
    echo "Invalid Environment"
    exit
    ;;
esac

env="rrau${env}"

ipAddressName="${env}-aks-application-gateway-ip"
resourcegroup="MC_${env}-aks-rg_${env}-aks-cluster_australiaeast"
gatewayname="${env}-aks-application-gateway"
location="australiaeast"
vnet="${env}-network"
subnet="${env}-aks-application-gateway-subnet"

az network public-ip create \
  --resource-group $resourcegroup \
  --name $ipAddressName \
  --allocation-method Static \
  --sku Standard

sleep 20

subnetid=$(az network vnet subnet show -g "${env}-network-rg" -n "${env}-aks-application-gateway-subnet" --vnet-name ${vnet} --query id)
cmd="az network application-gateway create --name $gatewayname \
                                      --resource-group $resourcegroup \
                                      --capacity 2  \
                                      --sku "WAF_v2" \
                                      --subnet $subnetid \
                                      --http-settings-cookie-based-affinity Disabled \
                                      --location $location \
                                      --frontend-port 80 \
                                      --public-ip-address $ipAddressName"
eval $cmd
az network application-gateway waf-config set -g $resourcegroup --gateway-name $gatewayname \
                            --enabled true --firewall-mode Detection --rule-set-version 3.0

#Setup AAD POD Identity to manage application gateway
az identity create -g $resourcegroup -n "${env}-aks-aad_pod_identity"
sleep 20

principalid=$(az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity" --query 'principalId')
appgatewayid=$(az network application-gateway show -g $resourcegroup -n $gatewayname --query 'id')

echo "Assign Role so AKS can manage the Application Gateway"

echo "Configuring Create Role for identity - $principalid - for gateway"
cmd="az role assignment create --role Contributor --assignee $principalid --scope $appgatewayid"
eval $cmd

resourceGroupid=$(az group show --name $resourcegroup --query 'id')

echo "Configuring Read Role for identity - $principalid - for gateway resourcegroup"
cmd="az role assignment create --role Reader --assignee $principalid --scope $resourceGroupid"
eval $cmd

az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity"
echo "Please use the azure identity details above to configure AKS via Help for the AG Ingress Controller"
echo "Careful with copy and paste. Hidden characters can affect the values!"


echo "Ingress Controller for Azure Application Gateway"
az aks get-credentials --resource-group "${env}-aks-rg" --name "${env}-aks-cluster"
kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

subscriptionid=$(az account show --query id | sed  's/\"//g')
appGatewayResourceId=$(az network application-gateway show -g $resourcegroup -n $gatewayname --query resourceGroup  | sed  's/\"//g')
identityClientid=$(az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity" --query clientId  | sed  's/\"//g')
aksfqdn=$(az aks show --resource-group "${env}-aks-rg" --name "${env}-aks-cluster" --query fqdn  | sed  's/\"//g')

cmd="helm upgrade ingress-azure application-gateway-kubernetes-ingress/ingress-azure \
     --install \
     --namespace default \
     --debug \
     --set appgw.name=$gatewayname \
     --set appgw.resourceGroup=$appGatewayResourceId \
     --set appgw.subscriptionId=$subscriptionid \
     --set appgw.shared=false \
     --set armAuth.type=aadPodIdentity \
     --set armAuth.identityResourceID=/subscriptions/$subscriptionid/resourcegroups/$appGatewayResourceId/providers/Microsoft.ManagedIdentity/userAssignedIdentities/$env-aks-aad_pod_identity \
     --set armAuth.identityClientID=$identityClientid \
     --set rbac.enabled=true \
     --set verbosityLevel=3 \
     --set aksClusterConfiguration.apiServerAddress=$aksfqdn"
eval $cmd
kubectl get pods


Conclusion

This post should provide a guide post to setup your infrastructure as code. By leveraging a rock solid naming convention, you can leverage fully automated scripts to deploy your environments. The above scripts for AKS and AG are also idempotent, so they can be run on a scheduled basis e.g. Azure Devops.

Advertisements

Automate #Azure Blob Snapshot backups with @Cerebrata

Hi,

Leveraging the cerebrata cmdlets for Azure, we can easily backup our blob containers via snapshot, this will prove useful for Page Blobs that are Random Access i.e. VHD’s on Cloud Drive

Here is how Purging Snapshots works

#requires -version 2.0
param (
	[parameter(Mandatory=$true)] [string]$AzureAccountName,
	[parameter(Mandatory=$true)] [string]$AzureAccountKey,
	[parameter(Mandatory=$true)] [array]$BlobContainers
)

$ErrorActionPreference = "Stop"

if ((Get-PSSnapin -Registered -Name AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue) -eq $null)
{
	throw "AzureManagementCmdletsSnapIn missing. Install them from Https://www.cerebrata.com/Products/AzureManagementCmdlets/Download.aspx"
}

Add-PSSnapin AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue

function SnapShotBlobContainer 
{
	param ( $containers, $blobContainerName )
	Write-Host "Starting snapshot $blobContainerName"

	$container = $containers | Where-Object { $_.BlobContainerName -eq $blobContainerName }

	if ($container -eq $null)
	{
		Write-Host  "Container $blobContainerName doesn't exist, skipping snapshot"
	}
	else
	{
        Write-Host  "Found blob container $blobContainerName"
Checkpoint-BlobContainer -Name $container.BlobContainerName -SaveSnapshotInformation -AccountName $AzureAccountName -AccountKey $AzureAccountKey
	Write-Host  "Snapshot complete for $blobContainerName"
	}
}

$containers = Get-BlobContainer -AccountName $AzureAccountName -AccountKey $AzureAccountKey
foreach($container in $BlobContainers)
{
	SnapShotBlobContainer $containers $container
}

Then just call the script with the params. remember an array of items is parsed in like this:

-BlobContainers:@(‘container1’, ‘contaner2’) -AzureAccountName romikoTown -AzureAccountKey blahblahblahblahblehblooblowblab==

Windows #Azure configuration transformations

Hi,

I needed to update the transformation today to support different csdef files for UAT/Test etc. I found myself forgetting the process, so i thought it would be a good idea to log the entries needed.

What I wanted was a way to disable New Relic in our performance environment, since we do not have a license key.

Since we use the Azure Tasks to run batch jobs before the worker role starts, it made sense that I create a TASK Environment variable that my batch script can check and see if it should install New relic, e.g.

if "%MYSTORYENV%" == "PERF" goto :EOF

So, in the above, my batch file startup.cmd will skip installing new relic if the environment variable is PERF. However we need to set this value in the csdef file.

So we go to the BASE servicedefinition.csdef file and have this entry for it.

<sd:Startup>
      <sd:Task commandLine="Startup.cmd" executionContext="elevated" taskType="background">
        <sd:Environment>
          <sd:Variable name="EMULATED">
            <sd:RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </sd:Variable>
          <sd:Variable name="MYSTORYENV" value="DEV" />
        </sd:Environment>
      </sd:Task>
    </sd:Startup>

Notice, that I have qualified all my csdef entries, this is important for transformations to occur (sd:)

Ok, the next step is that we create a transformation file

https://gist.github.com/1777060

Now, that we have this transform, we will need to edit the CSPROJ file. Please see below the parts added

Item Groups

<ItemGroup>
<ServiceDefinition Include="ServiceDefinition.csdef" />
<ServiceConfiguration Include="ServiceConfiguration.cscfg" />
</ItemGroup>
<ItemGroup>
<EnvironmentDefinition Include="ServiceDefinition.uat.csdef">
<BaseConfiguration>ServiceDefinition.csdef</BaseConfiguration>
</EnvironmentDefinition>
<EnvironmentDefinition Include="ServiceDefinition.perf.csdef">
<BaseConfiguration>ServiceDefinition.csdef</BaseConfiguration>
</EnvironmentDefinition>
<EnvironmentConfiguration Include="ServiceConfiguration.uat.cscfg">
<BaseConfiguration>ServiceConfiguration.cscfg</BaseConfiguration>
</EnvironmentConfiguration>
<EnvironmentConfiguration Include="ServiceConfiguration.perf.cscfg">
<BaseConfiguration>ServiceConfiguration.cscfg</BaseConfiguration>
</EnvironmentConfiguration>
<None Include="@(EnvironmentConfiguration)" />
<None Include="@(EnvironmentDefinition)" />
</ItemGroup>

Notice I have the include at the bottom, so I can see these in Visual Studio. I also have transformations for cscfg files, hence the reason why you see them here Smile

Targets Validation

<Target Name="ValidateServiceFiles"
		Inputs="@(EnvironmentConfiguration);@(EnvironmentConfiguration->'%(BaseConfiguration)');@(EnvironmentDefinition);@(EnvironmentDefinition->'%(BaseConfiguration)')"
		Outputs="@(EnvironmentConfiguration->'%(Identity).transformed.cscfg');@(EnvironmentDefinition->'%(Identity).transformed.csdef')">

	<Message Text="ValidateServiceFiles: Transforming %(EnvironmentConfiguration.BaseConfiguration) to %(EnvironmentConfiguration.Identity).tmp via %(EnvironmentConfiguration.Identity)" />
	<TransformXml Source="%(EnvironmentConfiguration.BaseConfiguration)" Transform="%(EnvironmentConfiguration.Identity)" Destination="%(EnvironmentConfiguration.Identity).tmp" />
	<Message Text="ValidateServiceFiles: Transformation complete; starting validation" />

	<Message Text="ValidateServiceFiles: Transforming %(EnvironmentDefinition.BaseConfiguration) to %(EnvironmentDefinition.Identity).tmp via %(EnvironmentDefinition.Identity)" />
	<TransformXml Source="%(EnvironmentDefinition.BaseConfiguration)" Transform="%(EnvironmentDefinition.Identity)" Destination="%(EnvironmentDefinition.Identity).tmp" />
	<Message Text="ValidateServiceFiles: Transformation complete; starting validation" />

	<ValidateServiceFiles ServiceDefinitionFile="@(ServiceDefinition)" ServiceConfigurationFile="%(EnvironmentConfiguration.Identity).tmp" />
	<ValidateServiceFiles ServiceDefinitionFile="%(EnvironmentDefinition.Identity).tmp" ServiceConfigurationFile="@(ServiceConfiguration)" />
	<Message Text="ValidateServiceFiles: Validation complete; renaming temporary file" />

	<Move SourceFiles="%(EnvironmentConfiguration.Identity).tmp" DestinationFiles="%(EnvironmentConfiguration.Identity).transformed.cscfg" />
	<Move SourceFiles="%(EnvironmentDefinition.Identity).tmp" DestinationFiles="%(EnvironmentDefinition.Identity).transformed.csdef" />
</Target>

Notice above I have them for BOTH CSCFG and CSDEF files!

Move transforms to the app.publish folder for azure packaging

<Target Name="MoveTransformedEnvironmentConfigurationXml" AfterTargets="AfterPackageComputeService" Inputs="@(EnvironmentConfiguration->'%(Identity).transformed.cscfg')" Outputs="@(EnvironmentConfiguration->'$(OutDir)app.publish\%(filename).cscfg')">
<Move SourceFiles="@(EnvironmentConfiguration->'%(Identity).transformed.cscfg')" DestinationFiles="@(EnvironmentConfiguration->'$(OutDir)app.publish\%(filename).cscfg')" />
<Move SourceFiles="@(EnvironmentDefinition->'%(Identity).transformed.csdef')" DestinationFiles="@(EnvironmentDefinition->'$(OutDir)app.publish\%(filename).csdef')" />
</Target>

Summary

So there you have it, you will now have csdef and cscfg files for different environments.

Windows #Azure–Pre Role Startup Tasks

Hi,

Imagine you need to boot up a web role in the cloud, but before the global.asax events kick in or even lower, before the WebRoleEntryPoint events kick in, you need to do some installations of prerequisite software.

The best way to go about doing this is to register a task in the ServiceDefinition.csdef file. Lets imagine we need to run a batch file that will do some sort of installation, say a monitoring service that is required to be installed BEFORE IIS starts our web application, so that it can get a hook point, say New Relic!

Below is a configuration example that will do this for you.

https://gist.github.com/1775222

You can also set elevation privileges, which are required if you are running PowerShell scripts etc.

<Task commandLine="Startup.cmd" executionContext="elevated" taskType="background">

You can read more about this here:

http://msdn.microsoft.com/en-us/library/windowsazure/hh124132.aspx

http://msdn.microsoft.com/en-us/library/windowsazure/gg456327.aspx

http://msdn.microsoft.com/en-us/library/windowsazure/gg432991.aspx

So, I hope you now have a cool way to bootstrap your prerequisite software before IIS kicks in.

Windows Azure, which affinity group or region is best for you?

A nice tool to use to see which Azure Affinity Group to use e.g. South America, North America or Asia is to download this tool and run checks from where your clients will be based.

http://research.microsoft.com/en-us/downloads/5c8189b9-53aa-4d6a-a086-013d927e15a7/default.aspx

Once you got it installed, add your storage accounts and then get started.

image

So above we will test from Sydney Australia to our UAT environment in America.

Lets click “run”

It will start executing the test, this is now a good time to plan a date, make a cup of coffee or write some JScript for your open source projects.

Results:

Sydney to North America

image

image

 

Sydney to South East Asia (Singapore)

image

image

Conclusion

For us, South East Asia was far more better (Web site, download is more important than upload), and the proof was in the pudding when we measures web site response times with MVC MiniProfiler.

However, this is not the ultimate conclusion, I bet these response times will vary depending on time of day, perhaps when Asia is awake and US is asleep, it could be the other way round, so test it at different times of day and pick the affinity or region that is best for you!