Patrick Debois from Belgium is the actual culprit to blame for the term Devops, he wanted more synergy between developers and operations back in 2007.
Fast-forward a few years and now we have “Devops” everywhere we go. If you using the coolest tools in town such as Kubernetes, Azure Devops Pipelines, Jenkins, Grafana etc – then you probably reckon that you are heavy into Devops. This can not be further from the truth.
The fact is that Devops is more about a set of patterns and practices within a culture that nurtures shared responsibilities across all teams during the software development life-cycle.
Put it this way, if you only have 1 dude in your team that is “doing Devops”, then you may want to consider if you are really implementing Devops or one of it’s anti-patterns. Ultimately you need to invest in everyone within the SDLC teams to get on board with the cultural shift.
If we cannot get the majority of engineers involved in the SDLC to share responsibilities, then we have failed at our objectives regarding Devops, even if we using the latest cool tools from Prometheus to AKS/GKE. In a recent project that I was engaged in there was only 1 devops dude, when he fell ill nobody from any of the other engineering teams could perform his duties. Despite the fact that confluence has numerous playbooks and “How To’s”. Why?
It comes down to people, process & culture. All of which can be remedied with strong technical leadership and encouraging your engineers to work with the process and tools in their daily routine. Hence why I encourage developers that are hosting their code on Kubernetes to use Minikube on their laptops.
If there is any advice that I can provide teams that want to implement Devops – Focus on People then Process and finally the Tools.
In order to setup the transition for success – we will discuss in the next part of this series the pillars of Devops.
This is a visual guide to compliment the process of setting up your own Kubernetes Cluster on Google Cloud. This is a visual guide to Kelsey Hightower GIT project called Kubernetes The Hard Way. It can be challenging to remember all the steps a long the way, I found having a visual guide like this valuable to refreshing my memory.
Provision the network in Google Cloud
VPC
Provision Network
Firewall Rules
External IP Address
Provision Controllers and Workers – Compute Instances
Generating Kubernetes Configuration Files for Authentication
Generating the Data Encryption Config and Key
Bootstrapping etcd cluster
Use TMUX set synchronize-panes on to run on multiple instances at same time. Saves time!
Notice where are using TMUX in a Windows Ubuntu
Linux Subsystem and running commands in parallel to save a lot of time.
The only manual command is actually ssh into each controller, once in, we activate tmux synchronize feature. So what you type in one panel will duplicate to all others.
Bootstrapping the Control Pane (services)
Bootstrapping the Control Pane (LB + Health)
Required Nginx as Google health checks does not support https
Once you have completed the install of your kubernetes cluster, ensure you tear it down after some time to ensure you do not get billed for the 6 compute instances, load balancer and public statis ip address.
A big thank you to Kelsey for setting up a really comprehensive instruction guide.
As a developer you can deploy your docker containers to a local Kubernetes cluster on your laptop using minikube. You can then use Google Cloud Code extension for Visual Studio Code.
You can then make real time changes to your code and the app will deploy in the background automatically.
We can see our new service is being deployed by VSCode Cloud Code extension. Whenever we make changes to the code, it will automatically deploy.
minikube service nodejs-hello-world-external --url
The above command will give us the url to browse the web app.
If I now change the text for Hello, world! It will automatically deploy. Just refresh the browser 🙂
Here in the status bar we can see deployments as we update code.
Debugging
Once you have deployed your app to Minikube, you can then kick off debugging. This is pretty awesome. Basically your development environment is now a full Kubernetes stack with attached debugging proving a seamless experience.
This article will discuss a minimum complex design if you have several SPA Apps or other Apps that call a global API . This design is future proof for when you later need to introduce an API gateway, since an API gateway uses a similar concept of 1 access token for all API’s.
If you have a new in house client app that requires AAD integration for Authentication and Authorization and your apps are all exposed within your organisation network only.
Anyone in the organisation can create apps, however it is important to adhere to some conventions, expecially if you want to reduce the costs around onboarding new apps and the administrative overhead.
If you can affort it, I would recommend https://www.okta.com/ , currently AAD still has a far higher OPEX cost associated with the day to day operations regarding Identity Management versus Azure AD. So lets get started with the Implicit Grant Flow for SPA Apps. Hopefully Microsoft will support the Code Authorization with PKCE for SPA’s soon.
Diagram showing the Implciti Grant Flow in an AAD environment
Design
App registrations need to be created per environment (Dev, QA, UAT, Prod).
All physical API’s you host are linked to 1 App Registration in AAD (Same Audience)
All Clients have their own App registration using the API Audience
API exposes a default scope – Later when you implement an API gateway SCOPE logic can reside on the gateway. This keeps the design future proof.
1 Global API App Registration per environment (Global API)
Application Roles are ONLY created in the Global API APp registration
AD Groups are assigned to Application Roles
Diagram Illustration the AAD Relationship in this design. We only support 1 Audience (The Global API).
Diagram illustrating the AAD implementation Stategy regarding Users, Groups, Apps and Application Roles (API)
Implicit Grant SPA ->Global API
Prerquisite – You already have a Global API App Registration that exposes an API in AAD.
Go to the Azure Portal
Click “App Registrations”Click “New Registration”
Provide a name that adheres to the convention – Department <env> <AppName>SPA
Once the app is created
Go to the Branding Blade and ensure the Domain is verified and select your domaiun name.
Add any custom branding e.g. LogoClick Authentication Blade
Enable Implicit Grant Flow by ticking – ID Token, Access Token
Record the APP ID by clicking Overview and copying the Application (client) ID
If you do not want a consent dialog to show when the user uses the app. You need to add your shiny new app to the Global API app as a trusted application. Each environment has a GLOBAL API e.g. Department <envcode> API
Navigate to Department<envcode> API
Click “Expose an API
”Click “Add a client Application”
There is another method to add API permissions via the SPA App instead of trusted clients, however this required Global Tenant Admin grants.
Add the API’s in the API Permissions blade then
Click the Grant/revoke admin consent for XYZ Company Limited button, and then select Yes when you are asked if you want to grant consent for the requested permissions for all account in the tenant. You need to contact an Azure AD tenant admin to do this.
All Access Tokens provided by the Global API are valid for 1 hour. You will need to refresh the token for a new one. The library will do this for you.
Ensure all Tokens are Verified on the SPA side (ID Token and Access Token)
NEVER send the ID Token to the API. Only the Access Token (Bearer)
AVOID storing the access token in an insecure storage location
Global API Manifest (Global API App Registration)
Create your Global API
Ensure the Branding is correct
Expose an API e.g. default
Add Trusted Client Apps like the SPA above
Edit the Manfiest to include any Application or User Roles. Application Roles are great for service to service communication which is not covered in this article.
Here is where you define roles in the app manifest on the Global API App.
Your API will implement RBAC based on the App Roles defined in the Global API Manifest. In AAD Portal, add additional App Roles
[Authorize(Roles = "Manager")]
public class ManagerController : Controller
{
}
[Authorize(Roles = "Administrator")]
public class AdministrationController : Controller
{
}
public class HomeController : Controller
{
}
//Policy based
[Authorize(Policy = "AtLeast21")]
public class AlcoholPurchaseController : Controller
{
public IActionResult Index() => View();
}
Ensure you are using Microsoft Identity V2!
This design will provide the least amount of administrative overhead for apps that need to be publish internally within the organisation and provides a decent level of security coverage.
When a product has been proved to be a success and has just come out of a MVP (Minimal Viable Product) or MMP (Minimal Marketable Product) state, usually a lot of corners would have been cut in order to get a product out and act on the valuable feedback. So inevitably there will be technical debt to take care of.
What is important is having a technical vision that will reduce costs and provide value/impact/scaleable/resilient/reliable which can then be communicated to all stakeholders.
A lot of cost savings can be made when scaling out by putting together a Cloud Architecture Roadmap. The roadmap can then be communicate with your stakeholders, development teams and most importantly finance. It will provide a high level “map” of where you are now and where you want to be at some point in the future.
A roadmap is every changing, just like when my wife and I go travelling around the world. We will have a roadmap of where want to go for a year but are open to making changes half way through the trip e.g. An earthquake hits a country we planned to visit etc. The same is true in IT, sometimes budgets are cut or a budget surplus needs to be consumed, such events can affect your roadmap.
It is something that you want to review on a regular schedule. Most importantly you want to communicate the roadmap and get feedback from others.
Feedback from other engineers and stakeholders is crucial – they may spot something that you did not or provide some better alternative solutions.
Decomposition
The first stage is to decompose your ideas. Below is a list that helps get me started in the right direction. This is by no means an exhausted list, it will differ based on your industry.
Component
Description
Example
Application Run-time
Where apps are hosted
Azure Kubernetes
Persistent Storage
Non-Volatile Data
File Store Block Store Object Store CDN Message Database Cache
Once you have an idea of all your components. The next step is to breakdown your road-map into milestones that will ultimately assist in reaching your final/target state. Which of course will not be final in a few years time 😉 or even months!
Sample Roadmap
Below is a link to a google slide presentation that you can use for your roadmap.
There are several decision we make every day some conscious and many sub conscious. We have a bit more control over the conscious decisions we make in the work place from an Architecture perspective.
If you are into Development, Devops, Site reliability, Technical Product Owner, Architect or event a contract/consultant; you will be contributing to significant decision regarding engineering.
What Database Technology should we use?
What Search Technology will we use that can scale and do we leverage eventual consistency?
What Container Orchestration or Micro-Service platform shall we use?
When making a decision in 2016, the decision may have been perfectly valid for the technology choices for that time. Fast forward to 2019 and if faced with the exact same decision your solution may be entirely different.
This is absolutely normal and this why it is important to have a “journal” where you outline the key reasons/rationale for a significant architecture decision.
It lays the foundation to effectively communicate with stakeholders and to “sell’ your decisions to others; even better to collaborate with others in a manner that is constructive to evaluating feedback and adjusting key decisions.
I keep a journal of decisions and use a powershell inspired naming convention of Verb-Noun. Secondly I will look at what is trending in the marketplace to use as a guide post. So for a logging/Tracking/Metrics stack, I might start off with reference materials.
This allows me to keep on track with what the industry is doing and forces me to keep up to date with best practices.
Below is sample Decision Record that I use. I hope you may find it useful. I often use them when on-boarding consultants/contractors or new members of the team. It is a great way for them to gain insights into the past and where we going.
In the next blog post, I will discuss formulating an Architecture Roadmap and how to effectively communicate your vision with key stakeholders. Until then, happy decisions and do not be surprised when you sometimes convince yourself out of a bad decision that you made 😉
Now…How do I tell my wife we should do this at home when buying the next sofa?
TITLE (Verb-Description-# e.g. Choose-MetricsTracingLoggingStack)
Status
Context
<what is the issue that we’re seeing that is motivating this decision or change.>
Constraints
<what boundaries are in place e.g. cost, technology knowledge/resources at hand>
Decision
<what is the change/transformation that we’re actually proposing or doing.>
A lot of organisations are still using the Implicit Flow for authorization when their client applications are browser based e.g. ReactJS/NodeJS. The problem with this workflow is that it was designed when browsers had a lot less capabilities.
Implicit Grant
Implicit Grant flow leverages a redirect with the access token in the url.
If someone gains access to your browser history and your token has not yet expired. They can then gain full access to your resources.
As we can see above, this is in the url on a redirect. So a simple script can find these tokens in your browser history. Step 6 below in the Implicit grant is where the issue occurs and the token is recorded in your history.
If your clients are using modern browsers that support CORS via javascript. Then the solution is to use a flow where step 6 is not an HTTP Get Redirect (302). Ideally we want an http post.
Authorization Workflow with PKCE
Proof Key for Code Exchange – The PKCE extension prevents an attack where the authorization code is intercepted and exchanged for an access token by a malicious client, by providing the authorization server with a way to verify the same client instance that exchanges the authorization code is the same one that initiated the flow.
Secondly instead of an HTTP GET, an HTTP POST is used to send the token over the wire. Thus the exchange is not recorded in the browser history at all. The token is sent as a payload in the HTTP data section and not the URL.
Notice below the token is requested via an HTTP POST (ClientID, (v), (a) on step 8.
The user clicks Login within the native/mobile application.
Auth0’s SDK creates a cryptographically-random code_verifier and from this generates a code_challenge.
Auth0’s SDK redirects the user to the Auth0 Authorization Server (/authorize endpoint) along with the code_challenge.
Your Auth0 Authorization Server redirects the user to the login and authorization prompt.
The user authenticates using one of the configured login options and may see a consent page listing the permissions Auth0 will give to the mobile application.
Your Auth0 Authorization Server stores the code_challenge and redirects the user back to the application with an authorization code.
Auth0’s SDK sends this code and the code_verifier (created in step 2) to the Auth0 Authorization Server (/oauth/token endpoint).
Your Auth0 Authorization Server verifies the code_challenge and code_verifier.
Your Auth0 Authorization Server responds with an ID Token and Access Token (and optionally, a Refresh Token).
Your application can use the Access Token to call an API to access information about the user.
The API responds with requested data.
So… I should just use Authorization Workflow with PKCE? Not so fast. If you have a large customer base that are using older browsers that do not support CORS via javascript, you might be stuck with implicit grant.
Another consideration is that the token endpoint on your Authorization Server of choice MUST support CORS for the trick to come together; not every major vendor supports it yet.
However, if you can influence you customers and client browsers to use later versions and security is a big item on your list. This might be the best case to put forward an upgrade plan.
In Summary
Does your Authorization Server supprot CORS?
Can your clients use modern browsers that support CORS?
If the answer is yes to both, then there is no need to use Implicit Grant with OAuth 2.0
OKTA ands Auth0 are some of the ID providers that support PKCE in SPA clients.
Note: Microsoft Identity V2.0 does not currently support Auth Code Workflow with SPA. This may change in the future as it is a new product and MS are investing in V2.
Recently a security patch by Microsoft has been released. We wanted to ensure we can have a predictable upgrade path.
Below is a Bash script that leverages the AzureCLI to control the uprgrade process.
It will: * Detect upgradable versions * Automatically selects the latest upgradable version
Test on Ubuntu 18
#!/usr/bin/env bash
set -e
echo "------------------------------------------------------------------------------------------------------------------"
echo "When you upgrade an AKS cluster, Kubernetes minor versions cannot be skipped."
echo "For example, upgrades between 1.12.x -> 1.13.x or 1.13.x -> 1.14.x are allowed, however 1.12.x -> 1.14.x is not."
echo "To upgrade, from 1.12.x -> 1.14.x, first upgrade from 1.12.x -> 1.13.x, then upgrade from 1.13.x -> 1.14.x."
echo "------------------------------------------------------------------------------------------------------------------"
while ! [[ "$env" =~ ^(sb|dv|ut|pd)$ ]]
do
echo "Please specifiy environment [sb, dv,ut,pd]?"
read -r env
done
case $env in
dv)
az account set --subscription 'RangerRom DEV'
subscriptionid=$(az account show --subscription 'RangerRom DEV' --query id | sed 's/\"//g')
;;
sb)
az account set --subscription 'RangerRom SANDBOX'
subscriptionid=$(az account show --subscription 'RangerRom SANDBOX' --query id | sed 's/\"//g')
;;
ut)
az account set --subscription 'RangerRom TEST'
subscriptionid=$(az account show --subscription 'RangerRom TEST' --query id | sed 's/\"//g')
;;
pd)
az account set --subscription 'RangerRom PROD'
subscriptionid=$(az account show --subscription 'RangerRom PROD' --query id | sed 's/\"//g')
;;
*)
echo "environment not found"
exit
;;
esac
env="dccau${env}"
az aks get-credentials --resource-group "${env}-k8s-rg" --name "${env}-k8s-cluster" --overwrite-existing
echo "Getting the upgrade versions available for a managed AKS: ${env}-k8s-cluster."
az aks get-upgrades --resource-group "${env}-k8s-rg" --name "${env}-k8s-cluster" --output table
echo "Detecting the next minor version to upgrade to."
versionToUpgradeTo=$(az aks get-upgrades --resource-group "${env}-k8s-rg" --name "${env}-k8s-cluster" | grep "kubernetesVersion" | cut -d'"' -f4 | sort | tail -n1)
echo "Upgrading to version $versionToUpgradeTo"
az aks upgrade --resource-group "${env}-k8s-rg" --name "${env}-k8s-cluster" --kubernetes-version $versionToUpgradeTo
echo "Upgrade complete. Please run this again if you need to upgrade to the next minor version."
A lot of people learning coding by starting with the traditional “hello world” application. I am intrigued that not a lot of time goes into discussing the coding environment setup.
When I have the luxury to work on a greenfields project. I will set the expectations to spend at least 3 weeks geting the “process” right.
Week 1 – Setup the Agile/Kanban board and plan PBI’s around CI/CD, Infrastructure as code.
Week 2 – Development environment setup
Week 3 – Fully automated deployment of a “hello world” application to the cloud. Encompassing – Automatyed Builds, Gated Releases, Centralised Containers (microservices) etc.
Coming back to the development environment. This alone can increase developer productivity by ensuring they are setup correctly. Otherwise they may be spending hours trying to resolve shared package libraries conflicts.
pyenv-virtualenv
pyenv-virtualenv is a pyenv plugin that provides features to manage virtualenvs and conda environments for Python on UNIX-like systems.
pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.
This project was forked from rbenv and ruby-build, and modified for Python.
It allows us to have complete isolation between projects. So you can focus on code and not package hell. Remember the dll hell days?
Now we need to work on another project, we do not need to worry about what data-collector package dependecies are installed, we can just switch to a new environment.
$ pyenv virtualenv 3.7.3 libor-bank-rates && pyenv activate libor-bank-rates
// Do some Coding!
(libor-bank-rates)$ pyenv deactivate
So use pyenv-virtualenv to auto-activate your environments as you work from one project to the next.
$ pyenv virtualenv 3.7.3 data-collector
$ pyenv virtualenv 3.7.3 libor-bank-rates
$ cd ~/data-collector
$ pyenv local data-collector
(data-collector)$ cd ~/libor-bank-rates
$ pyenv local libor-bank-rates
(libor-bank-rates)$ pyenv versions
system
3.7.3
3.7.3/envs/data-collector
3.7.3/envs/libor-bank-rates
data-collector
* libor-bank-rates (set by /home/romiko/libor-bank-rates/.python-version)
Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language
Conda will install the version of Python if it isn’t installed. You do not need to run conda install python=3.7.3 first. It has full support for managing virtual environments.
Summary
So the next time you decide you need to wip up some new scripts. Have a good think about how you want the environment to be setup and how package management/dependecies should be handled, before writing the infamous “hello world”.
This post will demonstrate how to deploy a AKS cluster using Advanced Networking. We will then deploy an Application Gateway Ingress Controller. Essentially this will install a dedicated ingress POD that fully manages the Application gateway.
This means all entries in the Application gateway are 100% managed by AKS. If you manually add an entry to the AG, it will be removed by AKS Ingress Controller.
Overview
Considerations
Decided to dedicate an entire /16 IP range to the AKS cluster for simplicity e.g. 10.69.0.0/16.
CNI provided the use of Application Gateway with WAFv2.
SSL offloading is configured. The actual private key (PEM – Base64 encoded) is stored in the default namespace in AKS. Whenever you deploy a new application, just –export (Deprecated) the key to the new namespace. The AG Ingress Controller will automatically be configured with the SSL certificate.
We will apply RBAC rules so AKS can manage the application gateway and VMSS scaleset.
RBAC to access container registry.
By using an Application Gateway, we can leverage additional benefits such as Web Application Firewall (V2), OWASP 3.0 firewall detection/prevention rules. Microsoft have totally refactors the AG WAF2 technology stack. It is much faster to provision and can deal with much larger amounts of traffic now.
By combining Load Balancing with WAF, we get the best of both worlds. If you have heavy traffic, it might be good to first do a performance test before making a final decision on AG + AKS stack.
Environment Setup + Tools
We are using AKS VMSS preview feature. Azure Virtual Machine Scale Sets have been around for a long time, and are in fact used by Microsoft Service Fabric. It makes total sense that this auto-scaling architecture is leveraged by AKS.
Due to the preview status of Container Services and VMSS+AKS, we will choose Azure CLI.
You can use Ubuntu Windows Shell or a Linux Ubuntu Shell.
Run the following code to setup your bash environment.
Helm is a client side tool to provide configuration settings to AKS. Tiller is a server side setting that runs on AKS that applies configuration settings that are applied from a helm client.
Create a config folder with 2 files.
Replace THECERTIFICATECHAIN with the contents of your base64 encoded .cer certificate chain. The script will replace <THEPRIVATEKEY> when you paste your private key. Future namespace or apps, will be able to find this in the default namespace. Thus a 1 time operations
e.g. (kubectl get secret rangerrom-tls –namespace=default ….).
Ensure you have * VNET in a resounce group – ${env}-network-rg * Subnet with a name ${env}-aks-cluster-subnet matching the IP rules
Install AKS into existing VNET
#!/bin/bash
aksversion='1.13.7'
while ! [[ "$env" =~ ^(sb|dv|ut|pd)$ ]]
do
echo "Please specifiy environment [sb, dv,ut,pd]?"
read -r env
done
case $env in
dv)
servicecidr="10.66.64.0/18"
dnsserver="10.66.64.10"
az account set --subscription 'RangerRom DEV'
subscriptionid=$(az account show --subscription 'RangerRom DEV' --query id | sed 's/\"//g')
;;
sb)
servicecidr="10.69.64.0/18"
dnsserver="10.69.64.10"
az account set --subscription 'RangerRom SANDBOX'
subscriptionid=$(az account show --subscription 'RangerRom SANDBOX' --query id | sed 's/\"//g')
;;
ut)
servicecidr="10.70.64.0/18"
dnsserver="10.70.64.10"
az account set --subscription 'RangerRom TEST'
subscriptionid=$(az account show --subscription 'RangerRom TEST' --query id | sed 's/\"//g')
;;
pd)
servicecidr="10.68.64.0/18"
dnsserver="10.68.64.10"
az account set --subscription 'RangerRom PROD'
subscriptionid=$(az account show --subscription 'RangerRom PROD' --query id | sed 's/\"//g')
;;
*)
echo "environment not found"
exit
;;
esac
env="rrau${env}"
location="australiaeast"
az group create --location $location --name "${env}-aks-rg"
sleep 5
az feature register -n VMSSPreview --namespace Microsoft.ContainerService
az provider register -n Microsoft.ContainerService
az aks create \
--resource-group "${env}-aks-rg" \
--name "${env}-aks-cluster" \
--enable-vmss \
--node-count 2 \
--kubernetes-version $aksversion \
--generate-ssh-keys \
--network-plugin azure \
--service-cidr $servicecidr \
--dns-service-ip $dnsserver \
--vnet-subnet-id "/subscriptions/${subscriptionid}/resourceGroups/${env}-network-rg/providers/Microsoft.Network/virtualNetworks/${env}-network/subnets/${env}-aks-cluster-subnet"
clusterprincipalid=$(az ad sp list --display-name ${env}-aks-cluster --query [0].objectId)
resourceGroupid=$(az group show --name ${env}-network-rg --query 'id')
echo "Configuring cluster to owner ${resourceGroupid}"
cmd="az role assignment create --role Contributor --assignee $clusterprincipalid --scope $resourceGroupid"
eval $cmd
echo "Configuring AKS Cluster with Tiller"
az aks get-credentials --resource-group "${env}-aks-rg" --name "${env}-aks-cluster" --overwrite-existing
kubectl apply -f ./config/tiller-rbac.yml
helm init --service-account tiller
while ! [[ ${#privatekey} -gt 2000 ]]
do
echo "Please provide TLS Private Key - BASE64 Encoded PEM?"
read -r privatekey
done
kubectl create namespace scpi-${env}
cat ./config/rangerrom-tls.yml | sed "s/THEPRIVATEKEY/$privatekey/" > temptls.yml
kubectl apply -f temptls.yml -n default
#The Flag --export is going to be deprecated - Below is workaround.
kubectl get secret rangerrom-tls --namespace=default -o yaml | \
sed '/^.*creationTimestamp:/d' |\
sed '/^.*namespace:/d' |\
sed '/^.*resourceVersion:/d' |\
sed '/^.*selfLink:/d' |\
sed '/^.*uid:/d' |\
kubectl apply --namespace=scpi-${env} -f -
rm -f ./temptls.yml
privatekey=""
echo "Setup container registry permissions"
az acr create -n "${env}containerregistry" -g "${env}-common-rg" --sku Premium
containerid=$(az acr show -n ${env}containerregistry --query id)
principalidaks=$(az ad sp list --all --query "([?contains(to_string(displayName),'"${env}-aks-cluster"')].objectId)[0]")
cmd1="az role assignment create --role acrpull --assignee $principalidaks --scope $containerid"
eval $cmd1
echo "Setup container registry permissions - Centralised"
containerid=$(az acr show --subscription 'RangerRom PROD' -n rraupdcontainerregistry --query id)
cmd2="az role assignment create --role acrpull --assignee $principalidaks --scope $containerid"
eval $cmd2
Install and Configure AKS to control the Ingress
#!/bin/bash
while ! [[ "$env" =~ ^(sb|dv|ut|pd)$ ]]
do
echo "Ensure you have owner permissions on the subscription before you continue."
echo "Please specifiy environment [sb, dv,ut,pd]?"
read -r env
done
case $env in
dv)
az account set --subscription 'RangerRom DEV'
;;
sb)
az account set --subscription 'RangerRom SANDBOX'
;;
ut)
az account set --subscription 'RangerRom TEST'
;;
pd)
az account set --subscription 'RangerRom PROD'
;;
*)
echo "Invalid Environment"
exit
;;
esac
env="rrau${env}"
ipAddressName="${env}-aks-application-gateway-ip"
resourcegroup="MC_${env}-aks-rg_${env}-aks-cluster_australiaeast"
gatewayname="${env}-aks-application-gateway"
location="australiaeast"
vnet="${env}-network"
subnet="${env}-aks-application-gateway-subnet"
az network public-ip create \
--resource-group $resourcegroup \
--name $ipAddressName \
--allocation-method Static \
--sku Standard
sleep 20
subnetid=$(az network vnet subnet show -g "${env}-network-rg" -n "${env}-aks-application-gateway-subnet" --vnet-name ${vnet} --query id)
cmd="az network application-gateway create --name $gatewayname \
--resource-group $resourcegroup \
--capacity 2 \
--sku "WAF_v2" \
--subnet $subnetid \
--http-settings-cookie-based-affinity Disabled \
--location $location \
--frontend-port 80 \
--public-ip-address $ipAddressName"
eval $cmd
az network application-gateway waf-config set -g $resourcegroup --gateway-name $gatewayname \
--enabled true --firewall-mode Detection --rule-set-version 3.0
#Setup AAD POD Identity to manage application gateway
az identity create -g $resourcegroup -n "${env}-aks-aad_pod_identity"
sleep 20
principalid=$(az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity" --query 'principalId')
appgatewayid=$(az network application-gateway show -g $resourcegroup -n $gatewayname --query 'id')
echo "Assign Role so AKS can manage the Application Gateway"
echo "Configuring Create Role for identity - $principalid - for gateway"
cmd="az role assignment create --role Contributor --assignee $principalid --scope $appgatewayid"
eval $cmd
resourceGroupid=$(az group show --name $resourcegroup --query 'id')
echo "Configuring Read Role for identity - $principalid - for gateway resourcegroup"
cmd="az role assignment create --role Reader --assignee $principalid --scope $resourceGroupid"
eval $cmd
az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity"
echo "Please use the azure identity details above to configure AKS via Help for the AG Ingress Controller"
echo "Careful with copy and paste. Hidden characters can affect the values!"
echo "Ingress Controller for Azure Application Gateway"
az aks get-credentials --resource-group "${env}-aks-rg" --name "${env}-aks-cluster"
kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update
subscriptionid=$(az account show --query id | sed 's/\"//g')
appGatewayResourceId=$(az network application-gateway show -g $resourcegroup -n $gatewayname --query resourceGroup | sed 's/\"//g')
identityClientid=$(az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity" --query clientId | sed 's/\"//g')
aksfqdn=$(az aks show --resource-group "${env}-aks-rg" --name "${env}-aks-cluster" --query fqdn | sed 's/\"//g')
cmd="helm upgrade ingress-azure application-gateway-kubernetes-ingress/ingress-azure \
--install \
--namespace default \
--debug \
--set appgw.name=$gatewayname \
--set appgw.resourceGroup=$appGatewayResourceId \
--set appgw.subscriptionId=$subscriptionid \
--set appgw.shared=false \
--set armAuth.type=aadPodIdentity \
--set armAuth.identityResourceID=/subscriptions/$subscriptionid/resourcegroups/$appGatewayResourceId/providers/Microsoft.ManagedIdentity/userAssignedIdentities/$env-aks-aad_pod_identity \
--set armAuth.identityClientID=$identityClientid \
--set rbac.enabled=true \
--set verbosityLevel=3 \
--set aksClusterConfiguration.apiServerAddress=$aksfqdn"
eval $cmd
kubectl get pods
Conclusion
This post should provide a guide post to setup your infrastructure as code. By leveraging a rock solid naming convention, you can leverage fully automated scripts to deploy your environments. The above scripts for AKS and AG are also idempotent, so they can be run on a scheduled basis e.g. Azure Devops.
You must be logged in to post a comment.