Python – Virtual Environments

A lot of people learning coding by starting with the traditional “hello world” application. I am intrigued that not a lot of time goes into discussing the coding environment setup.

When I have the luxury to work on a greenfields project. I will set the expectations to spend at least 3 weeks geting the “process” right.

Week 1 – Setup the Agile/Kanban board and plan PBI’s around CI/CD, Infrastructure as code.

Week 2 – Development environment setup

Week 3 – Fully automated deployment of a “hello world” application to the cloud. Encompassing – Automatyed Builds, Gated Releases, Centralised Containers (microservices) etc.

Coming back to the development environment. This alone can increase developer productivity by ensuring they are setup correctly. Otherwise they may be spending hours trying to resolve shared package libraries conflicts.

pyenv-virtualenv

pyenv-virtualenv is a pyenv plugin that provides features to manage virtualenvs and conda environments for Python on UNIX-like systems.

pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

This project was forked from rbenv and ruby-build, and modified for Python.

It allows us to have complete isolation between projects. So you can focus on code and not package hell. Remember the dll hell days?

// Create Environment
$ pyenv virtualenv 3.7.3 data-collector && pyenv activate data-collector

// Do some Coding!

// Exit virtual environment
(data-collector)$ pyenv deactivate

Now we need to work on another project, we do not need to worry about what data-collector package dependecies are installed, we can just switch to a new environment.

$ pyenv virtualenv 3.7.3 libor-bank-rates && pyenv activate libor-bank-rates

// Do some Coding!

(libor-bank-rates)$ pyenv deactivate

So use pyenv-virtualenv to auto-activate your environments as you work from one project to the next.

$ pyenv virtualenv 3.7.3 data-collector
$ pyenv virtualenv 3.7.3 libor-bank-rates
$ cd ~/data-collector
$ pyenv local data-collector
(data-collector)$ cd ~/libor-bank-rates
$ pyenv local libor-bank-rates
(libor-bank-rates)$ pyenv versions
  system
  3.7.3
  3.7.3/envs/data-collector
  3.7.3/envs/libor-bank-rates
  data-collector
* libor-bank-rates (set by /home/romiko/libor-bank-rates/.python-version)

Also check out Conda.

Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language

// Create virtual environment
$ conda create --name rangerrom python=3.7.3

// Activate virtual environment
$ conda activate rangerrom

// Exit virtual environment
(ranerrom)$ conda deactivate

Conda will install the version of Python if it isn’t installed. You do not need to run conda install python=3.7.3 first. It has full support for managing virtual environments.

Summary

So the next time you decide you need to wip up some new scripts. Have a good think about how you want the environment to be setup and how package management/dependecies should be handled, before writing the infamous “hello world”.

Check out https://realpython.com for awesome tips, tricks and inspiration.

Advertisements

Automate the Deployment of Azure Kubernetes Services Cluster + Application Gateway Ingress Controller

This post will demonstrate how to deploy a AKS cluster using Advanced Networking. We will then deploy an Application Gateway Ingress Controller. Essentially this will install a dedicated ingress POD that fully manages the Application gateway.

This means all entries in the Application gateway are 100% managed by AKS. If you manually add an entry to the AG, it will be removed by AKS Ingress Controller.

Overview

Considerations

  • Decided to dedicate an entire /16 IP range to the AKS cluster for simplicity e.g. 10.69.0.0/16.
  • Leverage AKS with Advanced Networking (CNI).
  • CNI provided the use of Application Gateway with WAFv2.
  • SSL offloading is configured. The actual private key (PEM – Base64 encoded) is stored in the default namespace in AKS. Whenever you deploy a new application, just –export (Deprecated) the key to the new namespace. The AG Ingress Controller will automatically be configured with the SSL certificate.
  • We will apply RBAC rules so AKS can manage the application gateway and VMSS scaleset.
  • RBAC to access container registry.

By using an Application Gateway, we can leverage additional benefits such as Web Application Firewall (V2), OWASP 3.0 firewall detection/prevention rules. Microsoft have totally refactors the AG WAF2 technology stack. It is much faster to provision and can deal with much larger amounts of traffic now.

By combining Load Balancing with WAF, we get the best of both worlds. If you have heavy traffic, it might be good to first do a performance test before making a final decision on AG + AKS stack.

Environment Setup + Tools

We are using AKS VMSS preview feature. Azure Virtual Machine Scale Sets have been around for a long time, and are in fact used by Microsoft Service Fabric. It makes total sense that this auto-scaling architecture is leveraged by AKS.

Due to the preview status of Container Services and VMSS+AKS, we will choose Azure CLI.

You can use Ubuntu Windows Shell or a Linux Ubuntu Shell.

Run the following code to setup your bash environment.

#!/bin/bash
echo "Updating system..."
sudo apt-get update
sudo apt-get upgrade

echo "Installing AzureCLI"
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

echo "Installing helm for AKS Admin"
curl -LO https://git.io/get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
helm init --service-account tiller --history-max 200

Helm is a client side tool to provide configuration settings to AKS. Tiller is a server side setting that runs on AKS that applies configuration settings that are applied from a helm client.

Create a config folder with 2 files.

Replace THECERTIFICATECHAIN with the contents of your base64 encoded .cer certificate chain. The script will replace <THEPRIVATEKEY> when you paste your private key. Future namespace or apps, will be able to find this in the default namespace. Thus a 1 time operations

e.g. (kubectl get secret rangerrom-tls –namespace=default ….).

apiVersion: v1
kind: Secret
metadata:
  name: rangerrom-tls
type: kubernetes.io/tls
data:
  tls.crt: THECERTIFICATECHAIN
  tls.key: THEPRIVATEKEY

rangerrom-tls.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

tiller-rbac.yml

Azure Prerequisites

Ensure you have
* VNET in a resounce group – ${env}-network-rg
* Subnet with a name ${env}-aks-cluster-subnet matching the IP rules

Install AKS into existing VNET

#!/bin/bash
aksversion='1.13.7'
while ! [[ "$env" =~ ^(sb|dv|ut|pd)$ ]] 
do
  echo "Please specifiy environment [sb, dv,ut,pd]?"
  read -r env
done 

case $env in

  dv)
    servicecidr="10.66.64.0/18"
    dnsserver="10.66.64.10"
    az account set --subscription 'RangerRom DEV'
    subscriptionid=$(az account show --subscription 'RangerRom DEV' --query id | sed  's/\"//g')
    ;;

  sb)
    servicecidr="10.69.64.0/18"
    dnsserver="10.69.64.10"
    az account set --subscription 'RangerRom SANDBOX'
    subscriptionid=$(az account show --subscription 'RangerRom SANDBOX' --query id | sed  's/\"//g')
    ;;

  ut)
    servicecidr="10.70.64.0/18"
    dnsserver="10.70.64.10"
    az account set --subscription 'RangerRom TEST'
    subscriptionid=$(az account show --subscription 'RangerRom TEST' --query id | sed  's/\"//g')
    ;;

  pd)
    servicecidr="10.68.64.0/18"
    dnsserver="10.68.64.10"
    az account set --subscription 'RangerRom PROD'
    subscriptionid=$(az account show --subscription 'RangerRom PROD' --query id | sed  's/\"//g')
    ;;  
  *)
    echo "environment not found"
    exit
    ;;
esac

env="rrau${env}"
location="australiaeast"

az group create --location $location --name "${env}-aks-rg"
sleep 5

az feature register -n VMSSPreview --namespace Microsoft.ContainerService
az provider register -n Microsoft.ContainerService

az aks create \
    --resource-group "${env}-aks-rg" \
    --name "${env}-aks-cluster" \
    --enable-vmss \
    --node-count 2 \
    --kubernetes-version $aksversion \
    --generate-ssh-keys \
    --network-plugin azure \
    --service-cidr $servicecidr \
    --dns-service-ip $dnsserver \
    --vnet-subnet-id "/subscriptions/${subscriptionid}/resourceGroups/${env}-network-rg/providers/Microsoft.Network/virtualNetworks/${env}-network/subnets/${env}-aks-cluster-subnet"



clusterprincipalid=$(az ad sp list --display-name ${env}-aks-cluster --query [0].objectId)
resourceGroupid=$(az group show --name ${env}-network-rg --query 'id')
echo "Configuring cluster to owner ${resourceGroupid}"
cmd="az role assignment create --role Contributor --assignee $clusterprincipalid --scope $resourceGroupid"
eval $cmd


echo "Configuring AKS Cluster with Tiller"
az aks get-credentials --resource-group "${env}-aks-rg" --name "${env}-aks-cluster" --overwrite-existing
kubectl apply -f ./config/tiller-rbac.yml
helm init --service-account tiller

while ! [[ ${#privatekey} -gt 2000 ]]
do
  echo "Please provide TLS Private Key - BASE64 Encoded PEM?"
  read -r privatekey
done

kubectl create namespace scpi-${env}
cat ./config/rangerrom-tls.yml  | sed "s/THEPRIVATEKEY/$privatekey/" > temptls.yml
kubectl apply -f temptls.yml -n default
#The Flag --export is going to be deprecated - Below is workaround.
kubectl get secret rangerrom-tls --namespace=default -o yaml | \
   sed '/^.*creationTimestamp:/d' |\
   sed '/^.*namespace:/d' |\
   sed '/^.*resourceVersion:/d' |\
   sed '/^.*selfLink:/d' |\
   sed '/^.*uid:/d' |\
   kubectl apply --namespace=scpi-${env} -f -

rm -f ./temptls.yml
privatekey=""

echo "Setup container registry permissions"
az acr create -n "${env}containerregistry" -g "${env}-common-rg" --sku Premium
containerid=$(az acr show -n ${env}containerregistry --query id)
principalidaks=$(az ad sp list --all --query "([?contains(to_string(displayName),'"${env}-aks-cluster"')].objectId)[0]")
cmd1="az role assignment create --role acrpull --assignee $principalidaks --scope $containerid"
eval $cmd1

echo "Setup container registry permissions - Centralised"
containerid=$(az acr show --subscription 'RangerRom PROD' -n rraupdcontainerregistry --query id)
cmd2="az role assignment create --role acrpull --assignee $principalidaks --scope $containerid"
eval $cmd2

Install and Configure AKS to control the Ingress

#!/bin/bash
while ! [[ "$env" =~ ^(sb|dv|ut|pd)$ ]] 
do
  echo "Ensure you have owner permissions on the subscription before you continue."
  echo "Please specifiy environment [sb, dv,ut,pd]?"
  read -r env
done 

case $env in

  dv)
    az account set --subscription 'RangerRom DEV'
    ;;

  sb)
    az account set --subscription 'RangerRom SANDBOX'
    ;;

  ut)
    az account set --subscription 'RangerRom TEST'
    ;;

  pd)
    az account set --subscription 'RangerRom PROD'
    ;;  
  *)
    echo "Invalid Environment"
    exit
    ;;
esac

env="rrau${env}"

ipAddressName="${env}-aks-application-gateway-ip"
resourcegroup="MC_${env}-aks-rg_${env}-aks-cluster_australiaeast"
gatewayname="${env}-aks-application-gateway"
location="australiaeast"
vnet="${env}-network"
subnet="${env}-aks-application-gateway-subnet"

az network public-ip create \
  --resource-group $resourcegroup \
  --name $ipAddressName \
  --allocation-method Static \
  --sku Standard

sleep 20

subnetid=$(az network vnet subnet show -g "${env}-network-rg" -n "${env}-aks-application-gateway-subnet" --vnet-name ${vnet} --query id)
cmd="az network application-gateway create --name $gatewayname \
                                      --resource-group $resourcegroup \
                                      --capacity 2  \
                                      --sku "WAF_v2" \
                                      --subnet $subnetid \
                                      --http-settings-cookie-based-affinity Disabled \
                                      --location $location \
                                      --frontend-port 80 \
                                      --public-ip-address $ipAddressName"
eval $cmd
az network application-gateway waf-config set -g $resourcegroup --gateway-name $gatewayname \
                            --enabled true --firewall-mode Detection --rule-set-version 3.0

#Setup AAD POD Identity to manage application gateway
az identity create -g $resourcegroup -n "${env}-aks-aad_pod_identity"
sleep 20

principalid=$(az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity" --query 'principalId')
appgatewayid=$(az network application-gateway show -g $resourcegroup -n $gatewayname --query 'id')

echo "Assign Role so AKS can manage the Application Gateway"

echo "Configuring Create Role for identity - $principalid - for gateway"
cmd="az role assignment create --role Contributor --assignee $principalid --scope $appgatewayid"
eval $cmd

resourceGroupid=$(az group show --name $resourcegroup --query 'id')

echo "Configuring Read Role for identity - $principalid - for gateway resourcegroup"
cmd="az role assignment create --role Reader --assignee $principalid --scope $resourceGroupid"
eval $cmd

az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity"
echo "Please use the azure identity details above to configure AKS via Help for the AG Ingress Controller"
echo "Careful with copy and paste. Hidden characters can affect the values!"


echo "Ingress Controller for Azure Application Gateway"
az aks get-credentials --resource-group "${env}-aks-rg" --name "${env}-aks-cluster"
kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

subscriptionid=$(az account show --query id | sed  's/\"//g')
appGatewayResourceId=$(az network application-gateway show -g $resourcegroup -n $gatewayname --query resourceGroup  | sed  's/\"//g')
identityClientid=$(az identity show -g $resourcegroup -n "${env}-aks-aad_pod_identity" --query clientId  | sed  's/\"//g')
aksfqdn=$(az aks show --resource-group "${env}-aks-rg" --name "${env}-aks-cluster" --query fqdn  | sed  's/\"//g')

cmd="helm upgrade ingress-azure application-gateway-kubernetes-ingress/ingress-azure \
     --install \
     --namespace default \
     --debug \
     --set appgw.name=$gatewayname \
     --set appgw.resourceGroup=$appGatewayResourceId \
     --set appgw.subscriptionId=$subscriptionid \
     --set appgw.shared=false \
     --set armAuth.type=aadPodIdentity \
     --set armAuth.identityResourceID=/subscriptions/$subscriptionid/resourcegroups/$appGatewayResourceId/providers/Microsoft.ManagedIdentity/userAssignedIdentities/$env-aks-aad_pod_identity \
     --set armAuth.identityClientID=$identityClientid \
     --set rbac.enabled=true \
     --set verbosityLevel=3 \
     --set aksClusterConfiguration.apiServerAddress=$aksfqdn"
eval $cmd
kubectl get pods


Conclusion

This post should provide a guide post to setup your infrastructure as code. By leveraging a rock solid naming convention, you can leverage fully automated scripts to deploy your environments. The above scripts for AKS and AG are also idempotent, so they can be run on a scheduled basis e.g. Azure Devops.

Writing a Singleton Class in .NET

Usually you will register your classes in your Dependency Container of choice.

However if you really need to do it manually. Here is sample code that always ensures there is only one taxi instance (Poor Business is not going to do too well – Definitely not a scaleable business).


using System;
using System.Collections.Generic;

namespace Patterns.Singleton
{
    class TaxiSchedule
    {
        static void Main()
        {
            var t1 = Taxi.GetTaxi();
            var t2 = Taxi.GetTaxi();
            var t3 = Taxi.GetTaxi();
            var t4 = Taxi.GetTaxi();

            if (t1 == t2 && t2 == t3 && t3 == t4)
                Console.WriteLine("They are the same!\n");

            var taxi = Taxi.GetTaxi();
            for (int i = 0; i < 24; i++)
                Console.WriteLine("Wake Up: " + taxi.NextDriver.Name);

            Console.ReadKey();
        }
    }

    /// 

<summary>
    /// Singleton Taxi
    /// </summary>


    sealed class Taxi
    {
        static readonly Taxi instance = new Taxi();
        IList<Driver> drivers;
        Random random = new Random();
        int currentDriver = 0;

        private Taxi()
        {
            drivers = new List<Driver>
                {
                  new Driver{ Name = "Joe" },
                  new Driver{ Name = "Bob" },
                  new Driver{ Name = "Harry" },
                  new Driver{ Name = "Ford" },
                };
        }

        public static Taxi GetTaxi()
        {
            return instance;
        }

        public Driver NextDriver
        {
            get
            {
                if (currentDriver >= drivers.Count)
                    currentDriver = 0;
                currentDriver++;
                return drivers[currentDriver -1];
            }
        }
    }

    class Driver
    {
        public string Name { get; set; }
    }
}
<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>

 

 

Query Azure AppInsights with Powershell

In order to query AppInsights using powershell, you will need your AppInsights AppId and APIKey.

The important consideration is to ensure your JSON is valid, so always run it through a parser and use the correct escape characters for both JSON and PowerShell. Have a look at the string in $queryData.

The following code will query appinsights and generate csv files based on the batch size. It also using paging by leveraging:

| serialize | extend rn = row_number()

Happy DevOps Reporting 🙂

param (
[Parameter(Mandatory = $true)]
[string] $AppInsightsId,

[Parameter(Mandatory = $true)]
[string] $apiKey,

[Parameter(Mandatory = $false)]
[string]
$Timespan = "P7D",

[Parameter(Mandatory = $false)]
[int]
$batchSize = "10000",

[Parameter(Mandatory = $false)]
[string]
$OutputFolder = "C:\Output\",

[Parameter(Mandatory = $false)]
[string]$logFileName = "AppQuery.log",

[Parameter(Mandatory = $false)]
[string]$logFolder = "C:\Logs\"
)

Add-Type -AssemblyName System.Web

Import-Module .\Helpers.psm1 -Force -ErrorAction Stop
Import-Module .\Shared.Logging.psm1 -Force -Global
Import-Module .\Security.Helpers.psm1 -Force -Global

function prepareFileHeader($filenumber, $columnNames) {
$csvString = ""
ForEach ( $Property in $columnNames )
{
$csvString += "$($Property.Name),"
}
$csvString = $csvString.Substring(0,$csvString.Length -1)
$file = Join-Path $OutputFolder "batch-$i.csv"
$csvString | Out-File -filepath $file -Encoding utf8
$csvString = $null
}

function writeRecordsToFile($records) {
ForEach ( $record in $records )
{
$csvString = ""
foreach($cell in $record) {
$csvString += "$cell,"
}
$csvString = $csvString.Substring(0,$csvString.Length -1)
$file = Join-Path $OutputFolder "batch-$i.csv"
$csvString | Out-File -filepath $file -Encoding utf8 -Append -NoClobber
$csvString = $null
}
}

$logFilePath = PrepareToLog $logFolder $logFileName

try {
$url = "https://api.applicationinsights.io/v1/apps/$AppInsightsId/query"
$headers = @{"Content-Type" = "application/json"}
$headers.add("x-api-key", $apiKey)
$queryString = "?timespan=$Timespan"
$fullUrl = $url + $queryString

$queryTotalMessageCount = "traces\r\n | where message contains \`"Max Retry Count reached\`" and message contains \`"MessageService\`"\r\n | summarize count()"
$queryTotalMessageCountBody = "{
`"query`": `"$queryTotalMessageCount`"
}"
$resultCount = Invoke-WebRequest -Uri $fullUrl -Headers $headers -Method POST -Body $queryTotalMessageCountBody -ErrorAction Continue
$totalObject = ConvertFrom-Json $resultCount.Content

$totalRecords = $totalObject.tables.rows[0]
$pages = [math]::ceiling($totalRecords/$batchSize)
$startRow = 0
$endRow = $batchSize

Write-Host "Total Files: $pages for Batch Size: $batchSize"
For ($i=1; $i -le $pages; $i++) {
Write-Host "Processing File: C:\batch-$i.csv"
$queryData = "traces\r\n | extend TenantId = extract(\`"Tenant Id \\\\[[a-z0-9A-Z-.]*\\\\]\`", 0, message) | extend UniqueTransactionId = extract(\`"\\\\[[a-z0-9A-Z-. _\\\\^]*\\\\]\`",0 ,extract(\`"Message Transaction \\\\[[a-z0-9A-Z-._]*\\\\]\`", 0, message))\r\n | extend TransactionId = trim_start(\`"\\\\[\`", tostring(split(UniqueTransactionId, \`"_\`") [0]))\r\n | extend TransactionDateTicks = tostring(split(UniqueTransactionId, \`"_\`") [1])\r\n | extend PrincipalId = trim_end(\`"\\\\]\`", tostring(split(UniqueTransactionId, \`"_\`") [2]))\r\n | where message contains \`"Max Retry Count reached\`" and message contains \`"MessageService\`"\r\n | project TransactionId, TransactionDateTicks, PrincipalId, TenantId\r\n | summarize ErrorCount = count(TransactionId) by TransactionId, TransactionDateTicks, PrincipalId, TenantId\r\n | serialize | extend rn = row_number()\r\n | where rn > $startRow and rn <= $endRow"
$queryBody = "{
`"query`": `"$queryData`"
}"
$result = Invoke-WebRequest -Uri $fullUrl -Headers $headers -Method POST -Body $queryBody -ErrorAction Continue
$data = ConvertFrom-Json $result.Content
$startRow += $batchSize
$endRow += $batchSize

if($i -eq 1) {
$columnNames = $data.tables.columns | select name
}

prepareFileHeader $i $columnNames
writeRecordsToFile $data.tables.rows
}
} catch {
LogErrorMessage -msg $error[0] -filePath $logFilePath -fatal $true
}

ARM – Modular Templates – Reference resources already created

Hi,

I noticed the Microsoft documentation related to the following function is a little bit vague.

reference(resourceName or resourceIdentifier, [apiVersion], [‘Full’])

The second issue is see a lot of people having is how do you reference a resource already created in ARM and get some of that objects properties e.g. FQDN on a public IP already created etc.

The clue to solve this issue, so that ARM Template B can reference a resource created in ARM Template A can be found here:

By using the reference function, you implicitly declare that one resource depends on another resource if the referenced resource is provisioned within same template and you refer to the resource by its name (not resource ID). You don’t need to also use the dependsOn property. The function isn’t evaluated until the referenced resource has completed deployment.

Or use linked templates (Linked templates is a huge rework and you need to host the files on the net). Lets see if we can do it via resourceId.

Therefore if we do reference a resource by resourceId, we will remove the implicit “depends on”, allowing ARM Template B to use a resource created in a totally different ARM template.

A great example might be the FQDN on an IP Address.

Imagine ARM Template A creates the IP Address


"resources": [
{
"apiVersion": "[variables('publicIPApiVersion')]",
"type": "Microsoft.Network/publicIPAddresses",
"name": "[variables('AppPublicIPName')]",
"location": "[variables('computeLocation')]",
"properties": {
"dnsSettings": {
"domainNameLabel": "[variables('AppDnsName')]"
},
"publicIPAllocationMethod": "Dynamic"
},
"tags": {
"resourceType": "Service Fabric",
"scaleSetName": "[parameters('scaleSetName')]"
}
}]

Now Imagine we need to get the FQDN of the IP Address in a ARM Template B

What we going to do is try this:

reference(resourceIdentifier, [apiVersion]) ->
reference(resourceId(), [apiVersion]) ->
e.g.

Here is an example where ARM template B references a resource in A and gets a property.


"managementEndpoint": "[concat('https://',reference(resourceId('Microsoft.Network/publicIPAddresses/',variables('AppPublicIPName')), variables('publicIPApiVersion')).dnsSettings.fqdn,':',variables('nodeTypePrimaryServiceFabricHttpPort'))]",

The important thing here is to ensure you always include the API Version. This pattern is a very powerful way to create smaller and more modular ARM templates.

Note: In the above pattern, you do not need to define DependsOn in ARM Template B, as we are explicitly defining a reference to an existing resource. ARM Template B is not responsible for creating a public IP. If you need it, you run ARM Template A.

So if you need a reference to existing resources use the above. If you need a reference to resources created in the SAME ARM template use:

reference(resourceName)

Cheers

Service Fabric – Upgrading VMSS Disks, Operating System on Primary Node Type

How do you upgrade the existing Data Disk on a primary Node Type Virtual Machine ScaleSet in Service Fabric?

How do you upgrade the existing Operating System on a primary Node Type VMSS in Service Fabric?

How do you move the Data Disk on a primary Node Type VMSS in Service Fabric?

How do you monitor the status during the upgrade, so you know exactly how many seed nodes have migrated over to the new scale set?

note – We successfully increased the SKU size as well, however this is not supported by Microsoft. However just increase your SKU in ARm and later, after the successful transfer to the new VMSS, run Update-AzureRmServiceFabricDurability.

Considerations

  • You have knowledge to use ARM to deploy an Azure Load Balancer
  • You have knowledge to use ARM to deploy a VMSS Scale Set
  • Service Fabric Durability Tier/Reliability Tier must be at least Silver
  • Keep the original Azure DNS name on the Load Balancer that is used to connect to the Service Fabric Endpoint. Very Important to write it down as a backup
  • You will need to reduce the TTL of all your DNS settings to reduce downtime during the upgrade which will just be the TTL value e.g. 10 minutes. (Ensure you have access to your primary DNS provider to do this)
  • Prepare an ARM template to add the new Azure Load Balancer that the new VMSS scaleset will attach to (Backend Pool)
  • Prepare an ARM template to add the new VMSS to an existing Service Fabric primary Node Type
  • Deploy the new Azure Load Balancer + Virtual Machine Scale Set to the Service Fabric Primary node
  • Run the RemoveScaleSetFromClusterController.ps1 – Run this script on the NEW node in the NEW VMSS. This script will monitor and facilitate moving the Primary Node Type to the new VMSS for you.  It will show you the status of the Seed nodes moving from the original Primary Node Type to the new VMSS.
  • When it completed, the last part will be to update DNS.
  • Run MoveDNSToNewPublicIPController.ps1

ARM Templates

You will need only 2 templates. One to Deploy a new Azure Load Balancer and one to Deploy the new VMSS Scale Set to the existing Service Fabric Cluster.

You will also need a powershell script that will run a custom script extension.

Custom Script – prepare_sf_vm.ps1


$disks = Get-Disk | Where partitionstyle -eq 'raw' | sort number

$letters = 70..89 | ForEach-Object { [char]$_ }
$count = 0
$label = "datadisk"

foreach ($disk in $disks) {
    $driveLetter = $letters[$count].ToString()
    $disk | 
    Initialize-Disk -PartitionStyle GPT -PassThru |
    New-Partition -UseMaximumSize -DriveLetter $driveLetter |
    Format-Volume -FileSystem NTFS -NewFileSystemLabel "$label$count" -Confirm:$false -Force
$count++
}

# Disable Windows Update
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU' -Name NoAutoUpdate -Value 1

 

Load Balancer – azuredeploy_servicefabric_loadbalancer.json

Use your particular Load Balancer ARM Templates. No need to attached a backend pool, as this will be done by the VMSS script below.

Service Fabric attach new VMSS – azuredeploy_add_new_VMSS_to_nodeType.json

Create your own VMSS scaleset that you attach to Service fabric. The important aspect are the following.

nodeTypeRef (To attach VMSS to existing PrimaryNodeType).
dataPath (To use a new Disk for data)
dataDisk (to add a new managed physical disk)

We use F:\ onwards as D is reserved for Temp storage and E: is reserved for a CD ROM in Azure VM’s.


{
                                "name": "[concat('ServiceFabricNodeVmExt',variables('vmNodeType0Name'))]",
                                "properties": {
                                    "type": "ServiceFabricNode",
                                    "autoUpgradeMinorVersion": true,
                                    "protectedSettings": {
                                        "StorageAccountKey1": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-05-01-preview').key1]",
                                        "StorageAccountKey2": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-05-01-preview').key2]"
                                    },
                                    "publisher": "Microsoft.Azure.ServiceFabric",
                                    "settings": {
                                        "clusterEndpoint": "[parameters('existingClusterConnectionEndpoint')]",
                                        "nodeTypeRef": "[parameters('existingNodeTypeName')]",
                                        "dataPath": "F:\\\\SvcFab",
                                        "durabilityLevel": "Silver",
                                        "enableParallelJobs": true,
                                        "nicPrefixOverride": "[variables('subnet0Prefix')]",
                                        "certificate": {
                                            "thumbprint": "[parameters('certificateThumbprint')]",
                                            "x509StoreName": "[parameters('certificateStoreValue')]"
                                        }
                                    },
                                    "typeHandlerVersion": "1.0"
                                }
                            },
....
.......
.........
"storageProfile": {
                        "imageReference": {
                            "publisher": "[parameters('vmImagePublisher')]",
                            "offer": "[parameters('vmImageOffer')]",
                            "sku": "2016-Datacenter-with-Containers",
                            "version": "[parameters('vmImageVersion')]"
                        },
                        "osDisk": {
                            "managedDisk": {
                                "storageAccountType": "[parameters('storageAccountType')]"
                            },
                            "caching": "ReadWrite",
                            "createOption": "FromImage"
                        },
                        "dataDisks": [
                            {
                                "managedDisk": {
                                    "storageAccountType": "[parameters('storageAccountType')]"
                                },
                                "lun": 0,
                                "createOption": "Empty",
                                "diskSizeGB": "[parameters('dataDiskSize')]",
                                "caching": "None"
                            }
                        ]
                    }

...
....
.....
 "virtualMachineProfile": {
                    "extensionProfile": {
                        "extensions": [
                            {
                                "name": "PrepareDataDisk",
                                "properties": {
                                    "publisher": "Microsoft.Compute",
                                    "type": "CustomScriptExtension",
                                    "typeHandlerVersion": "1.8",
                                    "autoUpgradeMinorVersion": true,
                                    "settings": {
                                    "fileUris": [
                                        "[variables('vmssSetupScriptUrl')]"
                                    ],
                                    "commandToExecute": "[concat('powershell -ExecutionPolicy Unrestricted -File prepare_sf_vm.ps1 ')]"
                                    }
                                }
                            },


 

Once you have a new VMSS scale set attached to the existing NodeType, you should see in Service Fabric the extra nodes. the next step is to disable and remove the existing VMSS scaleset. This is an online operation, so you should be fine. However later we will need to update DNS for the Cluster Endpoint. This is important for Powershell Admin tools to still connect to the Service Fabric cluster.

RemoveScaleSetFromClusterController.ps1

Remote into one of the NEW VMSS virtual machines and run the following command. It will make dead sure that your seed nodes migrate over. it can take a long time (Microsoft docs say it takes a long time, how long?). it depends, for a cluster with 5 seed nodes, it took nearly 4 hours! So be patient and update the loop timeout to match your environment, increase the timeout if you have more than 5 seed nodes. My general rule is allow 45 minutes per seed node transfer.


#Requires -Version 5.0
#Requires -RunAsAdministrator



param (
    [Parameter(Mandatory = $true)]
    [string]
    $subscriptionName,

    [Parameter(Mandatory = $true)]
    [string] 
    $scaleSetToDisable,

    [Parameter(Mandatory = $true)]
    [string]
    $scaleSetToEnable,

    [Parameter(Mandatory = $true)]
    [string] 
    $resourceGroupName
)

Install-Module AzureRM.Compute -Force

Import-Module ServiceFabric -Force
Import-Module AzureRM.Compute -Force

function Disable-InternetExplorerESC {
    $AdminKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}"
    $UserKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}"
    Set-ItemProperty -Path $AdminKey -Name "IsInstalled" -Value 0
    Set-ItemProperty -Path $UserKey -Name "IsInstalled" -Value 0
    Stop-Process -Name Explorer
    Write-Host "IE Enhanced Security Configuration (ESC) has been disabled." -ForegroundColor Green
}

function Enable-InternetExplorerESC {
    $AdminKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}"
    $UserKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}"
    Set-ItemProperty -Path $AdminKey -Name "IsInstalled" -Value 1
    Set-ItemProperty -Path $UserKey -Name "IsInstalled" -Value 1
    Stop-Process -Name Explorer
    Write-Host "IE Enhanced Security Configuration (ESC) has been enabled." -ForegroundColor Green
}

$ErrorActionPreference = "Stop"

Disable-InternetExplorerESC

Login-AzureRmAccount -SubscriptionName $subscriptionName

Write-Host "Before you continue:  Ensure IE Enhanced Security is off."
Write-Host "Before you continue:  Ensure your new scaleset is ALREADY added to the Service Fabric Cluster"
Pause

try {
    Connect-ServiceFabricCluster
    Get-ServiceFabricClusterHealth
} catch {
    Write-Error "Please run this script from one of the new nodes in the cluster."
}

Write-Host "Please do not continue unless the Cluster is healthy and both Scale Sets are present in the SFCluster."
Pause

$nodesToDisable = Get-ServiceFabricNode | Where NodeName -match "_($scaleSetToDisable)_\d+"
$OldSeedCount = ( $nodesToDisable | Where IsSeedNode -eq  $true | Measure-Object).Count
$nodesToEnable = Get-ServiceFabricNode | Where NodeName -match "_($scaleSetToEnable)_\d+"

if($OldSeedCount -eq 0){
    Write-Error "Node Seed count must be greater than zero."
    exit
}

if($nodesToDisable.Count -eq 0){
    Write-Error "No nodes to disable found."
    exit
}

if($nodesToEnable.Count -eq 0){
    Write-Error "No nodes to enable found."
    exit
}

If (-not ($nodesToEnable.Count -ge $OldSeedCount)) {
    Write-Error "The new VM Scale Set must have at least $OldSeedCount nodes in order for the Seed Nodes to migrate over."
    exit
}

Write-Host "Disabling nodes in VMSS $scaleSetToDisable. Are you sure?"
Pause

foreach($node in $nodesToDisable){
    Disable-ServiceFabricNode -NodeName $node.NodeName -Intent RemoveNode -Force
}

Write-Host "Checking node status..."
$loopTimeout = 360
$loopWait = 60
$oldNodesDeactivated = $false
$newSeedNodesReady = $false

while ($loopTimeout -ne 0) {
    Get-Date -Format o
    Write-Host
    Write-Host "Nodes To Remove"

    foreach($nodeToDisable in $nodesToDisable) {
        $state = Get-ServiceFabricNode -NodeName $nodeToDisable.NodeName
        $msg = "{0} NodeDeactivationInfo: {1} IsSeedNode: {2} NodeStatus {3}" -f $nodeToDisable.NodeName, $state.NodeDeactivationInfo.Status, $state.IsSeedNode, $state.NodeStatus
        Write-Host $msg
    }

    $oldNodesDeactivated = ($nodesToDisable |  Where-Object { ($_.NodeStatus -eq [System.Fabric.Query.NodeStatus]::Disabled) -and ($_.NodeDeactivationInfo.Status -eq "Completed") } | Measure-Object).Count -eq $nodesToDisable.Count

    Write-Host
    Write-Host "Nodes To Add Status"

    foreach($nodeToEnable in $nodesToEnable) {
        $state = Get-ServiceFabricNode -NodeName $nodeToEnable.NodeName
        $msg = "{0} IsSeedNode: {1}, NodeStatus: {2}" -f $nodeToEnable.NodeName, $state.IsSeedNode, $state.NodeStatus
        Write-Host $msg
    }
    $newSeedNodesReady = ($nodesToEnable |  Where-Object { ($_.NodeStatus -eq [System.Fabric.Query.NodeStatus]::Up) -and $_.IsSeedNode} | Measure-Object).Count -ge $OldSeedCount
    if($oldNodesDeactivated -and $newSeedNodesReady) {
        break
    }
    $loopTimeout -= 1
    Start-Sleep $loopWait
}

if (-not ($oldNodesDeactivated)) {
    Write-Error "A node failed to deactivate within the time period specified."
    exit
}

$loopTimeout = 180
while ($loopTimeout -ne 0) {
    Write-Host
    Write-Host "Nodes To Add Status"

    foreach($nodeToEnable in $nodesToEnable) {
        $state = Get-ServiceFabricNode -NodeName $nodeToEnable.NodeName
        $msg = "{0} IsSeedNode: {1}, NodeStatus: {2}" -f $nodeToEnable.NodeName, $state.IsSeedNode, $state.NodeStatus
        Write-Host $msg
    }
    $newSeedNodesReady = ($nodesToEnable |  Where-Object { ($_.NodeStatus -eq [System.Fabric.Query.NodeStatus]::Up) -and $_.IsSeedNode} | Measure-Object).Count -ge $OldSeedCount
    if($newSeedNodesReady) {
        break
    }
    $loopTimeout -= 1
    Start-Sleep $loopWait
}

$NewSeedNodes = Get-ServiceFabricNode | Where-Object {($_.NodeName -match "_($scaleSetToEnable)_\d+") -and ($_.IsSeedNode -eq $True)}
Write-Host "New Seed Nodes are:"
$NewSeedNodes | Select NodeName
$NewSeedNodesCount = ($NewSeedNodes  | Measure-Object).Count

if($NewSeedNodesCount -ge $OldSeedCount) {
    Write-Host "Removing the scale set $scaleSetToDisable"
    Remove-AzureRmVmss -ResourceGroupName $ResourceGroupName -VMScaleSetName $scaleSetToDisable -Force
    Write-Host "Removed scale set $scaleSetToDisable"

    Write-Host "Removing Node State for old nodes"
    $nodesToDisable | Remove-ServiceFabricNodeState -Force
    Write-Host "Done"

    Get-ServiceFabricClusterHealth
    Get-ServiceFabricNode
} else {
    Write-Host "New Seed Nodes do not match the minimum requirements $NewSeedNodesCount."
    Write-Host "Manually run  Remove-AzureRmVmss"
    Write-Host "Then Manually run  Remove-ServiceFabricNodeState"
    Get-ServiceFabricClusterHealth
    Get-ServiceFabricNode
}

Enable-InternetExplorerESC

This script is extremely useful, you can see the progress of the transfer of seed nodes and disabling of existing primary node types.

You know it is successful, when the old nodes have ZERO seed nodes. All SEED nodes must transfer over to the new nodes, and all nodes in the old  scale set shoul dbe set to false by the end of the script execution.

MoveDNSToNewPublicIPController.ps1

Lastly you MUST update DNS to use the original CNAME . This script can help with this, what it does is actually detach the original internal Azure CNAME from the old public IP and move it to your new public IP attached to the new load balancer.




param (
        [Parameter(Mandatory = $true)]
        [string]
        $subscriptionName,

        [Parameter(Mandatory = $true)]
        [string]
        $oldLoadBalancerName,

        [Parameter(Mandatory = $true)]
        [string]
        $resourceGroupName=,

        [Parameter(Mandatory = $true)]
        [string]
        $oldPublicIpName=,

        [Parameter(Mandatory = $true)]
        [string]
        $newPublicIpName=
)

    Install-Module AzureRM.Network -Force
    Import-Module AzureRM.Network -Force

    $ErrorActionPreference = "Stop"
    Login-AzureRmAccount -SubscriptionName $subscriptionName

    Write-Host "Are you sure you want to do this. There will be brief connectivty downtime?"
    Pause

    $oldprimaryPublicIP = Get-AzureRmPublicIpAddress -Name $oldPublicIpName -ResourceGroupName $resourceGroupName
    $primaryDNSName = $oldprimaryPublicIP.DnsSettings.DomainNameLabel
    $primaryDNSFqdn = $oldprimaryPublicIP.DnsSettings.Fqdn
    
    if($primaryDNSName.Length -gt 0 -and $primaryDNSFqdn -gt 0) {
        Write-Host "Found the Primary DNS Name" $primaryDNSName
        Write-Host "Found the Primary DNS FQDN" $primaryDNSFqdn
    } else {
        Write-Error "Could not find the DNS attached to Old IP $oldprimaryPublicIP"
        Exit
    }
    
        Write-Host "Moving the Azure DNS Names to the new Public IP"
    $PublicIP = Get-AzureRmPublicIpAddress -Name $newPublicIpName -ResourceGroupName $resourceGroupName
    $PublicIP.DnsSettings.DomainNameLabel = $primaryDNSName
    $PublicIP.DnsSettings.Fqdn = $primaryDNSFqdn
    Set-AzureRmPublicIpAddress -PublicIpAddress $PublicIP

    Get-AzureRmPublicIpAddress -Name $newPublicIpName -ResourceGroupName $resourceGroupName
    Write-Host "Transfer Done"

    Write-Host "Removing Load Balancer related to old Primary NodeType."
    Write-Host "Are you sure?"
    Pause

    Remove-AzureRmLoadBalancer -Name $oldLoadBalancerName -ResourceGroupName $resourceGroupName -Force
    Remove-AzureRmPublicIpAddress -Name $oldPublicIpName -ResourceGroupName $resourceGroupName -Force

    Write-Host "Done"

Summary

In this article you followed the process to:

  • Configure ARM to add a new VMSS with OS, Data Disk and Operating System
  • Add a new Virtual Machine Scale Set to an Existing Service Fabric Node Type
  • Ran a powershell script controller to monitor the outcome of the VMSS transfer.
  • Transferred the original management DNS CNAME to the new Public IP Address

Conclusion

This project requires a lot of testing for your environment, allocate at least a a few days to test the entire process before you try it out on your production services.

HTH

Puppet Enterprise – Structure your Hiera Data

Synopsis

This post will discuss how to structure your Hiera Data, so that your profiles will automatically inject the parameters.

Why? So we can keep our profile classes and other classes super clean and succinct.

If you have If else statements in your classed depending on what environment or node the code is running on, you might have a code structure smell. Lets dig in.

Assumptions

You are using a Puppet Control Repository and leveraging Code Manager (R10K) to manage your code with Puppet Enterprise

Secondly you are using Profiles and Roles pattern to structure your classes.

I highly recommend you download the Puppet Control Repository template here.

Profiles and Roles

The most important aspect to consider is structuring your Profiles and Roles to accept parameters that can be resolved and matched to Hiera Data.

Here we have a role for all our Jumpboxes that we can use to remote into.
As we can see it will have the following profiles applied:


class role::jumpbox {
include profile::base
include profile::jumpbox::jumpboxsoftware
include profile::jumpbox::firewall
include profile::jumpbox::hosts
}

Lets pick one of these profiles that require data from Hiera.


class profile::jumpbox::hosts (
  String $hostname = 'changeme',
  String $ip = 'changeme',
)
{
  host { $hostname:
    ensure => present,
    ip     => $ip,
  }
}

The above profile ensures that the /etc/hosts file has some entries in it.

It accepts two parameters:
profile::jumpbox::hosts::ip

profile::jumpbox::hosts::hostname

Similar to Java or C# we can use a sort of dependency injection technique, where puppet will automatically look for this parameter in Hiera; a key/value store.

Hiera

The trick is to structure your Hiera Data and use the same Fully Qualified Names in the keys.

Each environment needs a different set of host names.

I then have the following structure in the control repro

.data\<environment1>\jumpbox\conf.yaml
.data\<environment2>\jumpbox\conf.yaml
.data\<environment3>\jumpbox\conf.yaml

Each folder in data represents an Environment in Puppet Classifications:

The second important convention is we use a geography variable in each Environment to resolve Hiera Data automatically.

Go to your Puppet Master Enterprise Web Console and manage the Classifications.

What you are doing is creating a variable that can be used by the hiera.yaml file to dynamically load data for the correct environment when the agent runs.

On the Puppet Master we need to setup our environments to match the Control Repository and add the magic variable. Any Node that runs the puppet agent will then have this variable set. This can then be used to load the corresponding Hiera config file.

Here we can see Environment1 has a variable defined called geography that matches the Environment name. We can then leverage this convention:

Puppet Profile -> Hiere Data lookup -> Folder that matches the variable name -> resolve parameter

This is all done automatically for you.

Puppet Control Repository Structure

The repository then looks like this:

Let us dig a little deeper and see how this structure is configured.

hiera.yaml

.\hiera.yaml

This file now contains the instructions to tell Hiera how to load our data.

—hiera.yaml—


---
version: 1

defaults:
  datadir: "data"

hierarchy:
  - name: 'Yaml Key Value Store'
    data_hash: yaml_data
    paths:
      - "%{geography}/jumpbox/conf.yaml"
      - "common.yaml"

  - name: "Encrypted Data"
    lookup_key: eyaml_lookup_key
    paths: 
      - "%{geography}/jumpbox/secrets.eyaml"
      - "common.eyaml"
    options:
      pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
      pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem

Data – yaml

the .yaml files will contain the same variable names (fully qualified) that match the PROFILE files e.g.

—conf.yaml—


profile::jumpbox::hosts::hostname: 'rdp.rangerrom.com'
profile::jumpbox::hosts::ip: 8.8.8.8'

As you can see above. As long as your profiles and parameters match, Hiera will automatically inject the correct parameter for each environment.

Hiera will resolve – %{geography}/jumpbox/conf.yaml

In the Puppet master you had setup your classifications, so when the puppet agent runs on Environment1 nodes, it will get the jumpbox/conf.yaml that matches the variable name geography=”Environment1″

Encrypted Data – eyaml

Encrypted data is just as easy to store.
* Generate the encrypted data.
* Store the data in an eyaml file in the same folder as the yaml data.
* Add a path to the data in the hiera.yaml file.

We have encrypted data e.g. the default local admin account setup via the profile – include profile::base
We use the Puppet Master private key to generate the encrypted data, see the end of this blog on how to create encrypted data.

—secrets.eyaml—


profile::base::adminpassword: >
    ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAnMWlddVoU9lC8tBNvOLI9OYI6xtCD0y3NIVe
    Ylm25dUZ8sqGP+yVQ8Y0P5xIse5f/WVOkavByZJK5yV4fDYFpD6IhXk4IJUe
    dVUw8VmO/RG84AknDDrtNPlSPm4uQqYPOOa0BmgO1iiOY4rcAxhFzT5nzod3
    MIK7lmbuP859R5jtJ5PZxZKCNERGY+dxUZfcdPs0/zr/KgLGcHc/awzYtEuI
    0tOGPp80gTVkhmCHO7KuClsg97XTRGi0BfiuiyjOWLIeAx5hbhMHi65ZPl5U
    MlJFoTA1nw3ATcC6NL3ikECWaQrt2xyxZ1uoYKqvN0ClsFLIqBQ1gXRTvQPD
    SlBQqDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCWLuT77kT6q/ojfjKx
    wk17gBATvEM58mGyP5CGbMqlbEip]

How to Encrypt Data

SSH into the Puppet Master. Locate your Puppet Master Certificates. Then run the following


puppetmaster@rangerrom.com:~$ sudo /opt/puppetlabs/puppet/bin/eyaml encrypt -p --pkcs7-public-key=/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem

Enter password: ***
string: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAnMWlddVoU9lC8tBNvOLI9OYI6xtCD0y3NIVeYlm25dUZ8sqGP+yVQ8Y0P5xIse5f/WVOkavByZJK5yV4fDYFpD6IhXk4IJUedVUw8VmO/RG84AknDDrtNPlSPm4uQqYPOOa0BmgO1iiOY4rcAxhFzT5nzod3MIK7lmbuP859R5jtJ5PZxZKCNERGY+dxUZfcdPs0/zr/KgLGcHc/awzYtEuI0tOGPp80gTVkhmCHO7KuClsg97XTRGi0BfiuiyjOWLIeAx5hbhMHi65ZPl5UMlJFoTA1nw3ATcC6NL3ikECWaQrt2xyxZ1uoYKqvN0ClsFLIqBQ1gXRTvQPDSlBQqDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCWLuT77kT6q/ojfjKxwk17gBATvEM58mGyP5CGbMqlbEip]

OR

block: >
    ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAnMWlddVoU9lC8tBNvOLI9OYI6xtCD0y3NIVe
    Ylm25dUZ8sqGP+yVQ8Y0P5xIse5f/WVOkavByZJK5yV4fDYFpD6IhXk4IJUe
    dVUw8VmO/RG84AknDDrtNPlSPm4uQqYPOOa0BmgO1iiOY4rcAxhFzT5nzod3
    MIK7lmbuP859R5jtJ5PZxZKCNERGY+dxUZfcdPs0/zr/KgLGcHc/awzYtEuI
    0tOGPp80gTVkhmCHO7KuClsg97XTRGi0BfiuiyjOWLIeAx5hbhMHi65ZPl5U
    MlJFoTA1nw3ATcC6NL3ikECWaQrt2xyxZ1uoYKqvN0ClsFLIqBQ1gXRTvQPD
    SlBQqDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCWLuT77kT6q/ojfjKx
    wk17gBATvEM58mGyP5CGbMqlbEip]
puppetmaster@rangerrom.com:~$

 

Puppet – Join Ubuntu 16.04 Servers to an Azure Windows Active Directory Domain

We use Azure Active Directory Domain Services and wanted a single sign on solution for Windows and Linux. The decision was made to join all servers to the Windows Domain in addition to having SSH Key auth.

I assume you know how to use Puppet Code Manager and have a Puppet Control Repository to manage Roles and Profiles.

Ensure you have a Active Directory Service Account that has permissions to join a computer to a domain. You can store this user in Hiera.

The module we need is called Realmd, however the current version (Version 2.3.0 released Sep 3rd 2018) does not support Ubuntu  16.04. So I have a forked repro here that you can use.

Modules

puppetfile


mod 'romiko-realmd',
  :git    => 'https://github.com/Romiko/realmd.git',
  :branch => 'master'

mod 'saz-resolv_conf',                  '4.0.0'

linuxdomain.pp


class profile::domain::linuxdomain (
  String $domain = 'RANGERROM.COM',
  String $user = 'romiko.derbynew@rangerrom.com',
  String $password = 'info',
  Array  $aadds_dns = ['10.0.103.36','10.0.103.37']
) {

  $hostname = $trusted['hostname']
  $domaingroup = downcase($domain)

  host { 'aaddshost':
    ensure  => present,
    name    => "${hostname}.${domain}",
    comment => 'Azure Active Directory Domain Services',
    ip      => '127.0.0.1',
  }

  class { 'resolv_conf':
    nameservers => $aadds_dns,
    searchpath  => [$domain],
  }

  exec { "waagent":
    path        => ['/usr/bin', '/usr/sbin', '/bin'],
    command     => "waagent -start",
    refreshonly => true,
    subscribe   => [File_line['waagent.conf']],
  }

  exec { "changehostname":
    path        => ['/usr/bin', '/usr/sbin', '/bin'],
    command     => "hostnamectl set-hostname ${hostname}.${domain}",
    refreshonly => true,
    subscribe   => [File_line['waagent.conf']],
  }

  file_line { 'waagent.conf':
    ensure             => present,
    path               => '/etc/waagent.conf',
    line               => 'Provisioning.MonitorHostName=y',
    match              => '^Provisioning.MonitorHostName=n',
    append_on_no_match => false,
  }

  class { 'ntp':
    servers => [ $domain ],
  }

  class { '::realmd':
    domain                => $domain,
    domain_join_user      => $user,
    domain_join_password  => $password
  } ->

  file_line { 'sssd.conf':
  ensure             => present,
  path               => '/etc/sssd/sssd.conf',
  line               => '#use_fully_qualified_names = True',
  match              => '^use_fully_qualified_names',
  append_on_no_match => false,
  }

  file_line { 'common-session':
  path => '/etc/pam.d/common-session',
  line => 'session required        pam_mkhomedir.so skel=/etc/skel/ umask=0077',
  }

  file_line { 'sudo_rule':
  path => '/etc/sudoers',
  line => "%aad\ dc\ administrators@${domaingroup} ALL=(ALL) NOPASSWD:ALL",
  }
}

The above code ensures:

  • Its fqdn is in the /etc/hosts file so it can resolve to itself.
  • The Active Directory DNS servers are in resolv.conf and the default search domain is set. This allows nslookup to <computername> to work as well as <computername>.<domainname>
  • Renames the Linux Server and sets Azure to monitor host name changes
  • Sets the Network Time Protocol to use the Active Directory Severs
  • Uses Realmd 
    • Join the domain
    • configure Sssd, samba etc
    • We using my fork until the pull request is accepted
  • Makes changes to SSSD and PAM to ensure smooth operations with Azure Active Directory Domain Services
  • Adds the administrators from AADDS to the sudoers

 

With the above running on your Linux agent, you will have Linux machines using the domain and can leverage single sign on.

CASE SENSITIVE –  Ensure you use Upper Case for $domain.

My source of inspiration to do this configuration came from a Microsoft Document here:

https://docs.microsoft.com/en-us/azure/active-directory-domain-services/active-directory-ds-join-ubuntu-linux-vm

However, here is a Hiera config file to get you going for this class if you like using a key value data source.

common.yaml

#AADDS Domain must be Upper Case else Kerberos Tickets fail
profile::domain::linuxdomain::domain: 'RANGERROM.COM'
profile::domain::linuxdomain::user: 'serviceaccounttojoindomain@RANGERROM.COM'
profile::domain::linuxdomain::aadds_dns:
        - '10.0.103.36'
        - '10.0.103.37'

common.eyaml

profile::domain::linuxdomain::password:
    ENC[PKCS7,MIIBiQ.....==]

In my next article, I will write about setting up a flexible Puppet Control Repository and leveraging Hiera.

Now I can login to the server using my Azure Active Directory credentials that I even use for Outlook, Skype and other Microsoft Products.

ssh -l romiko.derbynew@rangerrom.com myserver   (resolv.conf has a search entry for rangerrom.com, so the suffix will get applied)
ssh -l romiko.derbynew@rangerrom.com myserver.rangerrom.com

Cheers

Azure Database for PostgreSQL – Backup/Restore

PSQL has some awesome tools pg_dump and pg_restore that can assist and cut down restore times in the event of a recovery event.

Scenario

User error – If a table was accidentally emptied at noon today:

• Restore point in time just before noon and retrieve the missing table and data from that new copy of the server (Azure Portal or CLI) , Powershell not supported yet.
• Update firewall rules to allow traffic internally on the new SQL Instance

Then log onto a JumpBox in the Azure cloud.
• Dump the restored database to file (pg_dump)


pg_dump -Fc -v -t mytable -h Source.postgres.database.azure.com -U pgadmin@source -d mydatabase > c:\Temp\mydatabase .dump

The switch -Fc – gives us flexibility with pg_restore later (Table Level Restore)

• Copy the table back to the original server using  pg_restore

You may need to truncate the target table before, as we doing data only restore (-a)
Table Only (not schema):


pg_restore -v -A -t mytable -h Target.postgres.database.azure.com -p 5432 -U pgadmin@target -d mydatabase "c:\Temp\mydatabase .dump"

note: Use a VM in the Azure cloud that has access to SQL via VNET Rules or SQL Firewalls.

We currently use PSQL 10. There is a known issue with VNet Rules. Just contact Microsoft if you need them, else use firewall rules for now.

Issues

VNET Rules and PostgreSQL 10

Issue Definition: If using VET rules to control access to the PostgreSQL 10.0 server, customer gets the following error: psql ‘sslmode=require host=server .postgres.database.azure.com port=5432 dbname=postgres’ –username=pgadmin@server psql: FATAL: unrecognized configuration parameter ‘connection_Vnet’ This is a known issue: https://social.msdn.microsoft.com/Forums/azure/en-US/0e99fb68-47fd-4053-a8be-5f8b87b3a660/azure-database-for-postgresql-vnet-service-endpoints-not-working?forum=AzureDatabaseforPostgreSQL

DNS CNAME

nslookup restoretest2.postgres.database.azure.com

Non-authoritative answer:

Name:    cr1.westus1-a.control.database.windows.net

Address:  23.9.34.71

Aliases:  restoretest2.postgres.database.azure.com

nslookup restoretest1.postgres.database.azure.com

Non-authoritative answer:

Name:    cr1.westus1-a.control.database.windows.net

Address:  23.9.34.71

Aliases:  restoretest1.postgres.database.azure.com

The Host is the same server. The way the connection string works is on the USERNAME.

These two command will connect to the SAME Server instance


pg_restore -v -a -t mytable -h restoretest2.postgres.database.azure.com -p 5432 -U pgadmin@restoretest1 -d mydatabase  "c:\Temp\mydatabase.dump"


pg_restore -v -a -t mytable -h restoretest1.postgres.database.azure.com -p 5432 -U pgadmin@restoretest1 -d mydatabase   "c:\Temp\mydatabase.dump"

The hostname just gets us to the Microsoft PSQL server, it is the username that points us to the correct instance. This is extremely important when updating connection strings on clients!

Happy Life – Happy Wife

Windows Azure – Restore Encrypted VM – BEK and KEK

When restoring an Encrypted Virtual Machine that is managed by Windows Azure, you will need to use PowerShell.

This is a two stage process.

Restore-BackupImageToDisk … | Restore-EncryptedBekVM …

Stage 1

Retrieve the Backup Container – This will contain the encrypted disks and json files with the secret keys.

  • keyEncryptionKey
  • diskEncryptionKey

This will allow you to retrieve the disks and JSON metadata in step 2.

Restore-BackupImageToDisk.ps1

param(
    [Parameter(Mandatory=$true)]
    [string]
    $SubscriptionName="RangerRom",

    [Parameter(Mandatory=$true)]
    [string]
    $ResourceGroup="Ranger-Rom-Prod",

    [Parameter(Mandatory=$true)]
    [string]
    $StorageAccount="RangerRomStorage",

    [Parameter(Mandatory=$true)]
    [string]
    $RecoveryVaultName="RecoveryVault",

    [Parameter(Mandatory=$true)]
    [string]
    $VMName,

    [Parameter(Mandatory=$true)]
    [datetime]
    $FromDate
)

Login-AzureRmAccount
Get-AzureRmSubscription -SubscriptionName $SubscriptionName | Select-AzureRmSubscription
$endDate = Get-Date

$namedContainer = Get-AzureRmRecoveryServicesBackupContainer  -ContainerType "AzureVM" -Status "Registered" | Where { $_.FriendlyName -eq $VMName }
$backupitem = Get-AzureRmRecoveryServicesBackupItem -Container $namedContainer  -WorkloadType "AzureVM"

$rp = Get-AzureRmRecoveryServicesBackupRecoveryPoint -Item $backupitem -StartDate $FromDate.ToUniversalTime() -EndDate $enddate.ToUniversalTime()
$restorejob = Restore-AzureRmRecoveryServicesBackupItem -RecoveryPoint $rp[0] -StorageAccountName $StorageAccount -StorageAccountResourceGroupName $ResourceGroup
Wait-AzureRmRecoveryServicesBackupJob -Job $restorejob -Timeout 43200
$restorejob = Get-AzureRmRecoveryServicesBackupJob -Job $restorejob
$details = Get-AzureRmRecoveryServicesBackupJobDetails -Job $restorejob
$details

Stage 2

We will grab the job details via the pipeline if provided or go find the earliest Restore Job after the FromDate and then initiate a restore of all the encrypted disks to the new Virtual Machine.

Restore-EncryptedBekVM.ps1

[CmdletBinding()]
param(
    [Parameter(ValueFromPipeline=$True, Mandatory=$false, HelpMessage="Requires a AzureVmJobDetails object e.g. Get-AzureRmRecoveryServicesBackupJobDetails. Optional.")]
    $JobDetails,

    [Parameter(Mandatory=$true, HelpMessage="Assumes osDisks, DataDisk, RestorePoints, AvailbilitySet, VM, Nic are all in same resource group.")]
    [string]
    $DefaultResourceGroup="Ranger-Rom-Prod",
    
    [Parameter(Mandatory=$true)]
    [string]
    $SourceVMName="Lisha",

    [Parameter(Mandatory=$true)]
    [string]
    $TargetVMName,

    [Parameter(Mandatory=$true)]
    [string]
    $BackupVaultName="RecoveryVault",

    [Parameter(Mandatory=$true)]
    [datetime]
    $FromDate
)

Begin {
    $FromDate = $FromDate.ToUniversalTime()
    Write-Verbose "Started. $(Get-Date)"
}

Process {
	Write-Verbose "Retrieving Backup Vault."
	if(-not $JobDetails)
	{
		Get-AzureRmRecoveryServicesVault -Name $BackupVaultName -ResourceGroupName $DefaultResourceGroup | Set-AzureRmRecoveryServicesVaultContext
		$Job = Get-AzureRmRecoveryServicesBackupJob -Status Completed -Operation Restore -From $FromDate | Sort -Property StartTime | Where { $_.WorkloadName -eq $SourceVMName} | Select -Last 1
		if($Job -eq $null) {
			throw "Job $workLoadName not found."
		}
		$JobDetails = Get-AzureRmRecoveryServicesBackupJobDetails -Job $Job
	}

	Write-Verbose "Query the restored disk properties for the job details."
	$properties = $JobDetails.properties
	$storageAccountName = $properties["Target Storage Account Name"]
	$containerName = $properties["Config Blob Container Name"]
	$blobName = $properties["Config Blob Name"]

	Write-Verbose "Found Restore Blob Set at $($Properties['Config Blob Uri'])"
	Write-Verbose "Set the Azure storage context and restore the JSON configuration file."

	Set-AzureRmCurrentStorageAccount -Name $storageaccountname -ResourceGroupName $DefaultResourceGroup
	$folder = New-Item "C:\RangerRom" -ItemType Directory -Force
	$destination_path = "C:\$($folder.Name)\$SourceVMName.config.json"
	Get-AzureStorageBlobContent -Container $containerName -Blob $blobName -Destination $destination_path
	Write-Verbose "Restore config saved to file. $destination_path"
	$restoreConfig = ((Get-Content -Path $destination_path -Raw -Encoding Unicode)).TrimEnd([char]0x00) | ConvertFrom-Json

	# 3. Use the JSON configuration file to create the VM configuration.
	$oldVM = Get-AzureRmVM | Where { $_.Name -eq $SourceVMName }
	$vm = New-AzureRmVMConfig -VMSize $restoreConfig.'properties.hardwareProfile'.vmSize -VMName $TargetVMName -AvailabilitySetId $oldVM.AvailabilitySetReference.Id
	$vm.Location = $oldVM.Location

	# 4. Attach the OS disk and data disks - Managed, encrypted VMs (BEK only
	$bekUrl = $restoreConfig.'properties.storageProfile'.osDisk.encryptionSettings.diskEncryptionKey.secretUrl
	$keyVaultId = $restoreConfig.'properties.storageProfile'.osDisk.encryptionSettings.diskEncryptionKey.sourceVault.id
	$kekUrl = $restoreConfig.'properties.storageProfile'.osDisk.encryptionSettings.keyEncryptionKey.keyUrl
	$storageType = "StandardLRS"
	$osDiskName = $vm.Name + "_Restored_" + (Get-Date).ToString("yyyy-MM-dd-hh-mm-ss") + "_osdisk"
	$osVhdUri = $restoreConfig.'properties.storageProfile'.osDisk.vhd.uri
	$diskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $restoreConfig.location -CreateOption Import -SourceUri $osVhdUri
	$osDisk = New-AzureRmDisk -DiskName $osDiskName -Disk $diskConfig -ResourceGroupName $DefaultResourceGroup
	Set-AzureRmVMOSDisk -VM $vm -ManagedDiskId $osDisk.Id -DiskEncryptionKeyUrl $bekUrl -DiskEncryptionKeyVaultId $keyVaultId -KeyEncryptionKeyUrl $kekUrl -KeyEncryptionKeyVaultId $keyVaultId -CreateOption "Attach" -Windows

	$count = 0
	foreach($dd in $restoreConfig.'properties.storageProfile'.dataDisks)
	{
		$dataDiskName = $vm.Name + "_Restored_" + (Get-Date).ToString("yyyy-MM-dd-hh-mm-ss") + "_datadisk" + $count ;
		$dataVhdUri = $dd.vhd.uri;
		$dataDiskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $restoreConfig.location -CreateOption Import -SourceUri $dataVhdUri
		$dataDisk = New-AzureRmDisk -DiskName $dataDiskName -Disk $dataDiskConfig -ResourceGroupName $DefaultResourceGroup
		Add-AzureRmVMDataDisk -VM $vm -Name $dataDiskName -ManagedDiskId $dataDisk.Id -Lun $dd.Lun -CreateOption "Attach"
		$count += 1
	}

	Write-Verbose  "Setting the Network settings."

	$oldNicId = $oldVM.NetworkProfile.NetworkInterfaces[0].Id
	$oldNic = Get-AzureRmNetworkInterface -Name $oldNicId.Substring($oldNicId.LastIndexOf("/")+ 1) -ResourceGroupName $DefaultResourceGroup
	$subnetItems =  $oldNic.IpConfigurations[0].Subnet.Id.Split("/")
	$networkResourceGroup = $subnetItems[$subnetItems.IndexOf("resourceGroups")+1]
	$nicName= $vm.Name + "_nic_" + (New-Guid).Guid
	$nic = New-AzureRmNetworkInterface -Name $nicName -ResourceGroupName $networkResourceGroup -Location $oldNic.location -SubnetId $oldNic.IpConfigurations[0].Subnet.Id
	$vm=Add-AzureRmVMNetworkInterface -VM $vm -Id $nic.Id
	$vm.DiagnosticsProfile = $oldVM.DiagnosticsProfile

	Write-Verbose "Provisioning VM."
	New-AzureRmVM -ResourceGroupName $oldVM.ResourceGroupName -Location $oldVM.Location -VM $vm

}

End {
    Write-Verbose "Completed. $(Get-Date)"
}


 Usage

.\Restore-BackupImageToDisk … | .\Restore-EncryptedBekVM.ps1 …

Summary

Ensure you have Azure VM Backups and they are backed on on a regular schedule.

Use this script to backup a Restore Point from a Recovery Vault Container.

Remember it will pick the earliest date of a Restore Point relative to your From Date. If there are 4 Restores in the recovery vault from 1 May, it will pick the first one.

Remember to:

  • Decommission the old VM
  • Update DNS with the new ip address
  • Update the Load Balance Sets (If using a Load Balancer)