Service Fabric – Upgrading VMSS Disks, Operating System on Primary Node Type

How do you upgrade the existing Data Disk on a primary Node Type Virtual Machine ScaleSet in Service Fabric?

How do you upgrade the existing Operating System on a primary Node Type VMSS in Service Fabric?

How do you move the Data Disk on a primary Node Type VMSS in Service Fabric?

How do you monitor the status during the upgrade, so you know exactly how many seed nodes have migrated over to the new scale set?

note – We successfully increased the SKU size as well, however this is not supported by Microsoft. However just increase your SKU in ARm and later, after the successful transfer to the new VMSS, run Update-AzureRmServiceFabricDurability.


  • You have knowledge to use ARM to deploy an Azure Load Balancer
  • You have knowledge to use ARM to deploy a VMSS Scale Set
  • Service Fabric Durability Tier/Reliability Tier must be at least Silver
  • Keep the original Azure DNS name on the Load Balancer that is used to connect to the Service Fabric Endpoint. Very Important to write it down as a backup
  • You will need to reduce the TTL of all your DNS settings to reduce downtime during the upgrade which will just be the TTL value e.g. 10 minutes. (Ensure you have access to your primary DNS provider to do this)
  • Prepare an ARM template to add the new Azure Load Balancer that the new VMSS scaleset will attach to (Backend Pool)
  • Prepare an ARM template to add the new VMSS to an existing Service Fabric primary Node Type
  • Deploy the new Azure Load Balancer + Virtual Machine Scale Set to the Service Fabric Primary node
  • Run the RemoveScaleSetFromClusterController.ps1 – Run this script on the NEW node in the NEW VMSS. This script will monitor and facilitate moving the Primary Node Type to the new VMSS for you.  It will show you the status of the Seed nodes moving from the original Primary Node Type to the new VMSS.
  • When it completed, the last part will be to update DNS.
  • Run MoveDNSToNewPublicIPController.ps1

ARM Templates

You will need only 2 templates. One to Deploy a new Azure Load Balancer and one to Deploy the new VMSS Scale Set to the existing Service Fabric Cluster.

You will also need a powershell script that will run a custom script extension.

Custom Script – prepare_sf_vm.ps1

$disks = Get-Disk | Where partitionstyle -eq 'raw' | sort number

$letters = 70..89 | ForEach-Object { [char]$_ }
$count = 0
$label = "datadisk"

foreach ($disk in $disks) {
    $driveLetter = $letters[$count].ToString()
    $disk | 
    Initialize-Disk -PartitionStyle GPT -PassThru |
    New-Partition -UseMaximumSize -DriveLetter $driveLetter |
    Format-Volume -FileSystem NTFS -NewFileSystemLabel "$label$count" -Confirm:$false -Force

# Disable Windows Update
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU' -Name NoAutoUpdate -Value 1


Load Balancer – azuredeploy_servicefabric_loadbalancer.json

Use your particular Load Balancer ARM Templates. No need to attached a backend pool, as this will be done by the VMSS script below.

Service Fabric attach new VMSS – azuredeploy_add_new_VMSS_to_nodeType.json

Create your own VMSS scaleset that you attach to Service fabric. The important aspect are the following.

nodeTypeRef (To attach VMSS to existing PrimaryNodeType).
dataPath (To use a new Disk for data)
dataDisk (to add a new managed physical disk)

We use F:\ onwards as D is reserved for Temp storage and E: is reserved for a CD ROM in Azure VM’s.

                                "name": "[concat('ServiceFabricNodeVmExt',variables('vmNodeType0Name'))]",
                                "properties": {
                                    "type": "ServiceFabricNode",
                                    "autoUpgradeMinorVersion": true,
                                    "protectedSettings": {
                                        "StorageAccountKey1": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-05-01-preview').key1]",
                                        "StorageAccountKey2": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-05-01-preview').key2]"
                                    "publisher": "Microsoft.Azure.ServiceFabric",
                                    "settings": {
                                        "clusterEndpoint": "[parameters('existingClusterConnectionEndpoint')]",
                                        "nodeTypeRef": "[parameters('existingNodeTypeName')]",
                                        "dataPath": "F:\\\\SvcFab",
                                        "durabilityLevel": "Silver",
                                        "enableParallelJobs": true,
                                        "nicPrefixOverride": "[variables('subnet0Prefix')]",
                                        "certificate": {
                                            "thumbprint": "[parameters('certificateThumbprint')]",
                                            "x509StoreName": "[parameters('certificateStoreValue')]"
                                    "typeHandlerVersion": "1.0"
"storageProfile": {
                        "imageReference": {
                            "publisher": "[parameters('vmImagePublisher')]",
                            "offer": "[parameters('vmImageOffer')]",
                            "sku": "2016-Datacenter-with-Containers",
                            "version": "[parameters('vmImageVersion')]"
                        "osDisk": {
                            "managedDisk": {
                                "storageAccountType": "[parameters('storageAccountType')]"
                            "caching": "ReadWrite",
                            "createOption": "FromImage"
                        "dataDisks": [
                                "managedDisk": {
                                    "storageAccountType": "[parameters('storageAccountType')]"
                                "lun": 0,
                                "createOption": "Empty",
                                "diskSizeGB": "[parameters('dataDiskSize')]",
                                "caching": "None"

 "virtualMachineProfile": {
                    "extensionProfile": {
                        "extensions": [
                                "name": "PrepareDataDisk",
                                "properties": {
                                    "publisher": "Microsoft.Compute",
                                    "type": "CustomScriptExtension",
                                    "typeHandlerVersion": "1.8",
                                    "autoUpgradeMinorVersion": true,
                                    "settings": {
                                    "fileUris": [
                                    "commandToExecute": "[concat('powershell -ExecutionPolicy Unrestricted -File prepare_sf_vm.ps1 ')]"


Once you have a new VMSS scale set attached to the existing NodeType, you should see in Service Fabric the extra nodes. the next step is to disable and remove the existing VMSS scaleset. This is an online operation, so you should be fine. However later we will need to update DNS for the Cluster Endpoint. This is important for Powershell Admin tools to still connect to the Service Fabric cluster.


Remote into one of the NEW VMSS virtual machines and run the following command. It will make dead sure that your seed nodes migrate over. it can take a long time (Microsoft docs say it takes a long time, how long?). it depends, for a cluster with 5 seed nodes, it took nearly 4 hours! So be patient and update the loop timeout to match your environment, increase the timeout if you have more than 5 seed nodes. My general rule is allow 45 minutes per seed node transfer.

#Requires -Version 5.0
#Requires -RunAsAdministrator

param (
    [Parameter(Mandatory = $true)]

    [Parameter(Mandatory = $true)]

    [Parameter(Mandatory = $true)]

    [Parameter(Mandatory = $true)]

Install-Module AzureRM.Compute -Force

Import-Module ServiceFabric -Force
Import-Module AzureRM.Compute -Force

function Disable-InternetExplorerESC {
    $AdminKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}"
    $UserKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}"
    Set-ItemProperty -Path $AdminKey -Name "IsInstalled" -Value 0
    Set-ItemProperty -Path $UserKey -Name "IsInstalled" -Value 0
    Stop-Process -Name Explorer
    Write-Host "IE Enhanced Security Configuration (ESC) has been disabled." -ForegroundColor Green

function Enable-InternetExplorerESC {
    $AdminKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}"
    $UserKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}"
    Set-ItemProperty -Path $AdminKey -Name "IsInstalled" -Value 1
    Set-ItemProperty -Path $UserKey -Name "IsInstalled" -Value 1
    Stop-Process -Name Explorer
    Write-Host "IE Enhanced Security Configuration (ESC) has been enabled." -ForegroundColor Green

$ErrorActionPreference = "Stop"


Login-AzureRmAccount -SubscriptionName $subscriptionName

Write-Host "Before you continue:  Ensure IE Enhanced Security is off."
Write-Host "Before you continue:  Ensure your new scaleset is ALREADY added to the Service Fabric Cluster"

try {
} catch {
    Write-Error "Please run this script from one of the new nodes in the cluster."

Write-Host "Please do not continue unless the Cluster is healthy and both Scale Sets are present in the SFCluster."

$nodesToDisable = Get-ServiceFabricNode | Where NodeName -match "_($scaleSetToDisable)_\d+"
$OldSeedCount = ( $nodesToDisable | Where IsSeedNode -eq  $true | Measure-Object).Count
$nodesToEnable = Get-ServiceFabricNode | Where NodeName -match "_($scaleSetToEnable)_\d+"

if($OldSeedCount -eq 0){
    Write-Error "Node Seed count must be greater than zero."

if($nodesToDisable.Count -eq 0){
    Write-Error "No nodes to disable found."

if($nodesToEnable.Count -eq 0){
    Write-Error "No nodes to enable found."

If (-not ($nodesToEnable.Count -ge $OldSeedCount)) {
    Write-Error "The new VM Scale Set must have at least $OldSeedCount nodes in order for the Seed Nodes to migrate over."

Write-Host "Disabling nodes in VMSS $scaleSetToDisable. Are you sure?"

foreach($node in $nodesToDisable){
    Disable-ServiceFabricNode -NodeName $node.NodeName -Intent RemoveNode -Force

Write-Host "Checking node status..."
$loopTimeout = 360
$loopWait = 60
$oldNodesDeactivated = $false
$newSeedNodesReady = $false

while ($loopTimeout -ne 0) {
    Get-Date -Format o
    Write-Host "Nodes To Remove"

    foreach($nodeToDisable in $nodesToDisable) {
        $state = Get-ServiceFabricNode -NodeName $nodeToDisable.NodeName
        $msg = "{0} NodeDeactivationInfo: {1} IsSeedNode: {2} NodeStatus {3}" -f $nodeToDisable.NodeName, $state.NodeDeactivationInfo.Status, $state.IsSeedNode, $state.NodeStatus
        Write-Host $msg

    $oldNodesDeactivated = ($nodesToDisable |  Where-Object { ($_.NodeStatus -eq [System.Fabric.Query.NodeStatus]::Disabled) -and ($_.NodeDeactivationInfo.Status -eq "Completed") } | Measure-Object).Count -eq $nodesToDisable.Count

    Write-Host "Nodes To Add Status"

    foreach($nodeToEnable in $nodesToEnable) {
        $state = Get-ServiceFabricNode -NodeName $nodeToEnable.NodeName
        $msg = "{0} IsSeedNode: {1}, NodeStatus: {2}" -f $nodeToEnable.NodeName, $state.IsSeedNode, $state.NodeStatus
        Write-Host $msg
    $newSeedNodesReady = ($nodesToEnable |  Where-Object { ($_.NodeStatus -eq [System.Fabric.Query.NodeStatus]::Up) -and $_.IsSeedNode} | Measure-Object).Count -ge $OldSeedCount
    if($oldNodesDeactivated -and $newSeedNodesReady) {
    $loopTimeout -= 1
    Start-Sleep $loopWait

if (-not ($oldNodesDeactivated)) {
    Write-Error "A node failed to deactivate within the time period specified."

$loopTimeout = 180
while ($loopTimeout -ne 0) {
    Write-Host "Nodes To Add Status"

    foreach($nodeToEnable in $nodesToEnable) {
        $state = Get-ServiceFabricNode -NodeName $nodeToEnable.NodeName
        $msg = "{0} IsSeedNode: {1}, NodeStatus: {2}" -f $nodeToEnable.NodeName, $state.IsSeedNode, $state.NodeStatus
        Write-Host $msg
    $newSeedNodesReady = ($nodesToEnable |  Where-Object { ($_.NodeStatus -eq [System.Fabric.Query.NodeStatus]::Up) -and $_.IsSeedNode} | Measure-Object).Count -ge $OldSeedCount
    if($newSeedNodesReady) {
    $loopTimeout -= 1
    Start-Sleep $loopWait

$NewSeedNodes = Get-ServiceFabricNode | Where-Object {($_.NodeName -match "_($scaleSetToEnable)_\d+") -and ($_.IsSeedNode -eq $True)}
Write-Host "New Seed Nodes are:"
$NewSeedNodes | Select NodeName
$NewSeedNodesCount = ($NewSeedNodes  | Measure-Object).Count

if($NewSeedNodesCount -ge $OldSeedCount) {
    Write-Host "Removing the scale set $scaleSetToDisable"
    Remove-AzureRmVmss -ResourceGroupName $ResourceGroupName -VMScaleSetName $scaleSetToDisable -Force
    Write-Host "Removed scale set $scaleSetToDisable"

    Write-Host "Removing Node State for old nodes"
    $nodesToDisable | Remove-ServiceFabricNodeState -Force
    Write-Host "Done"

} else {
    Write-Host "New Seed Nodes do not match the minimum requirements $NewSeedNodesCount."
    Write-Host "Manually run  Remove-AzureRmVmss"
    Write-Host "Then Manually run  Remove-ServiceFabricNodeState"


This script is extremely useful, you can see the progress of the transfer of seed nodes and disabling of existing primary node types.

You know it is successful, when the old nodes have ZERO seed nodes. All SEED nodes must transfer over to the new nodes, and all nodes in the old  scale set shoul dbe set to false by the end of the script execution.


Lastly you MUST update DNS to use the original CNAME . This script can help with this, what it does is actually detach the original internal Azure CNAME from the old public IP and move it to your new public IP attached to the new load balancer.

param (
        [Parameter(Mandatory = $true)]

        [Parameter(Mandatory = $true)]

        [Parameter(Mandatory = $true)]

        [Parameter(Mandatory = $true)]

        [Parameter(Mandatory = $true)]

    Install-Module AzureRM.Network -Force
    Import-Module AzureRM.Network -Force

    $ErrorActionPreference = "Stop"
    Login-AzureRmAccount -SubscriptionName $subscriptionName

    Write-Host "Are you sure you want to do this. There will be brief connectivty downtime?"

    $oldprimaryPublicIP = Get-AzureRmPublicIpAddress -Name $oldPublicIpName -ResourceGroupName $resourceGroupName
    $primaryDNSName = $oldprimaryPublicIP.DnsSettings.DomainNameLabel
    $primaryDNSFqdn = $oldprimaryPublicIP.DnsSettings.Fqdn
    if($primaryDNSName.Length -gt 0 -and $primaryDNSFqdn -gt 0) {
        Write-Host "Found the Primary DNS Name" $primaryDNSName
        Write-Host "Found the Primary DNS FQDN" $primaryDNSFqdn
    } else {
        Write-Error "Could not find the DNS attached to Old IP $oldprimaryPublicIP"
        Write-Host "Moving the Azure DNS Names to the new Public IP"
    $PublicIP = Get-AzureRmPublicIpAddress -Name $newPublicIpName -ResourceGroupName $resourceGroupName
    $PublicIP.DnsSettings.DomainNameLabel = $primaryDNSName
    $PublicIP.DnsSettings.Fqdn = $primaryDNSFqdn
    Set-AzureRmPublicIpAddress -PublicIpAddress $PublicIP

    Get-AzureRmPublicIpAddress -Name $newPublicIpName -ResourceGroupName $resourceGroupName
    Write-Host "Transfer Done"

    Write-Host "Removing Load Balancer related to old Primary NodeType."
    Write-Host "Are you sure?"

    Remove-AzureRmLoadBalancer -Name $oldLoadBalancerName -ResourceGroupName $resourceGroupName -Force
    Remove-AzureRmPublicIpAddress -Name $oldPublicIpName -ResourceGroupName $resourceGroupName -Force

    Write-Host "Done"


In this article you followed the process to:

  • Configure ARM to add a new VMSS with OS, Data Disk and Operating System
  • Add a new Virtual Machine Scale Set to an Existing Service Fabric Node Type
  • Ran a powershell script controller to monitor the outcome of the VMSS transfer.
  • Transferred the original management DNS CNAME to the new Public IP Address


This project requires a lot of testing for your environment, allocate at least a a few days to test the entire process before you try it out on your production services.


Puppet Enterprise – Structure your Hiera Data


This post will discuss how to structure your Hiera Data, so that your profiles will automatically inject the parameters.

Why? So we can keep our profile classes and other classes super clean and succinct.

If you have If else statements in your classed depending on what environment or node the code is running on, you might have a code structure smell. Lets dig in.


You are using a Puppet Control Repository and leveraging Code Manager (R10K) to manage your code with Puppet Enterprise

Secondly you are using Profiles and Roles pattern to structure your classes.

I highly recommend you download the Puppet Control Repository template here.

Profiles and Roles

The most important aspect to consider is structuring your Profiles and Roles to accept parameters that can be resolved and matched to Hiera Data.

Here we have a role for all our Jumpboxes that we can use to remote into.
As we can see it will have the following profiles applied:

class role::jumpbox {
include profile::base
include profile::jumpbox::jumpboxsoftware
include profile::jumpbox::firewall
include profile::jumpbox::hosts

Lets pick one of these profiles that require data from Hiera.

class profile::jumpbox::hosts (
  String $hostname = 'changeme',
  String $ip = 'changeme',
  host { $hostname:
    ensure => present,
    ip     => $ip,

The above profile ensures that the /etc/hosts file has some entries in it.

It accepts two parameters:


Similar to Java or C# we can use a sort of dependency injection technique, where puppet will automatically look for this parameter in Hiera; a key/value store.


The trick is to structure your Hiera Data and use the same Fully Qualified Names in the keys.

Each environment needs a different set of host names.

I then have the following structure in the control repro


Each folder in data represents an Environment in Puppet Classifications:

The second important convention is we use a geography variable in each Environment to resolve Hiera Data automatically.

Go to your Puppet Master Enterprise Web Console and manage the Classifications.

What you are doing is creating a variable that can be used by the hiera.yaml file to dynamically load data for the correct environment when the agent runs.

On the Puppet Master we need to setup our environments to match the Control Repository and add the magic variable. Any Node that runs the puppet agent will then have this variable set. This can then be used to load the corresponding Hiera config file.

Here we can see Environment1 has a variable defined called geography that matches the Environment name. We can then leverage this convention:

Puppet Profile -> Hiere Data lookup -> Folder that matches the variable name -> resolve parameter

This is all done automatically for you.

Puppet Control Repository Structure

The repository then looks like this:

Let us dig a little deeper and see how this structure is configured.



This file now contains the instructions to tell Hiera how to load our data.


version: 1

  datadir: "data"

  - name: 'Yaml Key Value Store'
    data_hash: yaml_data
      - "%{geography}/jumpbox/conf.yaml"
      - "common.yaml"

  - name: "Encrypted Data"
    lookup_key: eyaml_lookup_key
      - "%{geography}/jumpbox/secrets.eyaml"
      - "common.eyaml"
      pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
      pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem

Data – yaml

the .yaml files will contain the same variable names (fully qualified) that match the PROFILE files e.g.


profile::jumpbox::hosts::hostname: ''

As you can see above. As long as your profiles and parameters match, Hiera will automatically inject the correct parameter for each environment.

Hiera will resolve – %{geography}/jumpbox/conf.yaml

In the Puppet master you had setup your classifications, so when the puppet agent runs on Environment1 nodes, it will get the jumpbox/conf.yaml that matches the variable name geography=”Environment1″

Encrypted Data – eyaml

Encrypted data is just as easy to store.
* Generate the encrypted data.
* Store the data in an eyaml file in the same folder as the yaml data.
* Add a path to the data in the hiera.yaml file.

We have encrypted data e.g. the default local admin account setup via the profile – include profile::base
We use the Puppet Master private key to generate the encrypted data, see the end of this blog on how to create encrypted data.


profile::base::adminpassword: >

How to Encrypt Data

SSH into the Puppet Master. Locate your Puppet Master Certificates. Then run the following$ sudo /opt/puppetlabs/puppet/bin/eyaml encrypt -p --pkcs7-public-key=/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem

Enter password: ***


block: >


Puppet – Join Ubuntu 16.04 Servers to an Azure Windows Active Directory Domain

We use Azure Active Directory Domain Services and wanted a single sign on solution for Windows and Linux. The decision was made to join all servers to the Windows Domain in addition to having SSH Key auth.

I assume you know how to use Puppet Code Manager and have a Puppet Control Repository to manage Roles and Profiles.

Ensure you have a Active Directory Service Account that has permissions to join a computer to a domain. You can store this user in Hiera.

The module we need is called Realmd, however the current version (Version 2.3.0 released Sep 3rd 2018) does not support Ubuntu  16.04. So I have a forked repro here that you can use.



mod 'romiko-realmd',
  :git    => '',
  :branch => 'master'

mod 'saz-resolv_conf',                  '4.0.0'


class profile::domain::linuxdomain (
  String $domain = 'RANGERROM.COM',
  String $user = '',
  String $password = 'info',
  Array  $aadds_dns = ['','']
) {

  $hostname = $trusted['hostname']
  $domaingroup = downcase($domain)

  host { 'aaddshost':
    ensure  => present,
    name    => "${hostname}.${domain}",
    comment => 'Azure Active Directory Domain Services',
    ip      => '',

  class { 'resolv_conf':
    nameservers => $aadds_dns,
    searchpath  => [$domain],

  exec { "waagent":
    path        => ['/usr/bin', '/usr/sbin', '/bin'],
    command     => "waagent -start",
    refreshonly => true,
    subscribe   => [File_line['waagent.conf']],

  exec { "changehostname":
    path        => ['/usr/bin', '/usr/sbin', '/bin'],
    command     => "hostnamectl set-hostname ${hostname}.${domain}",
    refreshonly => true,
    subscribe   => [File_line['waagent.conf']],

  file_line { 'waagent.conf':
    ensure             => present,
    path               => '/etc/waagent.conf',
    line               => 'Provisioning.MonitorHostName=y',
    match              => '^Provisioning.MonitorHostName=n',
    append_on_no_match => false,

  class { 'ntp':
    servers => [ $domain ],

  class { '::realmd':
    domain                => $domain,
    domain_join_user      => $user,
    domain_join_password  => $password
  } ->

  file_line { 'sssd.conf':
  ensure             => present,
  path               => '/etc/sssd/sssd.conf',
  line               => '#use_fully_qualified_names = True',
  match              => '^use_fully_qualified_names',
  append_on_no_match => false,

  file_line { 'common-session':
  path => '/etc/pam.d/common-session',
  line => 'session required skel=/etc/skel/ umask=0077',

  file_line { 'sudo_rule':
  path => '/etc/sudoers',
  line => "%aad\ dc\ administrators@${domaingroup} ALL=(ALL) NOPASSWD:ALL",

The above code ensures:

  • Its fqdn is in the /etc/hosts file so it can resolve to itself.
  • The Active Directory DNS servers are in resolv.conf and the default search domain is set. This allows nslookup to <computername> to work as well as <computername>.<domainname>
  • Renames the Linux Server and sets Azure to monitor host name changes
  • Sets the Network Time Protocol to use the Active Directory Severs
  • Uses Realmd 
    • Join the domain
    • configure Sssd, samba etc
    • We using my fork until the pull request is accepted
  • Makes changes to SSSD and PAM to ensure smooth operations with Azure Active Directory Domain Services
  • Adds the administrators from AADDS to the sudoers


With the above running on your Linux agent, you will have Linux machines using the domain and can leverage single sign on.

CASE SENSITIVE –  Ensure you use Upper Case for $domain.

My source of inspiration to do this configuration came from a Microsoft Document here:

However, here is a Hiera config file to get you going for this class if you like using a key value data source.


#AADDS Domain must be Upper Case else Kerberos Tickets fail
profile::domain::linuxdomain::domain: 'RANGERROM.COM'
profile::domain::linuxdomain::user: 'serviceaccounttojoindomain@RANGERROM.COM'
        - ''
        - ''



In my next article, I will write about setting up a flexible Puppet Control Repository and leveraging Hiera.

Now I can login to the server using my Azure Active Directory credentials that I even use for Outlook, Skype and other Microsoft Products.

ssh -l myserver   (resolv.conf has a search entry for, so the suffix will get applied)
ssh -l


Azure Database for PostgreSQL – Backup/Restore

PSQL has some awesome tools pg_dump and pg_restore that can assist and cut down restore times in the event of a recovery event.


User error – If a table was accidentally emptied at noon today:

• Restore point in time just before noon and retrieve the missing table and data from that new copy of the server (Azure Portal or CLI) , Powershell not supported yet.
• Update firewall rules to allow traffic internally on the new SQL Instance

Then log onto a JumpBox in the Azure cloud.
• Dump the restored database to file (pg_dump)

pg_dump -Fc -v -t mytable -h -U pgadmin@source -d mydatabase > c:\Temp\mydatabase .dump

The switch -Fc – gives us flexibility with pg_restore later (Table Level Restore)

• Copy the table back to the original server using  pg_restore

You may need to truncate the target table before, as we doing data only restore (-a)
Table Only (not schema):

pg_restore -v -A -t mytable -h -p 5432 -U pgadmin@target -d mydatabase "c:\Temp\mydatabase .dump"

note: Use a VM in the Azure cloud that has access to SQL via VNET Rules or SQL Firewalls.

We currently use PSQL 10. There is a known issue with VNet Rules. Just contact Microsoft if you need them, else use firewall rules for now.


VNET Rules and PostgreSQL 10

Issue Definition: If using VET rules to control access to the PostgreSQL 10.0 server, customer gets the following error: psql ‘sslmode=require host=server port=5432 dbname=postgres’ –username=pgadmin@server psql: FATAL: unrecognized configuration parameter ‘connection_Vnet’ This is a known issue:



Non-authoritative answer:





Non-authoritative answer:




The Host is the same server. The way the connection string works is on the USERNAME.

These two command will connect to the SAME Server instance

pg_restore -v -a -t mytable -h -p 5432 -U pgadmin@restoretest1 -d mydatabase  "c:\Temp\mydatabase.dump"

pg_restore -v -a -t mytable -h -p 5432 -U pgadmin@restoretest1 -d mydatabase   "c:\Temp\mydatabase.dump"

The hostname just gets us to the Microsoft PSQL server, it is the username that points us to the correct instance. This is extremely important when updating connection strings on clients!

Happy Life – Happy Wife

Windows Azure – Restore Encrypted VM – BEK and KEK

When restoring an Encrypted Virtual Machine that is managed by Windows Azure, you will need to use PowerShell.

This is a two stage process.

Restore-BackupImageToDisk … | Restore-EncryptedBekVM …

Stage 1

Retrieve the Backup Container – This will contain the encrypted disks and json files with the secret keys.

  • keyEncryptionKey
  • diskEncryptionKey

This will allow you to retrieve the disks and JSON metadata in step 2.








Get-AzureRmSubscription -SubscriptionName $SubscriptionName | Select-AzureRmSubscription
$endDate = Get-Date

$namedContainer = Get-AzureRmRecoveryServicesBackupContainer  -ContainerType "AzureVM" -Status "Registered" | Where { $_.FriendlyName -eq $VMName }
$backupitem = Get-AzureRmRecoveryServicesBackupItem -Container $namedContainer  -WorkloadType "AzureVM"

$rp = Get-AzureRmRecoveryServicesBackupRecoveryPoint -Item $backupitem -StartDate $FromDate.ToUniversalTime() -EndDate $enddate.ToUniversalTime()
$restorejob = Restore-AzureRmRecoveryServicesBackupItem -RecoveryPoint $rp[0] -StorageAccountName $StorageAccount -StorageAccountResourceGroupName $ResourceGroup
Wait-AzureRmRecoveryServicesBackupJob -Job $restorejob -Timeout 43200
$restorejob = Get-AzureRmRecoveryServicesBackupJob -Job $restorejob
$details = Get-AzureRmRecoveryServicesBackupJobDetails -Job $restorejob

Stage 2

We will grab the job details via the pipeline if provided or go find the earliest Restore Job after the FromDate and then initiate a restore of all the encrypted disks to the new Virtual Machine.


    [Parameter(ValueFromPipeline=$True, Mandatory=$false, HelpMessage="Requires a AzureVmJobDetails object e.g. Get-AzureRmRecoveryServicesBackupJobDetails. Optional.")]

    [Parameter(Mandatory=$true, HelpMessage="Assumes osDisks, DataDisk, RestorePoints, AvailbilitySet, VM, Nic are all in same resource group.")]




Begin {
    $FromDate = $FromDate.ToUniversalTime()
    Write-Verbose "Started. $(Get-Date)"

Process {
	Write-Verbose "Retrieving Backup Vault."
	if(-not $JobDetails)
		Get-AzureRmRecoveryServicesVault -Name $BackupVaultName -ResourceGroupName $DefaultResourceGroup | Set-AzureRmRecoveryServicesVaultContext
		$Job = Get-AzureRmRecoveryServicesBackupJob -Status Completed -Operation Restore -From $FromDate | Sort -Property StartTime | Where { $_.WorkloadName -eq $SourceVMName} | Select -Last 1
		if($Job -eq $null) {
			throw "Job $workLoadName not found."
		$JobDetails = Get-AzureRmRecoveryServicesBackupJobDetails -Job $Job

	Write-Verbose "Query the restored disk properties for the job details."
	$properties = $
	$storageAccountName = $properties["Target Storage Account Name"]
	$containerName = $properties["Config Blob Container Name"]
	$blobName = $properties["Config Blob Name"]

	Write-Verbose "Found Restore Blob Set at $($Properties['Config Blob Uri'])"
	Write-Verbose "Set the Azure storage context and restore the JSON configuration file."

	Set-AzureRmCurrentStorageAccount -Name $storageaccountname -ResourceGroupName $DefaultResourceGroup
	$folder = New-Item "C:\RangerRom" -ItemType Directory -Force
	$destination_path = "C:\$($folder.Name)\$SourceVMName.config.json"
	Get-AzureStorageBlobContent -Container $containerName -Blob $blobName -Destination $destination_path
	Write-Verbose "Restore config saved to file. $destination_path"
	$restoreConfig = ((Get-Content -Path $destination_path -Raw -Encoding Unicode)).TrimEnd([char]0x00) | ConvertFrom-Json

	# 3. Use the JSON configuration file to create the VM configuration.
	$oldVM = Get-AzureRmVM | Where { $_.Name -eq $SourceVMName }
	$vm = New-AzureRmVMConfig -VMSize $restoreConfig.'properties.hardwareProfile'.vmSize -VMName $TargetVMName -AvailabilitySetId $oldVM.AvailabilitySetReference.Id
	$vm.Location = $oldVM.Location

	# 4. Attach the OS disk and data disks - Managed, encrypted VMs (BEK only
	$bekUrl = $restoreConfig.'properties.storageProfile'.osDisk.encryptionSettings.diskEncryptionKey.secretUrl
	$keyVaultId = $restoreConfig.'properties.storageProfile'
	$kekUrl = $restoreConfig.'properties.storageProfile'.osDisk.encryptionSettings.keyEncryptionKey.keyUrl
	$storageType = "StandardLRS"
	$osDiskName = $vm.Name + "_Restored_" + (Get-Date).ToString("yyyy-MM-dd-hh-mm-ss") + "_osdisk"
	$osVhdUri = $restoreConfig.'properties.storageProfile'.osDisk.vhd.uri
	$diskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $restoreConfig.location -CreateOption Import -SourceUri $osVhdUri
	$osDisk = New-AzureRmDisk -DiskName $osDiskName -Disk $diskConfig -ResourceGroupName $DefaultResourceGroup
	Set-AzureRmVMOSDisk -VM $vm -ManagedDiskId $osDisk.Id -DiskEncryptionKeyUrl $bekUrl -DiskEncryptionKeyVaultId $keyVaultId -KeyEncryptionKeyUrl $kekUrl -KeyEncryptionKeyVaultId $keyVaultId -CreateOption "Attach" -Windows

	$count = 0
	foreach($dd in $restoreConfig.'properties.storageProfile'.dataDisks)
		$dataDiskName = $vm.Name + "_Restored_" + (Get-Date).ToString("yyyy-MM-dd-hh-mm-ss") + "_datadisk" + $count ;
		$dataVhdUri = $dd.vhd.uri;
		$dataDiskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $restoreConfig.location -CreateOption Import -SourceUri $dataVhdUri
		$dataDisk = New-AzureRmDisk -DiskName $dataDiskName -Disk $dataDiskConfig -ResourceGroupName $DefaultResourceGroup
		Add-AzureRmVMDataDisk -VM $vm -Name $dataDiskName -ManagedDiskId $dataDisk.Id -Lun $dd.Lun -CreateOption "Attach"
		$count += 1

	Write-Verbose  "Setting the Network settings."

	$oldNicId = $oldVM.NetworkProfile.NetworkInterfaces[0].Id
	$oldNic = Get-AzureRmNetworkInterface -Name $oldNicId.Substring($oldNicId.LastIndexOf("/")+ 1) -ResourceGroupName $DefaultResourceGroup
	$subnetItems =  $oldNic.IpConfigurations[0].Subnet.Id.Split("/")
	$networkResourceGroup = $subnetItems[$subnetItems.IndexOf("resourceGroups")+1]
	$nicName= $vm.Name + "_nic_" + (New-Guid).Guid
	$nic = New-AzureRmNetworkInterface -Name $nicName -ResourceGroupName $networkResourceGroup -Location $oldNic.location -SubnetId $oldNic.IpConfigurations[0].Subnet.Id
	$vm=Add-AzureRmVMNetworkInterface -VM $vm -Id $nic.Id
	$vm.DiagnosticsProfile = $oldVM.DiagnosticsProfile

	Write-Verbose "Provisioning VM."
	New-AzureRmVM -ResourceGroupName $oldVM.ResourceGroupName -Location $oldVM.Location -VM $vm


End {
    Write-Verbose "Completed. $(Get-Date)"


.\Restore-BackupImageToDisk … | .\Restore-EncryptedBekVM.ps1 …


Ensure you have Azure VM Backups and they are backed on on a regular schedule.

Use this script to backup a Restore Point from a Recovery Vault Container.

Remember it will pick the earliest date of a Restore Point relative to your From Date. If there are 4 Restores in the recovery vault from 1 May, it will pick the first one.

Remember to:

  • Decommission the old VM
  • Update DNS with the new ip address
  • Update the Load Balance Sets (If using a Load Balancer)

Update Management solution in Azure – Workspaces


Update Management in Azure does not support system workspace. If you are using Security Center in Azure, the chances are; all your VM’s are allocated to the system workspace that Security Center created.

The selected workspace is a system workspace and cannot be linked to this account


The other error you can get is when you try enable Update Management on VM, and that VM is in another Workspace.

The selected Automation account is already linked to a Long Analytics workspace that is not the selected workspace


Deallocate All VM’s that you want to use Update Management from the System Workspace to a New Workspace.

  1.  In OMS enable Update Management and Create a New Workspace
  2. Open the Workspace associated with Update Management and note the
    1. Workspace ID
    2. Workspace Primary Key
  3.  Run the following Powershell Scripts
    1. Disconnect-AllVirtualMachinesFromWorkspace
    2. Connect-AllVirtualMachinesToWorkspace -omsID “12345…” -omsKey  “12345…==”
function Disconnect-AllVirtualMachinesFromWorkspace {
        $typeWin = "MicrosoftMonitoringAgent"
        $typeLin = "OmsAgentForLinux"

		$windows = Get-AzureRmVm  | Where { $_.StorageProfile.OsDisk.OsType -eq "Windows" }
		foreach ($vm in $windows) {
			Remove-AzureRmVMExtension -ResourceGroupName "$($vm.ResourceGroupName)" -Name $typeWin -VMName "$($vm.Name)" -Force

        $linux = Get-AzureRmVm  | Where { $_.StorageProfile.OsDisk.OsType -eq "Linux" }-Force
		foreach ($vm in $linux) {
			Remove-AzureRmVMExtension -ResourceGroupName "$($vm.ResourceGroupName)" -Name $typeLin -VMName "$($vm.Name)" -Force

function Connect-AllVirtualMachinesToWorkspace {
        [Parameter(Mandatory=$true, HelpMessage="Workspace Id")]

        [Parameter(Mandatory=$true, HelpMessage="Workspace Primary Key")]

    $typeWin = "MicrosoftMonitoringAgent"
    $typeLin = "OmsAgentForLinux"

    $PublicSettings = New-Object psobject | Add-Member -PassThru NoteProperty workspaceId $omsId | ConvertTo-Json
    $protectedSettings = New-Object psobject | Add-Member -PassThru NoteProperty workspaceKey $omsKey | ConvertTo-Json

    $windows = Get-AzureRmVm  | Where { $_.StorageProfile.OsDisk.OsType -eq "Windows" }
    foreach ($vm in $windows) {
        Set-AzureRmVMExtension -ExtensionName $typeWin -ResourceGroupName  "$($vm.ResourceGroupName)" -VMName "$($vm.Name)" -Publisher "Microsoft.EnterpriseCloud.Monitoring" -ExtensionType $typeWin -TypeHandlerVersion 1.0 -SettingString $PublicSettings  -ProtectedSettingString $protectedSettings  -Location $vm.location

    $linux = Get-AzureRmVm  | Where { $_.StorageProfile.OsDisk.OsType -eq "Linux" }-Force
    foreach ($vm in $linux) {
        Set-AzureRmVMExtension -ExtensionName $typeLin -ResourceGroupName  "$($vm.ResourceGroupName)" -VMName "$($vm.Name)" -Publisher "Microsoft.EnterpriseCloud.Monitoring" -ExtensionType $typeLin -TypeHandlerVersion 1.0 -SettingString $PublicSettings  -ProtectedSettingString $protectedSettings  -Location $vm.location

Octopus Deploy – AWS EC2 Web Servers and Elastic Load Balancer

Octopus Deploy has great add on template steps.  Using the

  • AWS – Register ELB Instance
  • AWS – Deregister ELB Instance

Octopus Steps Template

You can then leverage Environments to create Web Server Groups and thus automate deploying your web server instances.

  1. The first step is for Development and Staging, which do not use an ELB
  2. The next 4 steps is for Web Servers in Group 1
  3. The last 4 steps is for Web Servers in Group 2

A Powershell script is used to poll the web server, after the NUGET package is used to deploy the website. It will poll the Web Servers in the group until the homepage returns an HTTP 200 code for all of them. The homepage does various calls to the database, so this page is great to use. You could also create a custom up-time endpoint to query. A really cool way, is to have an API endpoint that returns which servers are up and use this to validate the step for polling.



Installing Puppet Enterprise on CentOS 7 in AWS EC2 with custom public HostName


I ran into a few issues when I wanted to install Puppet Enterprise 2-17 in AWS as an EC2 instance. The main issues were


  • Need to use hostnamectl and cloud.cfg to change my hostname, as I wanted puppet on a public address, not private address, just for a POC
  • I was using a t2.nano and t2.micro, which will not work with Puppet Enterprise 2017 (puppet-enterprise-2017.2.2-el-7-x86_64). The error you get is just Failed to run PE Installer…… So I used a t2.medium to get around the issue.
  • The usual /etc/hosts file needs some settings and DNS registration (Route53 for me)
  • Disabled SELinux (We usually use a VPN)
  • Configure security groups and have 4433 as backup port (Probably not needed)

Preliminary Install Tasks

  1. Get the latest image from CentOS 7 (x86_64) – with Updates HVM
  2. Spin up an instance with at least 4GB memory, I had a lot of installation issues with applying the catalog with low memory. T2.Medium should work. Bigger is better!
    [] Failed to run PE installer on
  3. If you not using a VPN then ensure you setup an elastic IP mapped to the instance for the public DNS name
  4. Register the hostname and elastic IP in DNS
  5. Add you hostnames to /etc/hosts (Important!), note I also added puppet as this is the default for installs. This is a crucial step, so make sure you add your hostnames that you want to use. Put the public hostname first. As this is our primary hostname127.0.0.1 puppet localhost
  6. Change the hostname of your EC2 Instance. We need to do the following

    #sudo hostnamectl set-hostname –static
    #sudo vi /etc/cloud/cloud.cfg

  7. Add the following to the end of cloud.cfg
    preserve_hostname: true
  8. This is the error I got when I first installed puppet (Due to low memory), therefore we will add port 4433 as well to the AWS security in the next step. I think this was due to insufficient memory, so use a T2.Medium instance size, so you have a minimum of 4GB of memory, else java kills itself. However I add it as a backup here in case you run some other service on 443.

    #sudo vi /var/log/puppetlabs/installer/2017-08-08T02.09.32+0000.install.log

    Failed to apply catalog: Connection refused – connect(2) for “” port 4433

  9. Create a security group with the following ports open and also do the same for the Centos Firewall.
  10. Run  netstat -anp | grep tcp to ensure no port conflicts.
  11. Disable SELinux or have it configured to work in a Puppet Master environment. Edit

    #sudo vi /etc/sysconfig/selinux


  12. Edit the sudo vi /etc/ssh/sshd_config and enable Root Logins
    PermitRootLogin yes
  13. Download Puppet Enterprise

    #curl -O
    #tar -xvf puppet-enterprise-2017.2.2-el-7-x86_64.tar.gz

  14. Install NC and use it to test if your ports are accessible.
    sudo yum install nc
    nc -nlvp 3000 (Run in one terminal) 
  15. nc puppet 3000 ( Run from another terminal)
    NC Test Firewalls.PNG
    This is a great way to ensure firewall rules are not restricting your installation. Secondly we testing that the local server can resolve itself, as it is important that you can resolve puppet and also your custom FQDN before running PE install.
  16. Reboot and run hostnamectl, the new hostname should be preserved.

    #sudo hostnamectl set-hostname –static
    [centos@ip-172-31-13-233 ~]$ hostnamectl
    Static hostname:
    Transient hostname: ip-172-31-13-233.ap-southeast-2.compute.internal
    Icon name: computer-vm
    Chassis: vm
    Machine ID: 8bd05758fdfc1903174c9fcaf82b71ca
    Boot ID: 0227f164ff23498cbd6a70fb71568745
    Virtualization: xen
    Operating System: CentOS Linux 7 (Core)
    CPE OS Name: cpe:/o:centos:centos:7
    Kernel: Linux 3.10.0-514.26.2.el7.x86_64
    Architecture: x86-64


  1. Now that we done all our preinstall checks, kick off the installer.

    #sudo ./puppet-enterprise-installer

  2. Enter 1 for a guided install.
  3. Wait until it asks you to connect to the server on https://<fqdn&gt;:3000
    This is what occurs if you did not configure your hostname correctly and you want a public hostname (EC2 internal is default):

    We want our public hostname.
    Puppet will basically run a thin web server to complete the installation with the following command:
    RACK_ENV=production /opt/puppetlabs/puppet/share/installer/vendor/bundler/bin/thin start –debug -p 3000 -a –ssl –ssl-disable-verify &> /dev/null

  4. Recall, we have the above FQDN in our host file, yours will be your hostname that you setup.
  5. Visit your Puppetmaster site at https://fqdn:3000
  6. Ensure in DNS Alias, you add puppet and all other DNS names you want to use. Otherwise the installation will fail.

    You should see the correct default hostname, if not, you got issues…. I added some alias names such as puppet and my internal and external ec2 addresses.


  7. Set an Admin password and click next
  8. Check and double check the settings to confirm.
  9. Check the validation rules, since this is for testing, I am happy with the warnings. It would be awesome if puppetlabs did DNS name resolution validation checks on the HostName. Anyways, here we get a warning about memory, 4GB is what is needed, so if you have install failures it may be due to memory!
  10. I am feeling lucky, lets try with 3533MB of RAM 🙂SuccessInstall.PNG

T-SQL UpperCase first letter of word

I am amazed by the complex solutions out on the internet to upper case the first letter of a word in SQL. Here is a way I think is nice and simple.

-- Test Data

declare @word varchar(100)
with good as (select 'good' as a union select 'nice' union select 'fine')
select @word = (SELECT TOP 1 a FROM good ORDER BY NEWID())

-- Implementation

select substring(Upper(@word),1,1) + substring(@word, 2, LEN(@word))

Request.Browser.IsMobileDevice & Tablet Devices


The problem with Request.Browser.IsMobileDevice is that it will classify a tablet as a mobile device.

If you need to discern between mobile, tablet and desktop. Then use the following extension method.

public static class HttpBrowserCapabilitiesBaseExtensions
 public static bool IsMobileNotTablet(this HttpBrowserCapabilitiesBase browser)
  var userAgent = browser.Capabilities[""].ToString();
  var r = new Regex("ipad|(android(?!.*mobile))|xoom|sch-i800|playbook|tablet|kindle|nexus|silk", RegexOptions.IgnoreCase);
  var isTablet = r.IsMatch(userAgent) && browser.IsMobileDevice;
  return !isTablet && browser.IsMobileDevice;


Then to use it is easy. Just import the namespace and reference the method.

using Web.Public.Helpers;
if (Request.Browser.IsMobileNotTablet() && !User.IsSubscribed)