Tag: Backups

Automate #Azure Blob Snapshot purging/deletes with @Cerebrata

The pricing model for snapshots can get rather complicated, so we need a way to automate the purging of snapshots.
Read how snapshots can accrue additional costs

Lets minimize these costs! We use this script to backup and manage snapshot retention for all our Neo4j Databases hosted in the Azure Cloud.

So a solution I have is that:
We have a retention period in days for all snapshots e.g. 30 days
We have a retention period for the last day of the month backups e.g. 180 days

Rules:
1. The purging will always ensure that there is always at least ONE snapshot, so it will never delete a backup if it is the only backup for a base blob.

2. The purging will delete snapshots greater than the retention period, respecting rule 1

3. The purging will delete snapshots greater than the last day month retention period, respecting rule 1

You can then schedule this script to run after the Backup Script in TeamCity or some other build server scheduler.

param(
	[parameter(Mandatory=$true)] [string]$AzureAccountName,
	[parameter(Mandatory=$true)] [string]$AzureAccountKey,
	[parameter(Mandatory=$true)] [array]$BlobContainers, #Blob Containers to backup
	[parameter(Mandatory=$true)] [int]$BackupRetentionDays, #Days to keep snapshot backups
	[parameter(Mandatory=$true)] [int]$BackupLastDayOfMonthRetentionDays # Days to keep last day of month backups
)


if( $BackupRetentionDays -ge $BackupLastDayOfMonthRetentionDays )
{
	$message = "Argument Exception: BackupRentionDays cannot be greater than or equal to BackupLastDayOfMonthRetentionDays"
	throw $message
}

Add-Type -Path 'C:\Program Files\Windows Azure SDK\v1.6\ref\Microsoft.WindowsAzure.StorageClient.dll'

$cred = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey($AzureAccountName,$AzureAccountKey)
$client = New-Object Microsoft.WindowsAzure.StorageClient
.CloudBlobClient("https://$AzureAccountName.blob.core.windows.net",$cred)

function PurgeSnapshots ($blobContainer)
{
	$container = $client.GetContainerReference($blobContainer)
	$options = New-Object  Microsoft.WindowsAzure.StorageClient.BlobRequestOptions
	$options.UseFlatBlobListing = $true
	$options.BlobListingDetails = [Microsoft.WindowsAzure.StorageClient.BlobListingDetails]::Snapshots

	$blobs = $container.ListBlobs($options)
	$baseBlobWithMoreThanOneSnapshot = $container.ListBlobs($options)| Group-Object Name | Where-Object {$_.Count -gt 1} | Select Name

	#Filter out blobs with more than one snapshot and only get SnapShots.
	$blobs = $blobs | Where-Object {$baseBlobWithMoreThanOneSnapshot  -match $_.Name -and $_.SnapshotTime -ne $null} | Sort-Object SnapshotTime -Descending

	foreach ($baseBlob in $baseBlobWithMoreThanOneSnapshot )
	{
		 $count = 0
		 foreach ( $blob in $blobs | Where-Object {$_.Name -eq $baseBlob.Name } )
		    {
				$count +=1
				$ageOfSnapshot = [System.DateTime]::UtcNow - $blob.SnapshotTime
				$blobAddress = $blob.Uri.AbsoluteUri + "?snapshot=" + $blob.SnapshotTime.ToString("yyyy-MM-ddTHH:mm:ss.fffffffZ")

				#Fail safe double check to ensure we only deleting a snapshot.
				if($blob.SnapShotTime -ne $null)
				{
					#Never delete the latest snapshot, so we always have at least one backup irrespective of retention period.
					if($ageOfSnapshot.Days -gt $BackupRetentionDays -and $count -eq 1)
					{
						Write-Host "Skipped Purging Latest Snapshot"  $blobAddress
						continue
					}

					if($ageOfSnapshot.Days -gt $BackupRetentionDays -and $count -gt 1 )
					{
					    #Do not backup last day of month backups
						if($blob.SnapshotTime.Month -eq $blob.SnapshotTime.AddDays(1).Month)
						{
							Write-Host "Purging Snapshot "  $blobAddress
							$blob.Delete()
							continue
						}
						#Purge last day of month backups based on monthly retention.
						elseif($blob.SnapshotTime.Month -ne $blob.SnapshotTime.AddDays(1).Month)
						{
							if($ageOfSnapshot.Days -gt $BackupLastDayOfMonthRetentionDays)
							{
							Write-Host "Purging Last Day of Month Snapshot "  $blobAddress
							$blob.Delete()
							continue
							}
						}
						else
						{
							Write-Host "Skipped Purging Last Day Of Month Snapshot"  $blobAddress
							continue
						}
					}
					
					if($count % 5 -eq 0)
					{
						Write-Host "Processing..."
					}
				}
				else
				{
					Write-Host "Skipped Purging "  $blobAddress
				}
		    }
	}
}

foreach ($container in $BlobContainers)
{
	Write-Host "Purging snapshots in " $container
	PurgeSnapshots $container
}

Advertisement

Automate #Azure Blob Snapshot backups with @Cerebrata

Hi,

Leveraging the cerebrata cmdlets for Azure, we can easily backup our blob containers via snapshot, this will prove useful for Page Blobs that are Random Access i.e. VHD’s on Cloud Drive

Here is how Purging Snapshots works

#requires -version 2.0
param (
	[parameter(Mandatory=$true)] [string]$AzureAccountName,
	[parameter(Mandatory=$true)] [string]$AzureAccountKey,
	[parameter(Mandatory=$true)] [array]$BlobContainers
)

$ErrorActionPreference = "Stop"

if ((Get-PSSnapin -Registered -Name AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue) -eq $null)
{
	throw "AzureManagementCmdletsSnapIn missing. Install them from Https://www.cerebrata.com/Products/AzureManagementCmdlets/Download.aspx"
}

Add-PSSnapin AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue

function SnapShotBlobContainer 
{
	param ( $containers, $blobContainerName )
	Write-Host "Starting snapshot $blobContainerName"

	$container = $containers | Where-Object { $_.BlobContainerName -eq $blobContainerName }

	if ($container -eq $null)
	{
		Write-Host  "Container $blobContainerName doesn't exist, skipping snapshot"
	}
	else
	{
        Write-Host  "Found blob container $blobContainerName"
Checkpoint-BlobContainer -Name $container.BlobContainerName -SaveSnapshotInformation -AccountName $AzureAccountName -AccountKey $AzureAccountKey
	Write-Host  "Snapshot complete for $blobContainerName"
	}
}

$containers = Get-BlobContainer -AccountName $AzureAccountName -AccountKey $AzureAccountKey
foreach($container in $BlobContainers)
{
	SnapShotBlobContainer $containers $container
}

Then just call the script with the params. remember an array of items is parsed in like this:

-BlobContainers:@(‘container1’, ‘contaner2’) -AzureAccountName romikoTown -AzureAccountKey blahblahblahblahblehblooblowblab==