Automate #WindowsAzure snapshot restores

Hi,
This is the last series in blog posts regarding the automation of backups, purging and restoring azure blobs.

Below is a PowerShell script that can take a file containing the contents of snapshot urls, it also supports the log file output from the backup restore script and just pasting that output in the event you want to restore a complete backup set.

Remember, when using the backup script, ALWAYS save the output of the script to use as a reference so that you have the URL’s of the snapshots you want to restore.

e.g. Sample restore.txt file.
[05:01:08]: [Publishing internal artifacts] Sending build.start.properties.gz file
[05:01:05]: Step 1/2: Command Line (14s)
[05:01:05]: [Step 1/2] in directory: C:\TeamCity\buildAgent\work\d9375448b88c1b75\Maintenance
[05:01:08]: [Step 1/2] Starting snapshot uniqueids
[05:01:08]: [Step 1/2] Found blob container uniqueids
[05:01:09]: [Step 1/2] https://uatmystory.blob.core.windows.net/uniqueids/agencies?snapshot=2012-04-22
[05:01:09]: [Step 1/2] T19:01:10.6488549Z
[05:01:09]: [Step 1/2] https://uatmystory.blob.core.windows.net/uniqueids/agency1-centres?snapshot=201
[05:01:09]: [Step 1/2] 2-04-22T19:01:10.8818083Z
[05:01:09]: [Step 1/2] https://uatmystory.blob.core.windows.net/uniqueids/agency1-clients?snapshot=201
[05:01:09]: [Step 1/2] 2-04-22T19:01:11.0257795Z
[05:01:09]: [Step 1/2] https://uatmystory.blob.core.windows.net/uniqueids/agency1-referrals?snapshot=2
[05:01:09]: [Step 1/2] 012-04-22T19:01:11.1717503Z

So the script will parse any restore file and just find URI’s in it, and then restore them.

#requires -version 2.0
param (
	[parameter(Mandatory=$true)] [string]$AzureAccountName,
	[parameter(Mandatory=$true)] [string]$AzureAccountKey,
	[parameter(Mandatory=$true)] [string]$FileContainingSnapshotAddresses
)

$ErrorActionPreference = "Stop"

if ((Get-PSSnapin -Registered -Name AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue) -eq $null)
{
	throw "AzureManagementCmdletsSnapIn missing. Install them from Https://www.cerebrata.com/Products/AzureManagementCmdlets/Download.aspx"
}

Add-PSSnapin AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue
Add-Type -Path 'C:\Program Files\Windows Azure SDK\v1.6\ref\Microsoft.WindowsAzure.StorageClient.dll'

$cred = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey($AzureAccountName,$AzureAccountKey)
$client = New-Object Microsoft.WindowsAzure.StorageClient
.CloudBlobClient("https://$AzureAccountName.blob.core.windows.net",$cred)

function RestoreSnapshot
{
	param ( $snapShotUri)
	Write-Host "Parsing snapshot restore for $SnapShotUri"

	$regex = new-object System.Text.RegularExpressions.Regex("http://.*?/(devstoreaccount1/)?(?<containerName>.*?)/.*")
	$match = $regex.Match($snapShotUri)
	$container = $match.Groups["containerName"].Value
	$parsedUri = $match
	
	if($match.Value -eq "")
	{
		return
	}
		
	if ($container -eq $null)
	{
		Write-Host  "Container $blobContainerName doesn't exist, skipping snapshot restore"
	}
	else
	{
		Write-Host  "Restoring $snapShotUri" 
		Copy-Blob -BlobUrl $parsedUri -AccountName $AzureAccountName -AccountKey $AzureAccountKey -TargetBlobContainerName $container
		Write-Host  "Restore snapshot complete for $parsedUri"
	}
}

$fileContent = Get-Content $FileContainingSnapshotAddresses

foreach($uri in $fileContent)
{
	RestoreSnapshot $uri
}


Cloning Disks and Partitions

I have been using CloneZilla to manage all my disk and partition backups, I find it very user friendly and support all my disks (USB, ESATA). I recommend TUXBOOT for making Clonezilla bootable USB drive and to keep it with you whenever you need to clone a disk, then just boot off the stick.

http://clonezilla.org/liveusb.php

Windows 7 bootable USB stick

On the subject, sometimes you need to get an OS on beforehand for Windows, I like using this tool to make a bootable windows 7 USB stick.

http://images2.store.microsoft.com/prod/clustera/framework/w7udt/1.0/en-us/Windows7-USB-DVD-tool.exe

Is you private data safe? Email, Passwords, Internet Banking etc

Hi,

I would like to discuss the concept that we all think our data is safe. If we browse our mail on Google. We see it is HTTPS, and see the nice little padlock and then we feel or fluffy that nobody will ever be able to read our mails as it travels through fibre optics switches across the oceans and many routers and switches.

WARNING: The view expressed here are based on my knowledge acquired from Wired, therefore I cannot guarantee the information on this blog is 100% correct, discretion is advised.

But you wrong?

There are organizations out, specifically in the US that STORE all data traffic coming into various telecommunication hubs such as AT&T.

You might say, so what?

Consider this:

Store Data Now – Crack the encryption later

The NSA (National Security Agency) in the US is storing your encrypted data if it goes via the US. Considering the you do not have total control of how network packets are sent over the Internet, chances are 90% of your data is going via the US data hubs.

You might say that, who cares if they storing your packet level data for encrypted emails. But lets see how they can decrypt it:

Imagine GMAIL:

image

image

Look at the size of the encryption that Google mail uses.

1024 Bits.

Ok, so this is a relatively strong encryption of 1024 Bits, which would take the age of several universes to crack on a normal home computer, but on a super computer, things are much more merrier.

Also, if you can get enough DATA using the same Public Key, algorithms on Super Computers can then be used to detect patterns in emails being sent e.g. Hello, Kind Regards etc. With these patterns, the Super Computer and increase its chances in cracking the private key.

Let’s take an example of the RSA 1024 Bit encryption (yes same that GMAIL uses) that was cracked on 7th January 2010. The ONE requirement for the pattern matching to work is that the email messages or data sent is using the SAME public/private key pair.

Source: RSA 1024 Cracked

The RSA crack this time, was achieved by Valeria Bertacco, Todd Austin and Andrea Pellegrini. They varied the voltage levels at the sender end to make faulty encryptions. This helped them recreate the private key by combining a number of fragments, achieved in the process. The complete operation took 100 hours. A quote from the research paper says, First, we develop a systematic fault-based attack on the modular exponentiation algorithm for RSA. Second, we expose and exploit a severe flaw on the implementation of the RSA signature algorithm on OpenSSL, a widely used package for SSL encryption and authentication. Third, we report on the first physical demonstration of a fault-based security attack of a complete microprocessor system running unmodified production software: we attack the original OpenSSL authentication library running on a SPARC Linux system implemented on FPGA, and extract the system’s 1024-bit RSA private key in approximately 100 hours.

 

This breaks the backbone of RSA which believes that as long as the private key is safe, it is impossible to break in, unless guessed.

Ok, the above is not how the NSA cracks work, but servers as an example. If you can find a PATTERN, then you improve the probabilities of cracking, and this is done by the assumption that your private/public key does not change often.

Data Mining

Ok, so the NSA has your historical emails, online banking etc. So, how do they crack it, first they would collect hundreds of emails sent to a particular recipient and run algorithms for common patterns, then this can get fed into a super computer.

Now, what super computer. Well, they have a few, the latest one is spanned across two buildings, consuming power to light up 200, 000 homes! It is near their current facility on the East Campus of Oak Ridge.

Another super computer ready by the end of 2013 is called CASCADE, developed in tangent with Cray, this new super beast can run at speeds of 20 petaflops!

Government Legislation

What should change, because, ultimately we are supposed to live a democratic society? Laws should be passed to prevent any organisation from storing data and especially prohibit the storage of encrypted data.

What can we do

Very simple. To make the algorithms weak that they use to detect patterns in several messages

1. Prevent the algorithm from using patterns to improve the chance of cracking the encryption key by CHANGING your public/private key pairs often.

2. For very secure data use an email provider that is NOT hosted in the US. Check this with visual  TraceRoute/Tracert tools.

3. Zip the contents of secure data and encrypt that over the HTTPS encryption in your email system

4. Create your own Self Hosted Email System that is NOT HOSTED in the US.

5. Use your OWN certificate server and share the Root CA with the people you need to send secure data to. E.g. Financial Intuitions (Letter of Credits etc.). Then build a PowerShell script to generate new Private/public keys randomly and email this to your SECURE member network. VERY important that they generated at random times throughout the day, since if you pick a specific date/time, then the crackers can use this to ESTIMATE when they change and only using data in that window to run pattern matching. Then use a syncing automate system that sends out the new public key to the recipients that automatically use a script or bot to load it into their KEY STORE.

6. DO NOT USE LAND LINES or Mobile Phone contract SIM cards for personnel/intimate calls via the US, they all get filtered and analysed. Use a Pay as You Go. It is not fool proof but helps. Allot of journalists personnel lives are known to the NSA and many other people that travel, your phone is not safe, remember that, it goes through a network exchange, it is easy to pick voice calls and store it!

By doing the above and changing Private Key often in an automated fashion, you will prohibit the best algorithms from trying to crack your data. The NSA will have no way of knowing what the schedule is of your Public/Private key regeneration is, thus the algorithm will make assumptions that the next hours worth of data is on the same key pair etc., and will just be running down a dead end.

Microsoft’s Cloud

One thing I noted with Microsoft Windows Azure’s Cloud, is even though you can set an affinity for a region out of the US, e.g. Singapore, the data going to and from your cloud hosted site in Singapore is going via the US! Here is the proof.

Consider this trace I did from Sydney to Singapore, it went VIA the US!

image

The General Public and End User

Not all users of the internet are geeks like us, so how can we help them. We build an Open Source Application that manages dynamic Public/Private Key pairs and a syncing notification system with a secure public key delivery mechanism, which in itself is extremely complicated, perhaps when I have some time, we can come up with a demo.

Summary

I hope I have revealed to you that your data is not safe and secure, there are Organisations out there that break all the laws and are allowed to and one of them is the NSA, your data in the next few years will be stored in a HUGE data centre in the UTAH Dessert.  As we speak the data centre being built in the UTAH dessert is going along smoothly..

PROTECT your most intermit data with the tips I have given here. Just because 0.01% of the population is bad does not mean the rest of use must be compromised and our privacy VIOLATED. It is an absolute disgrace where our society is heading in regards to privacy laws. Nobody has the right to listen in on your private phone calls, emails etc. And NOBODY has the right to store all your historic data, waiting until technology is powerful enough to decrypt it.

WARNING: The view expressed here are based on my knowledge acquired from Wired, therefore I cannot guarantee the information on this blog is 100% correct, discretion is advised.

NOTHING IS SECURE

Automate #Azure Blob Snapshot purging/deletes with @Cerebrata

The pricing model for snapshots can get rather complicated, so we need a way to automate the purging of snapshots.
Read how snapshots can accrue additional costs

Lets minimize these costs! We use this script to backup and manage snapshot retention for all our Neo4j Databases hosted in the Azure Cloud.

So a solution I have is that:
We have a retention period in days for all snapshots e.g. 30 days
We have a retention period for the last day of the month backups e.g. 180 days

Rules:
1. The purging will always ensure that there is always at least ONE snapshot, so it will never delete a backup if it is the only backup for a base blob.

2. The purging will delete snapshots greater than the retention period, respecting rule 1

3. The purging will delete snapshots greater than the last day month retention period, respecting rule 1

You can then schedule this script to run after the Backup Script in TeamCity or some other build server scheduler.

param(
	[parameter(Mandatory=$true)] [string]$AzureAccountName,
	[parameter(Mandatory=$true)] [string]$AzureAccountKey,
	[parameter(Mandatory=$true)] [array]$BlobContainers, #Blob Containers to backup
	[parameter(Mandatory=$true)] [int]$BackupRetentionDays, #Days to keep snapshot backups
	[parameter(Mandatory=$true)] [int]$BackupLastDayOfMonthRetentionDays # Days to keep last day of month backups
)


if( $BackupRetentionDays -ge $BackupLastDayOfMonthRetentionDays )
{
	$message = "Argument Exception: BackupRentionDays cannot be greater than or equal to BackupLastDayOfMonthRetentionDays"
	throw $message
}

Add-Type -Path 'C:\Program Files\Windows Azure SDK\v1.6\ref\Microsoft.WindowsAzure.StorageClient.dll'

$cred = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey($AzureAccountName,$AzureAccountKey)
$client = New-Object Microsoft.WindowsAzure.StorageClient
.CloudBlobClient("https://$AzureAccountName.blob.core.windows.net",$cred)

function PurgeSnapshots ($blobContainer)
{
	$container = $client.GetContainerReference($blobContainer)
	$options = New-Object  Microsoft.WindowsAzure.StorageClient.BlobRequestOptions
	$options.UseFlatBlobListing = $true
	$options.BlobListingDetails = [Microsoft.WindowsAzure.StorageClient.BlobListingDetails]::Snapshots

	$blobs = $container.ListBlobs($options)
	$baseBlobWithMoreThanOneSnapshot = $container.ListBlobs($options)| Group-Object Name | Where-Object {$_.Count -gt 1} | Select Name

	#Filter out blobs with more than one snapshot and only get SnapShots.
	$blobs = $blobs | Where-Object {$baseBlobWithMoreThanOneSnapshot  -match $_.Name -and $_.SnapshotTime -ne $null} | Sort-Object SnapshotTime -Descending

	foreach ($baseBlob in $baseBlobWithMoreThanOneSnapshot )
	{
		 $count = 0
		 foreach ( $blob in $blobs | Where-Object {$_.Name -eq $baseBlob.Name } )
		    {
				$count +=1
				$ageOfSnapshot = [System.DateTime]::UtcNow - $blob.SnapshotTime
				$blobAddress = $blob.Uri.AbsoluteUri + "?snapshot=" + $blob.SnapshotTime.ToString("yyyy-MM-ddTHH:mm:ss.fffffffZ")

				#Fail safe double check to ensure we only deleting a snapshot.
				if($blob.SnapShotTime -ne $null)
				{
					#Never delete the latest snapshot, so we always have at least one backup irrespective of retention period.
					if($ageOfSnapshot.Days -gt $BackupRetentionDays -and $count -eq 1)
					{
						Write-Host "Skipped Purging Latest Snapshot"  $blobAddress
						continue
					}

					if($ageOfSnapshot.Days -gt $BackupRetentionDays -and $count -gt 1 )
					{
					    #Do not backup last day of month backups
						if($blob.SnapshotTime.Month -eq $blob.SnapshotTime.AddDays(1).Month)
						{
							Write-Host "Purging Snapshot "  $blobAddress
							$blob.Delete()
							continue
						}
						#Purge last day of month backups based on monthly retention.
						elseif($blob.SnapshotTime.Month -ne $blob.SnapshotTime.AddDays(1).Month)
						{
							if($ageOfSnapshot.Days -gt $BackupLastDayOfMonthRetentionDays)
							{
							Write-Host "Purging Last Day of Month Snapshot "  $blobAddress
							$blob.Delete()
							continue
							}
						}
						else
						{
							Write-Host "Skipped Purging Last Day Of Month Snapshot"  $blobAddress
							continue
						}
					}
					
					if($count % 5 -eq 0)
					{
						Write-Host "Processing..."
					}
				}
				else
				{
					Write-Host "Skipped Purging "  $blobAddress
				}
		    }
	}
}

foreach ($container in $BlobContainers)
{
	Write-Host "Purging snapshots in " $container
	PurgeSnapshots $container
}

Automate #Azure Blob Snapshot backups with @Cerebrata

Hi,

Leveraging the cerebrata cmdlets for Azure, we can easily backup our blob containers via snapshot, this will prove useful for Page Blobs that are Random Access i.e. VHD’s on Cloud Drive

Here is how Purging Snapshots works

#requires -version 2.0
param (
	[parameter(Mandatory=$true)] [string]$AzureAccountName,
	[parameter(Mandatory=$true)] [string]$AzureAccountKey,
	[parameter(Mandatory=$true)] [array]$BlobContainers
)

$ErrorActionPreference = "Stop"

if ((Get-PSSnapin -Registered -Name AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue) -eq $null)
{
	throw "AzureManagementCmdletsSnapIn missing. Install them from Https://www.cerebrata.com/Products/AzureManagementCmdlets/Download.aspx"
}

Add-PSSnapin AzureManagementCmdletsSnapIn -ErrorAction SilentlyContinue

function SnapShotBlobContainer 
{
	param ( $containers, $blobContainerName )
	Write-Host "Starting snapshot $blobContainerName"

	$container = $containers | Where-Object { $_.BlobContainerName -eq $blobContainerName }

	if ($container -eq $null)
	{
		Write-Host  "Container $blobContainerName doesn't exist, skipping snapshot"
	}
	else
	{
        Write-Host  "Found blob container $blobContainerName"
Checkpoint-BlobContainer -Name $container.BlobContainerName -SaveSnapshotInformation -AccountName $AzureAccountName -AccountKey $AzureAccountKey
	Write-Host  "Snapshot complete for $blobContainerName"
	}
}

$containers = Get-BlobContainer -AccountName $AzureAccountName -AccountKey $AzureAccountKey
foreach($container in $BlobContainers)
{
	SnapShotBlobContainer $containers $container
}

Then just call the script with the params. remember an array of items is parsed in like this:

-BlobContainers:@(‘container1′, ‘contaner2′) -AzureAccountName romikoTown -AzureAccountKey blahblahblahblahblehblooblowblab==

Neo4jClient Cypher ResultSet Support

Sometimes when doing Cypher queries the result is only one column and not multiple columns, therefore it makes sense to have a method in the fluent API to let this be known, so we do not have to map the column to an object type.

So fluent support to deserialize common result sets where cypher returns a result with only 1 column with the help of Tatham Oddie is completed.

So in the Neo4jClient you can do this, when the result from Cypher is one column via REST:

var result = agencySource
                        .StartCypher("a1")
                        .AddStartPoint("a2", agency.Reference)
                        .Match("p = allShortestPaths( a1-[*..20]-a2 )")
                        .Return<PathsResult>("p")
                        .ResultSet;

So, if you need cypher results with only one column then use .ResultSet instead of .Results, thus no need for expression tree column matches to assist the deserializer with multiple column names.

Here is a sample rest response with 1 column result that is suited perfectly for ResultSet.

{
  "data" : [ [ {
    "start" : "http://localhost:20001/db/data/node/215",
    "nodes" : [ "http://localhost:20001/db/data/node/215", "http://localhost:20001/db/data/node/0", "http://localhost:20001/db/data/node/219" ],
    "length" : 2,
    "relationships" : [ "http://localhost:20001/db/data/relationship/247", "http://localhost:20001/db/data/relationship/257" ],
    "end" : "http://localhost:20001/db/data/node/219"
  } ], [ {
    "start" : "http://localhost:20001/db/data/node/215",
    "nodes" : [ "http://localhost:20001/db/data/node/215", "http://localhost:20001/db/data/node/1", "http://localhost:20001/db/data/node/219" ],
    "length" : 2,
    "relationships" : [ "http://localhost:20001/db/data/relationship/248", "http://localhost:20001/db/data/relationship/258" ],
    "end" : "http://localhost:20001/db/data/node/219"
  } ] ],
  "columns" : [ "p" ]
}

If you wondering what the hell is agencySource, it is just node references, that I got using gremlin, which can spin off cypher queries, cool is it not?

var agencies = graphClient
                .RootNode
                .Out<Agency>(Hosts.TypeKey)
                .ToList();

This just enumerate through the list of nodes to run your cypher queries off the node directly! Have these imports declarations:

using Neo4jClient.ApiModels.Cypher;
using Neo4jClient.Gremlin;
using Neo4jClient.Cypher;

Summary

Use .ResultSet for single column result sets and use .Results when dealing with multiple column results.

Follow

Get every new post delivered to your Inbox.

Join 131 other followers