Windows Azure–Diagnosing Role Start-ups

Hi Guys,

I want to walk through three issues you can have with Windows Azure Diagnostics and a Worker Role. I assume you want to access the Windows Azure Trace Logs, since you use the Trace command to write out exceptions and status messages in the onStart code.

e.g, Trace.WriteLine(“Starting my service.”)

Also you have the WAD trace listener on, which it is by default.

https://gist.github.com/1757147

Scenario – Role fails to start very early on

You might have a custom worker role that starts some sort of background service and perhaps it fails immediate due to some sort of configuration.

Symptoms

You notice the role keeps recycling and recovering.

On Demand Transfer or Schedule Transfer of Diagnostics logs do not work at all, so you cannot get any Trace Information whatsoever.

Solution

Put this method in your WorkerEntryPoint.cs file and call it at the beggining of OnStart() and in any Catch Exception block

https://gist.github.com/1757083

e.g Start

public override bool OnStart()
{
WaitForWindowsAzureDiagnosticsInfrastructureToCatchUp();
try
{

 

e.g Exception

catch (Exception ex)
           {
               TraceException(ex);
               Trace.Flush();
               WaitForWindowsAzureDiagnosticsInfrastructureToCatchUp();
               return true;
           }

 

Scenario – Role fails to start a bit later

You are able to diagnose the problem since On Demand Transfer/Scheduled Transfer works and you can then get to the trace logs to see error messages you have written to Trace. Recall that Windows Azure has a settings to automatically have a trace listener on to redirect trace to its WAD table.

 

Scenario – Role fails to start a bit later

Symptoms

You notice the role keeps recycling and recovering or event stars up but is unresponsive.

On Demand Transfer does not work – You try but it just does not complete or hangs

Below is screen shots of On Demand Transfers with the Cerebrata Diagnostics Manager.

image

image

Solution

If you cannot do an On Demand Transfer of trace logs, perhaps it keeps recycling and recovering to fast for a On Demand Transfer to occur. Then what you do is temporarily configure a Scheduled Transfer of the Trace Logs

If using Cerebrata Diagnostics Manager

Click Remote Diagnostics in the Subscriptions under your Data Sources

image

image

Once you have configured Schedules transfer, this tool will basically UPLOAD a configuration file into your BLOB container: wad-control-container

Azure will automatically detect changes in this container and apply them to the Diagnostics Manager. Hence configuration of Windows Azure Diagnostics On The Fly

image

Now that we have scheduled transfer in place REBOOT the role that is causing the issue and then wait for it to try start up and fail, and then just go download the trace logs and it should be there.

image

Summary

So, ensure you have a silly sleep command in your work entry point OnStart and in areas where you catch exceptions in case your worker role crashes before Windows Azure Diagnostics!

Try On Demand Transfers if there is an issue, and if that does not work,  configure a scheduled transfer on the fly and then reboot the role to get the start up logs.

WARNING!

Scheduled Transfers will impact your billing of Storage Services, MAKE SURE you turn it OFF when you finished diagnosing the issue, else you will get BILLED for it.

Notice in my screen shot I ALWAYS use a quota so I never over use diagnostics storage – and Windows Azure Trace Logs are stored in TABLE Storage:

image

Remember configuration of WAD is in Blob and the actual trace logs are in Tables.

image

Advertisement

One thought on “Windows Azure–Diagnosing Role Start-ups

  1. Hi Romiko!

    I’m a community curator at DZone.com and I really was impressed by your blog and the interesting Azure topics you write about. Over at our site, we have a Cloud Zone, http://cloud.dzone.com/

    I was going to ask you if you would be interested in giving me permission to republish one of your recent posts to benefit our audience and give some exposure to your content.

    If you’re interested, I’d also like to let you know about our MVB (most valuable blogger) program.

    Thanks!

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s