Memory Dump Analysis–W3WP IIS Process

At a customer I had prepared some Visual Studio 2010 WebTests which were calling their Java based website hosted on IBM Websphere, there is an IFrame on the Java page that points to an IIS hosted Asp.net 2.0 web site.

When running load tests I noticed that the webtests were taking a very long time to execute at one point the load test grinded the servers to a halt and a colleague of mine Tatham Oddie decided it was time we dump the memory from the IIS Process hosting the site.

image

So, right click the w3wp.exe process and click Create Dump File.

We also have windbg installed, which you can download from

http://www.windbg.org/

Ensure you click debugging tools for windows under redistributable, so you get 32/64 bit versions

image

So, the big picture is for us to look at what is on the stack, ideally in software we should have items living on the stack for a very short period of time, so lets analyze the stack and see what clues it might have about the web site behaving badly.

Open Windbg, it is located in

C:\Program Files\Debugging Tools for Windows (x64) or you might have the 32 bit version.

I also copied the following files

w3wp.dmp (the generated dump

mscordacwks.dll (From Server hosting the IIS service)

mscordacwks_64.dll (From Server hosting the IIS service)

image

Start windbg and open the dump file.

image

excellent, we are presented with

.image

SOS will need the Data Access Layer to make calls into the CLR, so we need the correct version of the mscordacwks.dll file from the Server where the software was running. This can be either 32 bit or 64 bit.

They are located at:

C:\Windows\Microsoft.NET\Framework\v2.0.50727

C:\Windows\Microsoft.NET\Framework64\v2.0.50727

Now, we need to tell the debugger where this file is, I copied these files and the dmp file from the server and put it into my E:\projects\debug folder

first, lets get the correct mscordacwks loaded, I renamed my 64bit version to this name:

mscordacwks_AMD64_AMD64_2.0.50727.5420.dll you can find the file version in the property page for mscordacwks_64

I then point my debugger to the renamed version of mscordackwks.

.sympath+ e:\projects\debug

then I load the DAC

.cordll –ve –u –l

I can now load the sos commands

.loadby sos mscorwks

if you have issues load the mscordacwks, you can also try

!sym noisy
.symfix e:\projects\debug
.cordll -ve -u -l

this will then try load the symbols from the microsoft server (http://msdl.microsoft.com/download/symbols), you must have an internet connection.

Lets see what threads are in the process:

0:000> !threads
ThreadCount: 37
UnstartedThread: 0
BackgroundThread: 10
PendingThread: 0
DeadThread: 27
Hosted Runtime: no
                                              PreEmptive                                                Lock
       ID OSID        ThreadOBJ     State   GC     GC Alloc Context                  Domain           Count APT Exception
   4    1 1378 00000000017fb730      8220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
  12    2  970 00000000017dd1d0      b220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Finalizer)
  13    4 107c 000000001a9c8cf0    80a220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Threadpool Completion Port)
  14    5 12d0 000000001a9d5fc0      1220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
XXXX    6    0 000000001aa36200   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Threadpool Worker)
XXXX    8    0 000000001aaac0b0      9820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
XXXX    9    0 000000001aaab850   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX    a    0 000000001ca2b840      9820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
XXXX    b    0 000000001caa3830   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX    c    0 000000001ab937c0   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX    d    0 000000001cab4680   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX    e    0 000000001ca52940   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX    f    0 000000001caaf450   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   10    0 000000001ca57290   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   11    0 000000001cab5c60   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Threadpool Worker)
XXXX   12    0 000000001aa52ca0   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   13    0 000000001aa53270   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
   2   14  f48 000000001aa53840       220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
XXXX   15    0 000000001aa543e0   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
  17   16  834 000000001aa549b0   180b220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Threadpool Worker)
XXXX   17    0 000000001aa54f80      9820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
XXXX   18    0 000000001aa55550   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Threadpool Worker)
XXXX   19    0 000000001aa55b20   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   1a    0 000000001aa560f0   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   1b    0 000000001aa566c0   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   1c    0 000000001aa56c90   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   1d    0 000000001aa57260   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   1e    0 000000001aa57830   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   1f    0 000000001aa57e00   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   20    0 000000001aa583d0   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
  18   21 121c 000000001aa589a0   380b220 Enabled  0000000000000000:0000000000000000 000000001ab979c0     1 MTA (Threadpool Worker)
XXXX   22    0 000000001aa58f70   1801820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn (Threadpool Worker)
XXXX   24    0 000000001aa59b10      9820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
XXXX   26    0 000000001ef10680      9820 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 Ukn
  19   25 11bc 000000001ef100b0   200b220 Enabled  0000000000000000:0000000000000000 000000001ab979c0     1 MTA
  21    3 13c8 000000001aa526d0   880b220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Threadpool Completion Port)
  22    7 12f4 000000001aa53e10   880b220 Enabled  0000000000000000:0000000000000000 00000000017ea4f0     0 MTA (Threadpool Completion Port)

Ok, from this we will need to try find the thread that is running all the IIS web pages etc. SO lets execute

0:000> !eestack
———————————————
Thread   4
Child-SP         RetAddr          Call Site
0000000001a2f7a8 000007fefd47169d ntdll!NtRemoveIoCompletion+0xa
0000000001a2f7b0 000000007731a4e1 KERNELBASE!GetQueuedCompletionStatus+0x3d
0000000001a2f810 000007fef9e81f7b kernel32!GetQueuedCompletionStatusStub+0x11
0000000001a2f850 000007fef9e82024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x3b
0000000001a2f8a0 000007fef9e820a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
0000000001a2f8d0 000000007732652d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
0000000001a2f900 000000007745c521 kernel32!BaseThreadInitThunk+0xd
0000000001a2f930 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  12
Child-SP         RetAddr          Call Site
000000001a9af5e8 000007fefd471420 ntdll!ZwWaitForMultipleObjects+0xa
000000001a9af5f0 0000000077321220 KERNELBASE!GetCurrentProcess+0x40
000000001a9af6f0 000007fef82a0353 kernel32!WaitForMultipleObjects+0xb0
000000001a9af780 000007fef8297b9a mscorwks!WKS::WaitForFinalizerEvent+0x93
000000001a9af7b0 000007fef8295a3c mscorwks!WKS::GCHeap::FinalizerThreadWorker+0x4a
000000001a9af7f0 000007fef827ba45 mscorwks!CClosedHashEx<CCustAttrHashKey,CCustAttrHash>::Hash+0x30
000000001a9af840 000007fef81735b5 mscorwks!SVR::gc_heap::make_heap_segment+0x155
000000001a9af910 000007fef829837a mscorwks!EEToProfilerExceptionInterfaceWrapper::ExceptionCatcherEnter+0x4d
000000001a9af950 000007fef82979b4 mscorwks!ManagedThreadBase_NoADTransition+0x42
000000001a9af9b0 000007fef8298164 mscorwks!WKS::GCHeap::FinalizerThreadStart+0x74
000000001a9af9f0 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001a9afac0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001a9afaf0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  13
Child-SP         RetAddr          Call Site
000000001a8afd28 000007fefd471203 ntdll!NtDelayExecution+0xa
000000001a8afd30 000007fef814812d KERNELBASE!SleepEx+0xb3
000000001a8afdd0 000007fef82a1d67 mscorwks!ThreadpoolMgr::TimerThreadFire+0x51
000000001a8afe70 000000007732652d mscorwks!ThreadpoolMgr::TimerThreadStart+0x6b
000000001a8afeb0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001a8afee0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  14
Child-SP         RetAddr          Call Site
000000001adcf368 000007fefd4710ac ntdll!NtWaitForSingleObject+0xa
000000001adcf370 000007fef81cb540 KERNELBASE!WaitForSingleObjectEx+0x9c
000000001adcf410 000007fef81cb42f mscorwks!CLREvent::WaitEx+0x174
000000001adcf460 000007fef82b1a9f mscorwks!CLREvent::WaitEx+0x63
000000001adcf510 000007fef8298164 mscorwks!AppDomain::ADUnloadThreadStart+0x153
000000001adcf630 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001adcf780 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001adcf7b0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread   2
Child-SP         RetAddr          Call Site
0000000000b3f578 000000007744fe0b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
0000000000b3f580 000000007732652d ntdll!RtlValidateHeap+0x3bb
0000000000b3f880 000000007745c521 kernel32!BaseThreadInitThunk+0xd
0000000000b3f8b0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  17
Child-SP         RetAddr          Call Site
000000001ea0ed78 000007fefd4710ac ntdll!NtWaitForSingleObject+0xa
000000001ea0ed80 000007fef81cb540 KERNELBASE!WaitForSingleObjectEx+0x9c
000000001ea0ee20 000007fef81cb42f mscorwks!CLREvent::WaitEx+0x174
000000001ea0ee70 000007fef81cb18b mscorwks!CLREvent::WaitEx+0x63
000000001ea0ef20 000007fef81d841a mscorwks!ThreadpoolMgr::SafeWait+0x7b
000000001ea0efe0 000007fef8298164 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11a
000000001ea0f080 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001ea0fad0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001ea0fb00 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  18
Child-SP         RetAddr          Call Site
000000001e88d578 000007fefd471420 ntdll!ZwWaitForMultipleObjects+0xa
000000001e88d580 0000000077332d53 KERNELBASE!GetCurrentProcess+0x40
000000001e88d680 000007fef817212d kernel32!WaitForMultipleObjectsExImplementation+0xb3
000000001e88d710 000007fef81769b9 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
000000001e88d7b0 000007fef8277cf9 mscorwks!Thread::DoAppropriateAptStateWait+0x41
000000001e88d810 000007fef8192b5c mscorwks!Thread::DoAppropriateWaitWorker+0x191
000000001e88d910 000007fef87133c1 mscorwks!Thread::DoAppropriateWait+0x5c
000000001e88d980 000007fef36c6653 mscorwks!WaitHandleNative::CorWaitOneNative+0x221
000000001e88dbc0 000007fef2b3e817 mscorlib_ni!System.Threading.WaitHandle.WaitOne(Int64, Boolean)+0x23
000000001e88dc00 000007fef2b48641 System_ni!System.Net.LazyAsyncResult.WaitForCompletion(Boolean)+0xc7
000000001e88dc60 000007fef2b47f34 System_ni!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)+0x2d1
000000001e88dcf0 000007fef2b474f8 System_ni!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)+0x84
000000001e88dd60 000007fef2b4b031 System_ni!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)+0x268
000000001e88ddc0 000007ff00705bbb System_ni!System.Net.HttpWebRequest.GetResponse()+0x291
000000001e88de50 000007fef09da30a RomikoCommon_72a00000!RomikoCommon.PageBase.OnPreInit(System.EventArgs)+0x36b
000000001e88df00 000007fef09dadb8 System_Web_ni!System.Web.UI.Page.PerformPreInit()+0x2a
000000001e88df30 000007fef09da750 System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x4e8
000000001e88e000 000007fef09da67b System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
000000001e88e060 000007fef09da610 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b
000000001e88e0c0 000007ff007049f4 System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0xf0
000000001e88e120 000007fef09e1ab7 App_Web_d_5hdouw!ASP.policysearch_aspx.ProcessRequest(System.Web.HttpContext)+0x34
000000001e88e150 000007fef09a571b System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x257
000000001e88e200 000007fef1090561 System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xab
000000001e88e2a0 000007fef10811c2 System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x511
000000001e88e430 000007fef1062df9 System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x72
000000001e88e480 000007fef1189931 System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+0x269
000000001e88e5a0 000007fef1189d8b System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x411
000000001e88e720 000007fef1189294 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x2b
000000001e88e780 000007fef831f18a System_Web_ni!DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)+0x24
000000001e88e7c0 000007fef7dd51a7 mscorwks!UMThunkStubAMD64+0x7a
000000001e88e850 000007fef7dd6262 webengine!RegisterModule+0x1e87
000000001e88e8a0 000007fef7dd1ade webengine!GetEcb+0x3c2
000000001e88e910 000007fef7dd2114 webengine!CSpinLock::ConvertSharedToExclusive+0x129e
000000001e88e9a0 000007fef8012de7 webengine!CSpinLock::ConvertSharedToExclusive+0x18d4
000000001e88e9e0 000007fef80146a4 iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x233
000000001e88ea20 000007fef801a775 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x174
000000001e88eb10 000007fef8015a03 iiscore!NOTIFICATION_CONTEXT::CallModules+0x25
000000001e88eb60 000007fef801a81c iiscore!W3_CONTEXT::DoWork+0x34d
000000001e88ee80 000007fef7dcfc41 iiscore!W3_CONTEXT::IndicateCompletion+0x8c
000000001e88eee0 000007fef8320e37 webengine!MgdIndicateCompletion+0x61
000000001e88ef10 000007fef108b25b mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
000000001e88efb0 000007fef1189a60 System_Web_ni!DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)+0x4b
000000001e88f090 000007fef1189d8b System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x540
000000001e88f210 000007fef1189294 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x2b
000000001e88f270 000007fef831f3db System_Web_ni!DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)+0x24
000000001e88f2b0 000007fef830d079 mscorwks!UM2MThunk_WrapperHelper+0x4b
000000001e88f2f0 000007fef830d165 mscorwks!UM2MThunk_WrapperWorker+0x99
000000001e88f380 000007fef8295bc9 mscorwks!UM2MThunk_Wrapper+0x19
000000001e88f3b0 000007fef82fe398 mscorwks!Thread::DoADCallBack+0x145
000000001e88f520 000007fef831f386 mscorwks!UM2MDoADCallBack+0x90
000000001e88f5b0 000007fef7dd51a7 mscorwks!UMThunkStubAMD64+0x276
000000001e88f640 000007fef7dd1066 webengine!RegisterModule+0x1e87
000000001e88f690 000007fef81c56f7 webengine!CSpinLock::ConvertSharedToExclusive+0x826
000000001e88f6c0 000007fef81d84ba mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x157
000000001e88f760 000007fef8298164 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1ba
000000001e88f800 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001e88fdd0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001e88fe00 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  19
Child-SP         RetAddr          Call Site
000000001b5ed648 000007fefd471420 ntdll!ZwWaitForMultipleObjects+0xa
000000001b5ed650 0000000077332d53 KERNELBASE!GetCurrentProcess+0x40
000000001b5ed750 000007fef817212d kernel32!WaitForMultipleObjectsExImplementation+0xb3
000000001b5ed7e0 000007fef81769b9 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
000000001b5ed880 000007fef8277cf9 mscorwks!Thread::DoAppropriateAptStateWait+0x41
000000001b5ed8e0 000007fef8192b5c mscorwks!Thread::DoAppropriateWaitWorker+0x191
000000001b5ed9e0 000007fef8712f21 mscorwks!Thread::DoAppropriateWait+0x5c
000000001b5eda50 000007fef3687e24 mscorwks!WaitHandleNative::CorWaitMultipleNative+0x2d1
000000001b5edcd0 000007fef2b11a67 mscorlib_ni!System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean)+0x64
000000001b5edd30 000007fef366dd38 System_ni!System.Net.TimerThread.ThreadProc()+0x327
000000001b5ede00 000007fef8321612 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x178
000000001b5edec0 000007fef824ee13 mscorwks!CallDescrWorker+0x82
000000001b5edf10 000007fef86fbc51 mscorwks!CallDescrWorkerWithHandler+0xd3
000000001b5edfb0 000007fef82456f2 mscorwks!MethodDesc::CallDescr+0x2b1
000000001b5ee200 000007fef877b182 mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0x12a
000000001b5ee490 000007fef3652b82 mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0x172
000000001b5ee6a0 000007fef36ea91d mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x62
000000001b5ee6f0 000007fef8321612 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x4d
000000001b5ee740 000007fef824ee13 mscorwks!CallDescrWorker+0x82
000000001b5ee790 000007fef86fbc51 mscorwks!CallDescrWorkerWithHandler+0xd3
000000001b5ee830 000007fef8137a7d mscorwks!MethodDesc::CallDescr+0x2b1
000000001b5eea70 000007fef8295a3c mscorwks!ThreadNative::KickOffThread_Worker+0x191
000000001b5eed90 000007fef827ba45 mscorwks!CClosedHashEx<CCustAttrHashKey,CCustAttrHash>::Hash+0x30
000000001b5eede0 000007fef81735b5 mscorwks!SVR::gc_heap::make_heap_segment+0x155
000000001b5eeeb0 000007fef82a277d mscorwks!EEToProfilerExceptionInterfaceWrapper::ExceptionCatcherEnter+0x4d
000000001b5eeef0 000007fef8295bc9 mscorwks!SharedFileLockHolderBase::DoRelease+0x4d
000000001b5eef20 000007fef8295a61 mscorwks!Thread::DoADCallBack+0x145
000000001b5ef090 000007fef827ba45 mscorwks!CClosedHashEx<CCustAttrHashKey,CCustAttrHash>::Hash+0x55
000000001b5ef0e0 000007fef81735b5 mscorwks!SVR::gc_heap::make_heap_segment+0x155
000000001b5ef1b0 000007fef82e05e9 mscorwks!EEToProfilerExceptionInterfaceWrapper::ExceptionCatcherEnter+0x4d
000000001b5ef1f0 000007fef829ffbb mscorwks!ZapNibbleMap::GetSize+0x55
000000001b5ef250 000007fef8298164 mscorwks!ThreadNative::KickOffThread+0xd3
000000001b5ef330 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001b5ef880 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001b5ef8b0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  21
Child-SP         RetAddr          Call Site
000000001de2eed8 000007fefd47169d ntdll!NtRemoveIoCompletion+0xa
000000001de2eee0 000000007731a4e1 KERNELBASE!GetQueuedCompletionStatus+0x3d
000000001de2ef40 000007fef82b347a kernel32!GetQueuedCompletionStatusStub+0x11
000000001de2ef80 000007fef8298164 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0xee
000000001de2f020 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001de2f7f0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001de2f820 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread  22
Child-SP         RetAddr          Call Site
000000001e07eed8 000007fefd47169d ntdll!NtRemoveIoCompletion+0xa
000000001e07eee0 000000007731a4e1 KERNELBASE!GetQueuedCompletionStatus+0x3d
000000001e07ef40 000007fef82b347a kernel32!GetQueuedCompletionStatusStub+0x11
000000001e07ef80 000007fef8298164 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0xee
000000001e07f020 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001e07f870 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001e07f8a0 0000000000000000 ntdll!RtlUserThreadStart+0x21

Thread 18 looks like a good candidate, we dealing with ASP>NET and we can see page loads etc, in fact a Page_Init is on the stack in that thread and a web request, also, the page I was testing was indeed a policysearch which had crashed!

000000001e88e120 000007fef09e1ab7 App_Web_d_5hdouw!ASP.policysearch_aspx.ProcessRequest

So, we on the right track, lets activate this as the active thread

we will then switch threads by typing ~18s, now we can load the stack as it is managed code.

0:000> ~18s
ntdll!ZwWaitForMultipleObjects+0xa:
00000000`774818ca c3              ret
0:018> !clrstack
OS Thread Id: 0x121c (18)
Child-SP         RetAddr          Call Site
000000001e88dbc0 000007fef2b3e817 System.Threading.WaitHandle.WaitOne(Int64, Boolean)
000000001e88dc00 000007fef2b48641 System.Net.LazyAsyncResult.WaitForCompletion(Boolean)
000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
000000001e88dcf0 000007fef2b474f8 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
000000001e88ddc0 000007ff00705bbb System.Net.HttpWebRequest.GetResponse()
000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)
000000001e88df00 000007fef09dadb8 System.Web.UI.Page.PerformPreInit()
000000001e88df30 000007fef09da750 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000001e88e000 000007fef09da67b System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000001e88e060 000007fef09da610 System.Web.UI.Page.ProcessRequest()
000000001e88e0c0 000007ff007049f4 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000001e88e120 000007fef09e1ab7 ASP.policysearch_aspx.ProcessRequest(System.Web.HttpContext)
000000001e88e150 000007fef09a571b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000001e88e200 000007fef1090561 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000001e88e2a0 000007fef10811c2 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000001e88e430 000007fef1062df9 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000001e88e480 000007fef1189931 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000001e88e5a0 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000001e88e720 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000001e88e780 000007fef831f18a DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000001e88efb0 000007fef1189a60 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000001e88f090 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000001e88f210 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000001e88f270 000007fef831f3db DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)

Interesting, we can see a SibmitRequest on the stack which is a synchronous call to a web page, so this web page is called another web page, and is waiting for a response. Lets get more info about this item:

000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)

lets get the paramters etc with the –p option for the clrstack

0:018> !clrstack -p
OS Thread Id: 0x121c (18)
Child-SP         RetAddr          Call Site
000000001e88dbc0 000007fef2b3e817 System.Threading.WaitHandle.WaitOne(Int64, Boolean)
    PARAMETERS:
        this = <no data>
        timeout = <no data>
        exitContext = <no data>

000000001e88dc00 000007fef2b48641 System.Net.LazyAsyncResult.WaitForCompletion(Boolean)
    PARAMETERS:
        this = 0x00000000033d4778
        snap = <no data>

000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
    PARAMETERS:
        this = <no data>
        request = <no data>

000000001e88dcf0 000007fef2b474f8 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
    PARAMETERS:
        this = <no data>
        request = <no data>
        connName = <no data>

000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
    PARAMETERS:
        this = 0x00000000033d3b48
        servicePoint = <no data>

000000001e88ddc0 000007ff00705bbb System.Net.HttpWebRequest.GetResponse()
    PARAMETERS:
        this = <no data>

000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)
    PARAMETERS:
        this = <no data>
        e = <no data>

000000001e88df00 000007fef09dadb8 System.Web.UI.Page.PerformPreInit()
    PARAMETERS:
        this = <no data>

000000001e88df30 000007fef09da750 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
    PARAMETERS:
        this = 0x00000000033adfc8
        includeStagesBeforeAsyncPoint = 0x0000000000000001
        includeStagesAfterAsyncPoint = 0x0000000000000001

000000001e88e000 000007fef09da67b System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
    PARAMETERS:
        this = 0x00000000033adfc8
        includeStagesBeforeAsyncPoint = <no data>
        includeStagesAfterAsyncPoint = 0x0000000000000001

000000001e88e060 000007fef09da610 System.Web.UI.Page.ProcessRequest()
    PARAMETERS:
        this = 0x00000000033adfc8

000000001e88e0c0 000007ff007049f4 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
    PARAMETERS:
        this = <no data>
        context = <no data>

000000001e88e120 000007fef09e1ab7 ASP.policysearch_aspx.ProcessRequest(System.Web.HttpContext)
    PARAMETERS:
        this = 0x00000000033adfc8
        context = 0x00000000033acc28

000000001e88e150 000007fef09a571b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    PARAMETERS:
        this = <no data>

000000001e88e200 000007fef1090561 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
    PARAMETERS:
        this = 0x000000000322d088
        step = 0x0000000003230990
        completedSynchronously = 0x000000001e88e359

000000001e88e2a0 000007fef10811c2 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
    PARAMETERS:
        this = 0x00000000032307c8
        error = <no data>

000000001e88e430 000007fef1062df9 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
    PARAMETERS:
        this = <no data>
        context = <no data>
        cb = <no data>

000000001e88e480 000007fef1189931 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
    PARAMETERS:
        this = 0x0000000002579188
        wr = 0x00000000033ac938
        context = 0x00000000033acc28

000000001e88e5a0 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
    PARAMETERS:
        managedHttpContext = <no data>
        nativeRequestContext = 0x000000001cd7fc58
        moduleData = <no data>
        flags = <no data>

000000001e88e720 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
    PARAMETERS:
        managedHttpContext = <no data>
        nativeRequestContext = <no data>
        moduleData = <no data>
        flags = <no data>

000000001e88e780 000007fef831f18a DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
    PARAMETERS:
        0x000000001cd7fc58
        <no data>
        <no data>
        <no data>

000000001e88efb0 000007fef1189a60 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
    PARAMETERS:
        0x0000000000b51500

000000001e88f090 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
    PARAMETERS:
        managedHttpContext = <no data>
        nativeRequestContext = <no data>
        moduleData = <no data>
        flags = <no data>

000000001e88f210 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
    PARAMETERS:
        managedHttpContext = <no data>
        nativeRequestContext = <no data>
        moduleData = <no data>
        flags = <no data>

000000001e88f270 000007fef831f3db DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
    PARAMETERS:
        0x000007fef830d14c
        <no data>
        <no data>
        <no data>

This has an object we can look at:

000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
    PARAMETERS:
        this = 0x00000000033d3b48
        servicePoint = <no data>

so lets use the !do command.

0:018> !do 0x00000000033d3b48
Name: System.Net.HttpWebRequest
MethodTable: 000007fef2c22f48
EEClass: 000007fef29997d8
Size: 400(0x190) bytes
(C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef37973f8  400018a        8        System.Object  0 instance 0000000000000000 __identity
000007fef31529b0  4001d76       28         System.Int32  1 instance                1 m_AuthenticationLevel
000007fef3f306a0  4001d77       2c         System.Int32  1 instance                4 m_ImpersonationLevel
000007fef2c22c70  4001d78       10 …equestCachePolicy  0 instance 00000000026b2930 m_CachePolicy
000007fef316c8a8  4001d79       18 …uestCacheProtocol  0 instance 0000000000000000 m_CacheProtocol
000007fef2c0bd50  4001d7a       20 …questCacheBinding  0 instance 00000000026b35d0 m_CacheBinding
000007fef379e9f0  4001d73      de0 …ections.ArrayList  0   shared           static s_PrefixList
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bee98 000000001ab979c0:00000000026b00a0 <<
000007fef37973f8  4001d74      de8        System.Object  0   shared           static s_InternalSyncObject
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bce20 000000001ab979c0:00000000026ae428 <<
000007fef3172938  4001d75      df0 …TimerThread+Queue  0   shared           static s_DefaultTimerQueue
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bc428 000000001ab979c0:00000000026ae368 <<
000007fef316af60  4001d7b      df8 System.Net.IWebProxy  0   shared           static s_DefaultWebProxy
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022c3740 000000001ab979c0:00000000026b39b0 <<
000007fef3796cd8  4001d7c      990       System.Boolean  1   shared           static s_DefaultWebProxyInitialized
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:1 000000001ab979c0:1 <<
000007fef3796cd8  4001f37      164       System.Boolean  1 instance                0 m_Saw100Continue
000007fef3796cd8  4001f38      165       System.Boolean  1 instance                1 m_KeepAlive
000007fef3796cd8  4001f39      166       System.Boolean  1 instance                0 m_LockConnection
000007fef3796cd8  4001f3a      167       System.Boolean  1 instance                0 m_NtlmKeepAlive
000007fef3796cd8  4001f3b      168       System.Boolean  1 instance                0 m_PreAuthenticate
000007fef3144c48  4001f3c      130         System.Int32  1 instance                0 m_AutomaticDecompression
000007fef379ed78  4001f3d      134         System.Int32  1 instance                0 m_Aborted
000007fef3796cd8  4001f3e      169       System.Boolean  1 instance                0 m_OnceFailed
000007fef3796cd8  4001f3f      16a       System.Boolean  1 instance                1 m_Pipelined
000007fef3796cd8  4001f40      16b       System.Boolean  1 instance                0 m_Retry
000007fef3796cd8  4001f41      16c       System.Boolean  1 instance                0 m_HeadersCompleted
000007fef3796cd8  4001f42      16d       System.Boolean  1 instance                0 m_IsCurrentAuthenticationStateProxy
000007fef3796cd8  4001f43      16e       System.Boolean  1 instance                0 m_SawInitialResponse
000007fef3796cd8  4001f44      16f       System.Boolean  1 instance                0 m_BodyStarted
000007fef3796cd8  4001f45      170       System.Boolean  1 instance                1 m_RequestSubmitted
000007fef3796cd8  4001f46      171       System.Boolean  1 instance                0 m_OriginallyBuffered
000007fef3796cd8  4001f47      172       System.Boolean  1 instance                0 m_Extra401Retry
000007fef3144dd8  4001f4f      138        System.UInt32  1 instance                7 _Booleans
000007fef37d8040  4001f50      178      System.DateTime  1 instance 00000000033d3cc0 _CachedIfModifedSince
000007fef2c0db18  4001f51       30 …TimerThread+Timer  0 instance 0000000000000000 m_ContinueTimer
000007fef2c0f0c0  4001f52      180 …t.InterlockedGate  1 instance 00000000033d3cc8 m_ContinueGate
000007fef37973f8  4001f53       38        System.Object  0 instance 0000000002573160 m_PendingReturnResult
000007fef2c00018  4001f54       40 …t.LazyAsyncResult  0 instance 0000000000000000 _WriteAResult
000007fef2c00018  4001f55       48 …t.LazyAsyncResult  0 instance 00000000033d4500 _ReadAResult
000007fef2c00018  4001f56       50 …t.LazyAsyncResult  0 instance 00000000033d4778 _ConnectionAResult
000007fef2c00018  4001f57       58 …t.LazyAsyncResult  0 instance 00000000033d47c0 _ConnectionReaderAResult
000007fef3146c10  4001f58      13c         System.Int32  1 instance                0 _RequestIsAsync
000007fef316ea50  4001f59       60 …pContinueDelegate  0 instance 0000000000000000 _ContinueDelegate
000007fef2c0c9e8  4001f5a       68 ….Net.ServicePoint  0 instance 00000000026c34d0 _ServicePoint
000007fef2c23c08  4001f5b       70 …t.HttpWebResponse  0 instance 0000000000000000 _HttpResponse
000007fef37973f8  4001f5c       78        System.Object  0 instance 0000000002573160 _CoreResponse
000007fef379ed78  4001f5d      140         System.Int32  1 instance                0 _NestedWriteSideCheck
000007fef2c0c968  4001f5e       80 …Net.KnownHttpVerb  0 instance 00000000026b3b28 _Verb
000007fef2c0c968  4001f5f       88 …Net.KnownHttpVerb  0 instance 00000000026b3b28 _OriginVerb
000007fef2bffea8  4001f60       90 …bHeaderCollection  0 instance 00000000033d3e58 _HttpRequestHeaders
000007fef379fb48  4001f61       98        System.Byte[]  0 instance 0000000000000000 _WriteBuffer
000007fef3146718  4001f62      144         System.Int32  1 instance                4 _HttpWriteMode
000007fef2c28bd8  4001f63       a0           System.Uri  0 instance 00000000033d3a50 _Uri
000007fef2c28bd8  4001f64       a8           System.Uri  0 instance 00000000033d3a50 _OriginUri
000007fef3797b08  4001f65       b0        System.String  0 instance 0000000000000000 _MediaType
000007fef379b028  4001f66      128         System.Int64  1 instance -1 _ContentLength
000007fef316af60  4001f67       b8 System.Net.IWebProxy  0 instance 00000000026b39b0 _Proxy
000007fef3171a00  4001f68       c0 …em.Net.ProxyChain  0 instance 00000000033d45f0 _ProxyChain
000007fef3797b08  4001f69       c8        System.String  0 instance 0000000000000000 _ConnectionGroupName
000007fef3796cd8  4001f6a      173       System.Boolean  1 instance                0 m_InternalConnectionGroup
000007fef2c0caf8  4001f6b       d0 …thenticationState  0 instance 00000000033d4b68 _ProxyAuthenticationState
000007fef2c0caf8  4001f6c       d8 …thenticationState  0 instance 00000000033d4bb8 _ServerAuthenticationState
000007fef316ac68  4001f6d       e0 ….Net.ICredentials  0 instance 0000000000000000 _AuthInfo
000007fef2c0cbf8  4001f6e       e8 …HttpAbortDelegate  0 instance 00000000027f3c00 _AbortDelegate
000007fef2c0ea00  4001f6f       f0 …Net.ConnectStream  0 instance 0000000000000000 _SubmitWriteStream
000007fef2c0ea00  4001f70       f8 …Net.ConnectStream  0 instance 0000000000000000 _OldSubmitWriteStream
000007fef379ed78  4001f71      148         System.Int32  1 instance               50 _MaximumAllowedRedirections
000007fef379ed78  4001f72      14c         System.Int32  1 instance                0 _AutoRedirects
000007fef379ed78  4001f73      150         System.Int32  1 instance                0 _RerequestCount
000007fef379ed78  4001f74      154         System.Int32  1 instance           100000 _Timeout
000007fef2c0db18  4001f75      100 …TimerThread+Timer  0 instance 00000000033d4548 _Timer
000007fef3172938  4001f76      108 …TimerThread+Queue  0 instance 00000000026ae368 _TimerQueue
000007fef379ed78  4001f77      158         System.Int32  1 instance                0 _RequestContinueCount
000007fef379ed78  4001f78      15c         System.Int32  1 instance           300000 _ReadWriteTimeout
000007fef3169e88  4001f79      110 …t.CookieContainer  0 instance 0000000000000000 _CookieContainer
000007fef379ed78  4001f7a      160         System.Int32  1 instance               64 _MaximumResponseHeadersLength
000007fef2c0d8b0  4001f7b      118 …onnectionDelegate  0 instance 0000000000000000 _UnlockDelegate
000007fef2c04da0  4001f7c      120 …ificateCollection  0 instance 0000000000000000 _ClientCertificates
000007fef379fb48  4001f48      e80        System.Byte[]  0   shared           static HttpBytes
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022befe0 000000001ab979c0:00000000026b6928 <<
000007fef3786130  4001f49      e88 …ding.WaitCallback  0   shared           static s_EndWriteHeaders_Part2Callback
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bf000 000000001ab979c0:00000000026b6948 <<
000007fef2c1d668  4001f4a      e90 …erThread+Callback  0   shared           static s_ContinueTimeoutCallback
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bf040 000000001ab979c0:00000000026b6988 <<
000007fef3172938  4001f4b      e98 …TimerThread+Queue  0   shared           static s_ContinueTimerQueue
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bf080 000000001ab979c0:00000000026b69c8 <<
000007fef2c1d668  4001f4c      ea0 …erThread+Callback  0   shared           static s_TimeoutCallback
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bf158 000000001ab979c0:00000000026b6aa0 <<
000007fef3786130  4001f4d      ea8 …ding.WaitCallback  0   shared           static s_AbortWrapper
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022bf198 000000001ab979c0:00000000026b6ae0 <<
000007fef379ed78  4001f4e      9a8         System.Int32  1   shared           static s_UniqueGroupId
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0 000000001ab979c0:0 <<

now we getting somewhere, I see this as an HttpWebRequest Object, and in here I can access the URI:

000007fef2c28bd8  4001f63       a0           System.Uri  0 instance 00000000033d3a50 _Uri

ok, so lets dump this out

0:018> !DumpObj 00000000033d3a50
Name: System.Uri
MethodTable: 000007fef2c28bd8
EEClass: 000007fef29543b8
Size: 72(0x48) bytes
(C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef3797b08  4001b8a        8        System.String  0 instance 00000000025ee6a0 m_String
000007fef3797b08  4001b8b       10        System.String  0 instance 0000000000000000 m_originalUnicodeString
000007fef2c28e60  4001b8c       18     System.UriParser  0 instance 000000000257e0c0 m_Syntax
000007fef3797b08  4001b8d       20        System.String  0 instance 0000000000000000 m_DnsSafeHost
000007fef313f598  4001b8e       30        System.UInt64  1 instance 37615763456 m_Flags
000007fef2c28fd0  4001b8f       28   System.Uri+UriInfo  0 instance 00000000033d3ac0 m_Info
000007fef3796cd8  4001b90       38       System.Boolean  1 instance                0 m_iriParsing
000007fef3797b08  4001b7f      c88        System.String  0   shared           static UriSchemeFile
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002122830 000000001ab979c0:0000000002122830 <<
000007fef3797b08  4001b80      c90        System.String  0   shared           static UriSchemeFtp
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002122810 000000001ab979c0:0000000002122810 <<
000007fef3797b08  4001b81      c98        System.String  0   shared           static UriSchemeGopher
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002122858 000000001ab979c0:0000000002122858 <<
000007fef3797b08  4001b82      ca0        System.String  0   shared           static UriSchemeHttp
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000021227c0 000000001ab979c0:00000000021227c0 <<
000007fef3797b08  4001b83      ca8        System.String  0   shared           static UriSchemeHttps
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000021227e8 000000001ab979c0:00000000021227e8 <<
000007fef3797b08  4001b84      cb0        System.String  0   shared           static UriSchemeMailto
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000021228d0 000000001ab979c0:00000000021228d0 <<
000007fef3797b08  4001b85      cb8        System.String  0   shared           static UriSchemeNews
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000021228a8 000000001ab979c0:00000000021228a8 <<
000007fef3797b08  4001b86      cc0        System.String  0   shared           static UriSchemeNntp
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002122880 000000001ab979c0:0000000002122880 <<
000007fef3797b08  4001b87      cc8        System.String  0   shared           static UriSchemeNetTcp
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002122970 000000001ab979c0:0000000002122970 <<
000007fef3797b08  4001b88      cd0        System.String  0   shared           static UriSchemeNetPipe
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002122998 000000001ab979c0:0000000002122998 <<
000007fef3797b08  4001b89      cd8        System.String  0   shared           static SchemeDelimiter
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000021227a0 000000001ab979c0:00000000021227a0 <<
000007fef31691a8  4001b91      ce0 …etSecurityManager  0   shared           static s_ManagerRef
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000000000000 000000001ab979c0:0000000000000000 <<
000007fef37973f8  4001b92      ce8        System.Object  0   shared           static s_IntranetLock
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002123388 000000001ab979c0:000000000257e280 <<
000007fef3796cd8  4001b93      970       System.Boolean  1   shared           static s_ConfigInitialized
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:1 000000001ab979c0:0 <<
000007fef31403c8  4001b94      974         System.Int32  1   shared           static s_IdnScope
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0 000000001ab979c0:0 <<
000007fef3796cd8  4001b95      978       System.Boolean  1   shared           static s_IriParsing
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0 000000001ab979c0:0 <<
000007fef37973f8  4001b96      cf0        System.Object  0   shared           static s_initLock
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000022ec898 000000001ab979c0:0000000000000000 <<
000007fef3799400  4001b97      cf8        System.Char[]  0   shared           static HexUpperChars
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000021233a0 000000001ab979c0:000000000257e298 <<
000007fef3799400  4001b98      d00        System.Char[]  0   shared           static HexLowerChars
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:00000000021233d8 000000001ab979c0:000000000257e2d0 <<
000007fef3799400  4001b99      d08        System.Char[]  0   shared           static _WSchars
                                 >> Domain:Value  00000000017ea4f0:NotInit  000000001a9d6a70:0000000002123410 000000001ab979c0:000000000257e308 <<

excellent, now lets get the string value

0:018> !DumpObj 00000000025ee6a0
Name: System.String
MethodTable: 000007fef3797b08
EEClass: 000007fef339e550
Size: 126(0x7e) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: http://Romiko.com.au/test.jsp
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef379ed78  4000096        8         System.Int32  1 instance               51 m_arrayLength
000007fef379ed78  4000097        c         System.Int32  1 instance               50 m_stringLength
000007fef3799550  4000098       10          System.Char  1 instance               68 m_firstChar
000007fef3797b08  4000099       20        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000017ea4f0:0000000002101308 000000001a9d6a70:0000000002101308 000000001ab979c0:0000000002101308 <<
000007fef3799400  400009a       28        System.Char[]  0   shared           static WhitespaceChars
                                 >> Domain:Value  00000000017ea4f0:0000000002101af8 000000001a9d6a70:0000000002109b68 000000001ab979c0:0000000002572798 <<

YAY ! We can see from here that the policy search is calling the test.jsp page, also we notice this happening in the stack from within the Page_Init. Lets go and get this source code my disassembling the module Smile

We need to tell the customer the exact code causing this problem, the way to do that is to get the source code and look at the Page_Init for the policysearch class.

Lets go back to the stack and find out which dll this code is coming from

0:018> !clrstack
OS Thread Id: 0x121c (18)
Child-SP         RetAddr          Call Site
000000001e88dbc0 000007fef2b3e817 System.Threading.WaitHandle.WaitOne(Int64, Boolean)
000000001e88dc00 000007fef2b48641 System.Net.LazyAsyncResult.WaitForCompletion(Boolean)
000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
000000001e88dcf0 000007fef2b474f8 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
000000001e88ddc0 000007ff00705bbb System.Net.HttpWebRequest.GetResponse()
000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)

FOUND IT:

000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)

what we need to do now is get this dll file, lets look at what is in the domain

0:018> !DumpDomain
————————————–
System Domain: 000007fef8933f20
LowFrequencyHeap: 000007fef8933f68
HighFrequencyHeap: 000007fef8933ff8
StubHeap: 000007fef8934088
Stage: OPEN
Name: None
————————————–
Shared Domain: 000007fef8934820
LowFrequencyHeap: 000007fef8934868
HighFrequencyHeap: 000007fef89348f8
StubHeap: 000007fef8934988
Stage: OPEN
Name: None
Assembly: 000000000172e0c0
Assembly: 000000001aa8f210
Assembly: 000000001ab529f0
Assembly: 000000001aa247e0
Assembly: 000000001aa9b220
Assembly: 000000001aac5970
Assembly: 000000000172e3c0
Assembly: 000000001aa8ef10
Assembly: 000000001aac5730
Assembly: 000000001ab52c30
Assembly: 000000001ca16590
Assembly: 00000000017e4f30
Assembly: 000000000172e540
Assembly: 000000001aa8f5d0
Assembly: 000000001ca03fd0
Assembly: 000000001aa8df50
Assembly: 00000000017e5230
Assembly: 000000001aa24ba0
Assembly: 00000000017e59b0
Assembly: 000000001aa25560
Assembly: 000000001ca16290
Assembly: 000000001aa253e0
Assembly: 000000001aa9a9e0
Assembly: 000000001aa8e0d0
Assembly: 000000001aa8f450
————————————–
Domain 1: 00000000017ea4f0
LowFrequencyHeap: 00000000017ea538
HighFrequencyHeap: 00000000017ea5c8
StubHeap: 00000000017ea658
Stage: OPEN
SecurityDescriptor: 00000000017c9d60
Name: DefaultDomain
Assembly: 000000000172e0c0 [C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 000000000172e180
SecurityDescriptor: 000000000172df40
  Module Name
000007fef3361000 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
000007ff000e2568 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
000007ff000e2020 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp

I cut allot out, but lets look at this, we have our romikocommon.dll further down

Assembly: 00000000017e6c70 [C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\purchasepath\2d20f8c3\9af2cafc\assembly\dl3\455d0c2c\ff027527_32ffcb01\RomikoCommon.DLL]
ClassLoader: 00000000017e6d30
SecurityDescriptor: 00000000017e6bb0
  Module Name
000007ff00176880 C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\purchasepath\2d20f8c3\9af2cafc\assembly\dl3\455d0c2c\ff027527_32ffcb01\CommonCommon.DLL

Nice, now, we going to dump this to a file Smile

We will use the savemodule command, but we need the base address, so issue the LM command

0:018> LM
start             end                 module name
00000000`1c170000 00000000`1c197000   System_EnterpriseServices_Wrapper_1c170000   (no symbols)          
00000000`1c290000 00000000`1c2b6000   ICSharpCode_SharpZipLib   (no symbols)          
00000000`1cdf0000 00000000`1d0ef000   System_Data   (export symbols)       System.Data.dll
00000000`6b570000 00000000`6bb04000   AjaxControlToolkit   (no symbols)                   
00000000`72a00000 00000000`72a18000   RomikoCommon_72a00000 C (no symbols)                 
00000000`72ee0000 00000000`72ef8000   App_Web_jweapfn4 C (no symbols)          
00000000`72f00000 00000000`72f0a000   App_Web_xg2ybqom C (no symbols)          
00000000`72f10000 00000000`72f18000   App_Web_1elshlii C (no symbols)          
00000000`72f20000 00000000`72f38000   RomikoCommon C (no symbols) 

perfect now we export the dll for .net reflector to use or ildasm, grabbing the base address above and dumping the contents as a dll.

0:018> !SaveModule 00000000`72a00000  e:\projects\debug\romikocommom.dll

remember in the stack we had this

RomikoCommon.PageBase.OnPreInit(System.EventArgs)

The savemodule has saved/dumped the dll, in reflector we can see the code causing the problem!

image

Here is the code

 protected override void OnPreInit(EventArgs e)
    {
        string str = (this._deployOrDev.ToUpper() == "DEV") ? this._sessionId : base.Request.QueryString["sid"];
        if (!string.IsNullOrEmpty(str))
        {
            string str2 = this.Session["sid"] as string;
            if (!string.IsNullOrEmpty(str2) && (string.Compare(str2, str, true) != 0))
            {
                this.Session.Clear();
            }
            this.Session["sid"] = str;
        }
        else
        {
            str = this.Session["sid"] as string;
        }
        if (string.IsNullOrEmpty(str))
        {
            throw new SecurityException("Session is empty.");
        }
        try
        {
            using (SecurityServicesProxy proxy = new SecurityServicesProxy())
            {
                UserCredential = proxy.GetUserCredential(str, Encrypt.EncryptData(str));
                if (UserCredential == null)
                {
                    throw new SessionExpiredException("The session is invalid. Please login again.");
                }
            }
        }
        catch (Exception exception)
        {
            HandleException(base.GetType().Name, MethodBase.GetCurrentMethod().Name, "Session Indentifier:" + str, exception, new Action<string>(this.DisplayError));
        }
        this.SetUserCredentialsForSession();
        try
        {
            WebRequest request = WebRequest.Create(this._keepAliveUrl);
            request.Headers.Add(HttpRequestHeader.Cookie, "JSESSIONID=" + str);
            request.GetResponse();
        }
        catch (Exception exception2)
        {
            _log.Info(exception2);
            throw new SessionExpiredException("Connecting Java system failed.", exception2);
        }
    }

WOW, look, every PAGE in their system is inheriting a custom pagebase, and this is called a external web page (remember test.jsp. It is doing this to keep the java session alive since the .net session is in an Iframe.

My load test always showed errors Connecting to Java system failed and here we can see why. The code should be changed, instead of all page loads calling java, rather have a background thread run and do it every 30 seconds whatever and we improve the performance of EVERY page.

we can also check that the request was not working and was waiting for the javasystem which crashed by checking cpu time.
0:018> kerneltime

Couldn’t resolve error at ‘erneltime’

0:018> .time

Debug session time: Fri May  6 15:31:14.000 2011 (UTC + 10:00)

System Uptime: 58 days 6:25:30.968

Process Uptime: 0 days 0:28:42.000

  Kernel time: 0 days 0:00:01.000

  User time: 0 days 0:00:05.000

it only using a fraction of a second, so it was definitely handing and waiting for a response, which of course would timeout at some point.

I hope you enjoyed this debugging session as much as I did, and thank you Tatham Oddie for geeking it up with me on this Smile

Romiko Derbynew

Advertisements

6 thoughts on “Memory Dump Analysis–W3WP IIS Process

  1. He Romiko,

    Great post man! Absolutely no idea what you are talking about :). How is everything down there?

    Rgds, Douwe.

  2. Pingback: Continuous Learner #12 « smnbss's Blog

  3. Pingback: Fix Memory Dump Iis Errors - Windows XP, Vista & Windows 7, 8

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s