At a customer I had prepared some Visual Studio 2010 WebTests which were calling their Java based website hosted on IBM Websphere, there is an IFrame on the Java page that points to an IIS hosted Asp.net 2.0 web site.
When running load tests I noticed that the webtests were taking a very long time to execute at one point the load test grinded the servers to a halt and a colleague of mine Tatham Oddie decided it was time we dump the memory from the IIS Process hosting the site.
So, right click the w3wp.exe process and click Create Dump File.
We also have windbg installed, which you can download from
Ensure you click debugging tools for windows under redistributable, so you get 32/64 bit versions
So, the big picture is for us to look at what is on the stack, ideally in software we should have items living on the stack for a very short period of time, so lets analyze the stack and see what clues it might have about the web site behaving badly.
Open Windbg, it is located in
C:\Program Files\Debugging Tools for Windows (x64) or you might have the 32 bit version.
I also copied the following files
w3wp.dmp (the generated dump
mscordacwks.dll (From Server hosting the IIS service)
mscordacwks_64.dll (From Server hosting the IIS service)
Start windbg and open the dump file.
excellent, we are presented with
.
SOS will need the Data Access Layer to make calls into the CLR, so we need the correct version of the mscordacwks.dll file from the Server where the software was running. This can be either 32 bit or 64 bit.
They are located at:
C:\Windows\Microsoft.NET\Framework\v2.0.50727
C:\Windows\Microsoft.NET\Framework64\v2.0.50727
Now, we need to tell the debugger where this file is, I copied these files and the dmp file from the server and put it into my E:\projects\debug folder
first, lets get the correct mscordacwks loaded, I renamed my 64bit version to this name:
mscordacwks_AMD64_AMD64_2.0.50727.5420.dll you can find the file version in the property page for mscordacwks_64
I then point my debugger to the renamed version of mscordackwks.
.sympath+ e:\projects\debug
then I load the DAC
.cordll –ve –u –l
I can now load the sos commands
.loadby sos mscorwks
if you have issues load the mscordacwks, you can also try
!sym noisy
.symfix e:\projects\debug
.cordll -ve -u -l
this will then try load the symbols from the microsoft server (http://msdl.microsoft.com/download/symbols), you must have an internet connection.
Lets see what threads are in the process:
0:000> !threads
ThreadCount: 37
UnstartedThread: 0
BackgroundThread: 10
PendingThread: 0
DeadThread: 27
Hosted Runtime: no
PreEmptive Lock
ID OSID ThreadOBJ State GC GC Alloc Context Domain Count APT Exception
4 1 1378 00000000017fb730 8220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
12 2 970 00000000017dd1d0 b220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Finalizer)
13 4 107c 000000001a9c8cf0 80a220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Threadpool Completion Port)
14 5 12d0 000000001a9d5fc0 1220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
XXXX 6 0 000000001aa36200 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Threadpool Worker)
XXXX 8 0 000000001aaac0b0 9820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
XXXX 9 0 000000001aaab850 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX a 0 000000001ca2b840 9820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
XXXX b 0 000000001caa3830 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX c 0 000000001ab937c0 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX d 0 000000001cab4680 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX e 0 000000001ca52940 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX f 0 000000001caaf450 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 10 0 000000001ca57290 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 11 0 000000001cab5c60 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Threadpool Worker)
XXXX 12 0 000000001aa52ca0 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 13 0 000000001aa53270 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
2 14 f48 000000001aa53840 220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
XXXX 15 0 000000001aa543e0 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
17 16 834 000000001aa549b0 180b220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Threadpool Worker)
XXXX 17 0 000000001aa54f80 9820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
XXXX 18 0 000000001aa55550 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Threadpool Worker)
XXXX 19 0 000000001aa55b20 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 1a 0 000000001aa560f0 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 1b 0 000000001aa566c0 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 1c 0 000000001aa56c90 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 1d 0 000000001aa57260 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 1e 0 000000001aa57830 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 1f 0 000000001aa57e00 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 20 0 000000001aa583d0 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
18 21 121c 000000001aa589a0 380b220 Enabled 0000000000000000:0000000000000000 000000001ab979c0 1 MTA (Threadpool Worker)
XXXX 22 0 000000001aa58f70 1801820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn (Threadpool Worker)
XXXX 24 0 000000001aa59b10 9820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
XXXX 26 0 000000001ef10680 9820 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 Ukn
19 25 11bc 000000001ef100b0 200b220 Enabled 0000000000000000:0000000000000000 000000001ab979c0 1 MTA
21 3 13c8 000000001aa526d0 880b220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Threadpool Completion Port)
22 7 12f4 000000001aa53e10 880b220 Enabled 0000000000000000:0000000000000000 00000000017ea4f0 0 MTA (Threadpool Completion Port)
Ok, from this we will need to try find the thread that is running all the IIS web pages etc. SO lets execute
0:000> !eestack
———————————————
Thread 4
Child-SP RetAddr Call Site
0000000001a2f7a8 000007fefd47169d ntdll!NtRemoveIoCompletion+0xa
0000000001a2f7b0 000000007731a4e1 KERNELBASE!GetQueuedCompletionStatus+0x3d
0000000001a2f810 000007fef9e81f7b kernel32!GetQueuedCompletionStatusStub+0x11
0000000001a2f850 000007fef9e82024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x3b
0000000001a2f8a0 000007fef9e820a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
0000000001a2f8d0 000000007732652d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
0000000001a2f900 000000007745c521 kernel32!BaseThreadInitThunk+0xd
0000000001a2f930 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 12
Child-SP RetAddr Call Site
000000001a9af5e8 000007fefd471420 ntdll!ZwWaitForMultipleObjects+0xa
000000001a9af5f0 0000000077321220 KERNELBASE!GetCurrentProcess+0x40
000000001a9af6f0 000007fef82a0353 kernel32!WaitForMultipleObjects+0xb0
000000001a9af780 000007fef8297b9a mscorwks!WKS::WaitForFinalizerEvent+0x93
000000001a9af7b0 000007fef8295a3c mscorwks!WKS::GCHeap::FinalizerThreadWorker+0x4a
000000001a9af7f0 000007fef827ba45 mscorwks!CClosedHashEx<CCustAttrHashKey,CCustAttrHash>::Hash+0x30
000000001a9af840 000007fef81735b5 mscorwks!SVR::gc_heap::make_heap_segment+0x155
000000001a9af910 000007fef829837a mscorwks!EEToProfilerExceptionInterfaceWrapper::ExceptionCatcherEnter+0x4d
000000001a9af950 000007fef82979b4 mscorwks!ManagedThreadBase_NoADTransition+0x42
000000001a9af9b0 000007fef8298164 mscorwks!WKS::GCHeap::FinalizerThreadStart+0x74
000000001a9af9f0 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001a9afac0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001a9afaf0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 13
Child-SP RetAddr Call Site
000000001a8afd28 000007fefd471203 ntdll!NtDelayExecution+0xa
000000001a8afd30 000007fef814812d KERNELBASE!SleepEx+0xb3
000000001a8afdd0 000007fef82a1d67 mscorwks!ThreadpoolMgr::TimerThreadFire+0x51
000000001a8afe70 000000007732652d mscorwks!ThreadpoolMgr::TimerThreadStart+0x6b
000000001a8afeb0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001a8afee0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 14
Child-SP RetAddr Call Site
000000001adcf368 000007fefd4710ac ntdll!NtWaitForSingleObject+0xa
000000001adcf370 000007fef81cb540 KERNELBASE!WaitForSingleObjectEx+0x9c
000000001adcf410 000007fef81cb42f mscorwks!CLREvent::WaitEx+0x174
000000001adcf460 000007fef82b1a9f mscorwks!CLREvent::WaitEx+0x63
000000001adcf510 000007fef8298164 mscorwks!AppDomain::ADUnloadThreadStart+0x153
000000001adcf630 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001adcf780 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001adcf7b0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 2
Child-SP RetAddr Call Site
0000000000b3f578 000000007744fe0b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
0000000000b3f580 000000007732652d ntdll!RtlValidateHeap+0x3bb
0000000000b3f880 000000007745c521 kernel32!BaseThreadInitThunk+0xd
0000000000b3f8b0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 17
Child-SP RetAddr Call Site
000000001ea0ed78 000007fefd4710ac ntdll!NtWaitForSingleObject+0xa
000000001ea0ed80 000007fef81cb540 KERNELBASE!WaitForSingleObjectEx+0x9c
000000001ea0ee20 000007fef81cb42f mscorwks!CLREvent::WaitEx+0x174
000000001ea0ee70 000007fef81cb18b mscorwks!CLREvent::WaitEx+0x63
000000001ea0ef20 000007fef81d841a mscorwks!ThreadpoolMgr::SafeWait+0x7b
000000001ea0efe0 000007fef8298164 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11a
000000001ea0f080 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001ea0fad0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001ea0fb00 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 18
Child-SP RetAddr Call Site
000000001e88d578 000007fefd471420 ntdll!ZwWaitForMultipleObjects+0xa
000000001e88d580 0000000077332d53 KERNELBASE!GetCurrentProcess+0x40
000000001e88d680 000007fef817212d kernel32!WaitForMultipleObjectsExImplementation+0xb3
000000001e88d710 000007fef81769b9 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
000000001e88d7b0 000007fef8277cf9 mscorwks!Thread::DoAppropriateAptStateWait+0x41
000000001e88d810 000007fef8192b5c mscorwks!Thread::DoAppropriateWaitWorker+0x191
000000001e88d910 000007fef87133c1 mscorwks!Thread::DoAppropriateWait+0x5c
000000001e88d980 000007fef36c6653 mscorwks!WaitHandleNative::CorWaitOneNative+0x221
000000001e88dbc0 000007fef2b3e817 mscorlib_ni!System.Threading.WaitHandle.WaitOne(Int64, Boolean)+0x23
000000001e88dc00 000007fef2b48641 System_ni!System.Net.LazyAsyncResult.WaitForCompletion(Boolean)+0xc7
000000001e88dc60 000007fef2b47f34 System_ni!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)+0x2d1
000000001e88dcf0 000007fef2b474f8 System_ni!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)+0x84
000000001e88dd60 000007fef2b4b031 System_ni!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)+0x268
000000001e88ddc0 000007ff00705bbb System_ni!System.Net.HttpWebRequest.GetResponse()+0x291
000000001e88de50 000007fef09da30a RomikoCommon_72a00000!RomikoCommon.PageBase.OnPreInit(System.EventArgs)+0x36b
000000001e88df00 000007fef09dadb8 System_Web_ni!System.Web.UI.Page.PerformPreInit()+0x2a
000000001e88df30 000007fef09da750 System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x4e8
000000001e88e000 000007fef09da67b System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
000000001e88e060 000007fef09da610 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b
000000001e88e0c0 000007ff007049f4 System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0xf0
000000001e88e120 000007fef09e1ab7 App_Web_d_5hdouw!ASP.policysearch_aspx.ProcessRequest(System.Web.HttpContext)+0x34
000000001e88e150 000007fef09a571b System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x257
000000001e88e200 000007fef1090561 System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xab
000000001e88e2a0 000007fef10811c2 System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x511
000000001e88e430 000007fef1062df9 System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x72
000000001e88e480 000007fef1189931 System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+0x269
000000001e88e5a0 000007fef1189d8b System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x411
000000001e88e720 000007fef1189294 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x2b
000000001e88e780 000007fef831f18a System_Web_ni!DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)+0x24
000000001e88e7c0 000007fef7dd51a7 mscorwks!UMThunkStubAMD64+0x7a
000000001e88e850 000007fef7dd6262 webengine!RegisterModule+0x1e87
000000001e88e8a0 000007fef7dd1ade webengine!GetEcb+0x3c2
000000001e88e910 000007fef7dd2114 webengine!CSpinLock::ConvertSharedToExclusive+0x129e
000000001e88e9a0 000007fef8012de7 webengine!CSpinLock::ConvertSharedToExclusive+0x18d4
000000001e88e9e0 000007fef80146a4 iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x233
000000001e88ea20 000007fef801a775 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x174
000000001e88eb10 000007fef8015a03 iiscore!NOTIFICATION_CONTEXT::CallModules+0x25
000000001e88eb60 000007fef801a81c iiscore!W3_CONTEXT::DoWork+0x34d
000000001e88ee80 000007fef7dcfc41 iiscore!W3_CONTEXT::IndicateCompletion+0x8c
000000001e88eee0 000007fef8320e37 webengine!MgdIndicateCompletion+0x61
000000001e88ef10 000007fef108b25b mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
000000001e88efb0 000007fef1189a60 System_Web_ni!DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)+0x4b
000000001e88f090 000007fef1189d8b System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x540
000000001e88f210 000007fef1189294 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x2b
000000001e88f270 000007fef831f3db System_Web_ni!DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)+0x24
000000001e88f2b0 000007fef830d079 mscorwks!UM2MThunk_WrapperHelper+0x4b
000000001e88f2f0 000007fef830d165 mscorwks!UM2MThunk_WrapperWorker+0x99
000000001e88f380 000007fef8295bc9 mscorwks!UM2MThunk_Wrapper+0x19
000000001e88f3b0 000007fef82fe398 mscorwks!Thread::DoADCallBack+0x145
000000001e88f520 000007fef831f386 mscorwks!UM2MDoADCallBack+0x90
000000001e88f5b0 000007fef7dd51a7 mscorwks!UMThunkStubAMD64+0x276
000000001e88f640 000007fef7dd1066 webengine!RegisterModule+0x1e87
000000001e88f690 000007fef81c56f7 webengine!CSpinLock::ConvertSharedToExclusive+0x826
000000001e88f6c0 000007fef81d84ba mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x157
000000001e88f760 000007fef8298164 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1ba
000000001e88f800 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001e88fdd0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001e88fe00 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 19
Child-SP RetAddr Call Site
000000001b5ed648 000007fefd471420 ntdll!ZwWaitForMultipleObjects+0xa
000000001b5ed650 0000000077332d53 KERNELBASE!GetCurrentProcess+0x40
000000001b5ed750 000007fef817212d kernel32!WaitForMultipleObjectsExImplementation+0xb3
000000001b5ed7e0 000007fef81769b9 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
000000001b5ed880 000007fef8277cf9 mscorwks!Thread::DoAppropriateAptStateWait+0x41
000000001b5ed8e0 000007fef8192b5c mscorwks!Thread::DoAppropriateWaitWorker+0x191
000000001b5ed9e0 000007fef8712f21 mscorwks!Thread::DoAppropriateWait+0x5c
000000001b5eda50 000007fef3687e24 mscorwks!WaitHandleNative::CorWaitMultipleNative+0x2d1
000000001b5edcd0 000007fef2b11a67 mscorlib_ni!System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean)+0x64
000000001b5edd30 000007fef366dd38 System_ni!System.Net.TimerThread.ThreadProc()+0x327
000000001b5ede00 000007fef8321612 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x178
000000001b5edec0 000007fef824ee13 mscorwks!CallDescrWorker+0x82
000000001b5edf10 000007fef86fbc51 mscorwks!CallDescrWorkerWithHandler+0xd3
000000001b5edfb0 000007fef82456f2 mscorwks!MethodDesc::CallDescr+0x2b1
000000001b5ee200 000007fef877b182 mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0x12a
000000001b5ee490 000007fef3652b82 mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0x172
000000001b5ee6a0 000007fef36ea91d mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x62
000000001b5ee6f0 000007fef8321612 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x4d
000000001b5ee740 000007fef824ee13 mscorwks!CallDescrWorker+0x82
000000001b5ee790 000007fef86fbc51 mscorwks!CallDescrWorkerWithHandler+0xd3
000000001b5ee830 000007fef8137a7d mscorwks!MethodDesc::CallDescr+0x2b1
000000001b5eea70 000007fef8295a3c mscorwks!ThreadNative::KickOffThread_Worker+0x191
000000001b5eed90 000007fef827ba45 mscorwks!CClosedHashEx<CCustAttrHashKey,CCustAttrHash>::Hash+0x30
000000001b5eede0 000007fef81735b5 mscorwks!SVR::gc_heap::make_heap_segment+0x155
000000001b5eeeb0 000007fef82a277d mscorwks!EEToProfilerExceptionInterfaceWrapper::ExceptionCatcherEnter+0x4d
000000001b5eeef0 000007fef8295bc9 mscorwks!SharedFileLockHolderBase::DoRelease+0x4d
000000001b5eef20 000007fef8295a61 mscorwks!Thread::DoADCallBack+0x145
000000001b5ef090 000007fef827ba45 mscorwks!CClosedHashEx<CCustAttrHashKey,CCustAttrHash>::Hash+0x55
000000001b5ef0e0 000007fef81735b5 mscorwks!SVR::gc_heap::make_heap_segment+0x155
000000001b5ef1b0 000007fef82e05e9 mscorwks!EEToProfilerExceptionInterfaceWrapper::ExceptionCatcherEnter+0x4d
000000001b5ef1f0 000007fef829ffbb mscorwks!ZapNibbleMap::GetSize+0x55
000000001b5ef250 000007fef8298164 mscorwks!ThreadNative::KickOffThread+0xd3
000000001b5ef330 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001b5ef880 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001b5ef8b0 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 21
Child-SP RetAddr Call Site
000000001de2eed8 000007fefd47169d ntdll!NtRemoveIoCompletion+0xa
000000001de2eee0 000000007731a4e1 KERNELBASE!GetQueuedCompletionStatus+0x3d
000000001de2ef40 000007fef82b347a kernel32!GetQueuedCompletionStatusStub+0x11
000000001de2ef80 000007fef8298164 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0xee
000000001de2f020 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001de2f7f0 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001de2f820 0000000000000000 ntdll!RtlUserThreadStart+0x21
———————————————
Thread 22
Child-SP RetAddr Call Site
000000001e07eed8 000007fefd47169d ntdll!NtRemoveIoCompletion+0xa
000000001e07eee0 000000007731a4e1 KERNELBASE!GetQueuedCompletionStatus+0x3d
000000001e07ef40 000007fef82b347a kernel32!GetQueuedCompletionStatusStub+0x11
000000001e07ef80 000007fef8298164 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0xee
000000001e07f020 000000007732652d mscorwks!Thread::intermediateThreadProc+0x78
000000001e07f870 000000007745c521 kernel32!BaseThreadInitThunk+0xd
000000001e07f8a0 0000000000000000 ntdll!RtlUserThreadStart+0x21
Thread 18 looks like a good candidate, we dealing with ASP>NET and we can see page loads etc, in fact a Page_Init is on the stack in that thread and a web request, also, the page I was testing was indeed a policysearch which had crashed!
000000001e88e120 000007fef09e1ab7 App_Web_d_5hdouw!ASP.policysearch_aspx.ProcessRequest
So, we on the right track, lets activate this as the active thread
we will then switch threads by typing ~18s, now we can load the stack as it is managed code.
0:000> ~18s
ntdll!ZwWaitForMultipleObjects+0xa:
00000000`774818ca c3 ret
0:018> !clrstack
OS Thread Id: 0x121c (18)
Child-SP RetAddr Call Site
000000001e88dbc0 000007fef2b3e817 System.Threading.WaitHandle.WaitOne(Int64, Boolean)
000000001e88dc00 000007fef2b48641 System.Net.LazyAsyncResult.WaitForCompletion(Boolean)
000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
000000001e88dcf0 000007fef2b474f8 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
000000001e88ddc0 000007ff00705bbb System.Net.HttpWebRequest.GetResponse()
000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)
000000001e88df00 000007fef09dadb8 System.Web.UI.Page.PerformPreInit()
000000001e88df30 000007fef09da750 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000001e88e000 000007fef09da67b System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000001e88e060 000007fef09da610 System.Web.UI.Page.ProcessRequest()
000000001e88e0c0 000007ff007049f4 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000001e88e120 000007fef09e1ab7 ASP.policysearch_aspx.ProcessRequest(System.Web.HttpContext)
000000001e88e150 000007fef09a571b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000001e88e200 000007fef1090561 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000001e88e2a0 000007fef10811c2 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000001e88e430 000007fef1062df9 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000001e88e480 000007fef1189931 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000001e88e5a0 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000001e88e720 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000001e88e780 000007fef831f18a DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000001e88efb0 000007fef1189a60 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000001e88f090 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000001e88f210 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000001e88f270 000007fef831f3db DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
Interesting, we can see a SibmitRequest on the stack which is a synchronous call to a web page, so this web page is called another web page, and is waiting for a response. Lets get more info about this item:
000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
lets get the paramters etc with the –p option for the clrstack
0:018> !clrstack -p
OS Thread Id: 0x121c (18)
Child-SP RetAddr Call Site
000000001e88dbc0 000007fef2b3e817 System.Threading.WaitHandle.WaitOne(Int64, Boolean)
PARAMETERS:
this = <no data>
timeout = <no data>
exitContext = <no data>
000000001e88dc00 000007fef2b48641 System.Net.LazyAsyncResult.WaitForCompletion(Boolean)
PARAMETERS:
this = 0x00000000033d4778
snap = <no data>
000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
PARAMETERS:
this = <no data>
request = <no data>
000000001e88dcf0 000007fef2b474f8 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
PARAMETERS:
this = <no data>
request = <no data>
connName = <no data>
000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
PARAMETERS:
this = 0x00000000033d3b48
servicePoint = <no data>
000000001e88ddc0 000007ff00705bbb System.Net.HttpWebRequest.GetResponse()
PARAMETERS:
this = <no data>
000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)
PARAMETERS:
this = <no data>
e = <no data>
000000001e88df00 000007fef09dadb8 System.Web.UI.Page.PerformPreInit()
PARAMETERS:
this = <no data>
000000001e88df30 000007fef09da750 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
PARAMETERS:
this = 0x00000000033adfc8
includeStagesBeforeAsyncPoint = 0x0000000000000001
includeStagesAfterAsyncPoint = 0x0000000000000001
000000001e88e000 000007fef09da67b System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
PARAMETERS:
this = 0x00000000033adfc8
includeStagesBeforeAsyncPoint = <no data>
includeStagesAfterAsyncPoint = 0x0000000000000001
000000001e88e060 000007fef09da610 System.Web.UI.Page.ProcessRequest()
PARAMETERS:
this = 0x00000000033adfc8
000000001e88e0c0 000007ff007049f4 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
PARAMETERS:
this = <no data>
context = <no data>
000000001e88e120 000007fef09e1ab7 ASP.policysearch_aspx.ProcessRequest(System.Web.HttpContext)
PARAMETERS:
this = 0x00000000033adfc8
context = 0x00000000033acc28
000000001e88e150 000007fef09a571b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
PARAMETERS:
this = <no data>
000000001e88e200 000007fef1090561 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
PARAMETERS:
this = 0x000000000322d088
step = 0x0000000003230990
completedSynchronously = 0x000000001e88e359
000000001e88e2a0 000007fef10811c2 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
PARAMETERS:
this = 0x00000000032307c8
error = <no data>
000000001e88e430 000007fef1062df9 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
PARAMETERS:
this = <no data>
context = <no data>
cb = <no data>
000000001e88e480 000007fef1189931 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
PARAMETERS:
this = 0x0000000002579188
wr = 0x00000000033ac938
context = 0x00000000033acc28
000000001e88e5a0 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
PARAMETERS:
managedHttpContext = <no data>
nativeRequestContext = 0x000000001cd7fc58
moduleData = <no data>
flags = <no data>
000000001e88e720 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
PARAMETERS:
managedHttpContext = <no data>
nativeRequestContext = <no data>
moduleData = <no data>
flags = <no data>
000000001e88e780 000007fef831f18a DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
PARAMETERS:
0x000000001cd7fc58
<no data>
<no data>
<no data>
000000001e88efb0 000007fef1189a60 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
PARAMETERS:
0x0000000000b51500
000000001e88f090 000007fef1189d8b System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
PARAMETERS:
managedHttpContext = <no data>
nativeRequestContext = <no data>
moduleData = <no data>
flags = <no data>
000000001e88f210 000007fef1189294 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
PARAMETERS:
managedHttpContext = <no data>
nativeRequestContext = <no data>
moduleData = <no data>
flags = <no data>
000000001e88f270 000007fef831f3db DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
PARAMETERS:
0x000007fef830d14c
<no data>
<no data>
<no data>
This has an object we can look at:
000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
PARAMETERS:
this = 0x00000000033d3b48
servicePoint = <no data>
so lets use the !do command.
0:018> !do 0x00000000033d3b48
Name: System.Net.HttpWebRequest
MethodTable: 000007fef2c22f48
EEClass: 000007fef29997d8
Size: 400(0x190) bytes
(C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef37973f8 400018a 8 System.Object 0 instance 0000000000000000 __identity
000007fef31529b0 4001d76 28 System.Int32 1 instance 1 m_AuthenticationLevel
000007fef3f306a0 4001d77 2c System.Int32 1 instance 4 m_ImpersonationLevel
000007fef2c22c70 4001d78 10 …equestCachePolicy 0 instance 00000000026b2930 m_CachePolicy
000007fef316c8a8 4001d79 18 …uestCacheProtocol 0 instance 0000000000000000 m_CacheProtocol
000007fef2c0bd50 4001d7a 20 …questCacheBinding 0 instance 00000000026b35d0 m_CacheBinding
000007fef379e9f0 4001d73 de0 …ections.ArrayList 0 shared static s_PrefixList
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bee98 000000001ab979c0:00000000026b00a0 <<
000007fef37973f8 4001d74 de8 System.Object 0 shared static s_InternalSyncObject
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bce20 000000001ab979c0:00000000026ae428 <<
000007fef3172938 4001d75 df0 …TimerThread+Queue 0 shared static s_DefaultTimerQueue
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bc428 000000001ab979c0:00000000026ae368 <<
000007fef316af60 4001d7b df8 System.Net.IWebProxy 0 shared static s_DefaultWebProxy
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022c3740 000000001ab979c0:00000000026b39b0 <<
000007fef3796cd8 4001d7c 990 System.Boolean 1 shared static s_DefaultWebProxyInitialized
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:1 000000001ab979c0:1 <<
000007fef3796cd8 4001f37 164 System.Boolean 1 instance 0 m_Saw100Continue
000007fef3796cd8 4001f38 165 System.Boolean 1 instance 1 m_KeepAlive
000007fef3796cd8 4001f39 166 System.Boolean 1 instance 0 m_LockConnection
000007fef3796cd8 4001f3a 167 System.Boolean 1 instance 0 m_NtlmKeepAlive
000007fef3796cd8 4001f3b 168 System.Boolean 1 instance 0 m_PreAuthenticate
000007fef3144c48 4001f3c 130 System.Int32 1 instance 0 m_AutomaticDecompression
000007fef379ed78 4001f3d 134 System.Int32 1 instance 0 m_Aborted
000007fef3796cd8 4001f3e 169 System.Boolean 1 instance 0 m_OnceFailed
000007fef3796cd8 4001f3f 16a System.Boolean 1 instance 1 m_Pipelined
000007fef3796cd8 4001f40 16b System.Boolean 1 instance 0 m_Retry
000007fef3796cd8 4001f41 16c System.Boolean 1 instance 0 m_HeadersCompleted
000007fef3796cd8 4001f42 16d System.Boolean 1 instance 0 m_IsCurrentAuthenticationStateProxy
000007fef3796cd8 4001f43 16e System.Boolean 1 instance 0 m_SawInitialResponse
000007fef3796cd8 4001f44 16f System.Boolean 1 instance 0 m_BodyStarted
000007fef3796cd8 4001f45 170 System.Boolean 1 instance 1 m_RequestSubmitted
000007fef3796cd8 4001f46 171 System.Boolean 1 instance 0 m_OriginallyBuffered
000007fef3796cd8 4001f47 172 System.Boolean 1 instance 0 m_Extra401Retry
000007fef3144dd8 4001f4f 138 System.UInt32 1 instance 7 _Booleans
000007fef37d8040 4001f50 178 System.DateTime 1 instance 00000000033d3cc0 _CachedIfModifedSince
000007fef2c0db18 4001f51 30 …TimerThread+Timer 0 instance 0000000000000000 m_ContinueTimer
000007fef2c0f0c0 4001f52 180 …t.InterlockedGate 1 instance 00000000033d3cc8 m_ContinueGate
000007fef37973f8 4001f53 38 System.Object 0 instance 0000000002573160 m_PendingReturnResult
000007fef2c00018 4001f54 40 …t.LazyAsyncResult 0 instance 0000000000000000 _WriteAResult
000007fef2c00018 4001f55 48 …t.LazyAsyncResult 0 instance 00000000033d4500 _ReadAResult
000007fef2c00018 4001f56 50 …t.LazyAsyncResult 0 instance 00000000033d4778 _ConnectionAResult
000007fef2c00018 4001f57 58 …t.LazyAsyncResult 0 instance 00000000033d47c0 _ConnectionReaderAResult
000007fef3146c10 4001f58 13c System.Int32 1 instance 0 _RequestIsAsync
000007fef316ea50 4001f59 60 …pContinueDelegate 0 instance 0000000000000000 _ContinueDelegate
000007fef2c0c9e8 4001f5a 68 ….Net.ServicePoint 0 instance 00000000026c34d0 _ServicePoint
000007fef2c23c08 4001f5b 70 …t.HttpWebResponse 0 instance 0000000000000000 _HttpResponse
000007fef37973f8 4001f5c 78 System.Object 0 instance 0000000002573160 _CoreResponse
000007fef379ed78 4001f5d 140 System.Int32 1 instance 0 _NestedWriteSideCheck
000007fef2c0c968 4001f5e 80 …Net.KnownHttpVerb 0 instance 00000000026b3b28 _Verb
000007fef2c0c968 4001f5f 88 …Net.KnownHttpVerb 0 instance 00000000026b3b28 _OriginVerb
000007fef2bffea8 4001f60 90 …bHeaderCollection 0 instance 00000000033d3e58 _HttpRequestHeaders
000007fef379fb48 4001f61 98 System.Byte[] 0 instance 0000000000000000 _WriteBuffer
000007fef3146718 4001f62 144 System.Int32 1 instance 4 _HttpWriteMode
000007fef2c28bd8 4001f63 a0 System.Uri 0 instance 00000000033d3a50 _Uri
000007fef2c28bd8 4001f64 a8 System.Uri 0 instance 00000000033d3a50 _OriginUri
000007fef3797b08 4001f65 b0 System.String 0 instance 0000000000000000 _MediaType
000007fef379b028 4001f66 128 System.Int64 1 instance -1 _ContentLength
000007fef316af60 4001f67 b8 System.Net.IWebProxy 0 instance 00000000026b39b0 _Proxy
000007fef3171a00 4001f68 c0 …em.Net.ProxyChain 0 instance 00000000033d45f0 _ProxyChain
000007fef3797b08 4001f69 c8 System.String 0 instance 0000000000000000 _ConnectionGroupName
000007fef3796cd8 4001f6a 173 System.Boolean 1 instance 0 m_InternalConnectionGroup
000007fef2c0caf8 4001f6b d0 …thenticationState 0 instance 00000000033d4b68 _ProxyAuthenticationState
000007fef2c0caf8 4001f6c d8 …thenticationState 0 instance 00000000033d4bb8 _ServerAuthenticationState
000007fef316ac68 4001f6d e0 ….Net.ICredentials 0 instance 0000000000000000 _AuthInfo
000007fef2c0cbf8 4001f6e e8 …HttpAbortDelegate 0 instance 00000000027f3c00 _AbortDelegate
000007fef2c0ea00 4001f6f f0 …Net.ConnectStream 0 instance 0000000000000000 _SubmitWriteStream
000007fef2c0ea00 4001f70 f8 …Net.ConnectStream 0 instance 0000000000000000 _OldSubmitWriteStream
000007fef379ed78 4001f71 148 System.Int32 1 instance 50 _MaximumAllowedRedirections
000007fef379ed78 4001f72 14c System.Int32 1 instance 0 _AutoRedirects
000007fef379ed78 4001f73 150 System.Int32 1 instance 0 _RerequestCount
000007fef379ed78 4001f74 154 System.Int32 1 instance 100000 _Timeout
000007fef2c0db18 4001f75 100 …TimerThread+Timer 0 instance 00000000033d4548 _Timer
000007fef3172938 4001f76 108 …TimerThread+Queue 0 instance 00000000026ae368 _TimerQueue
000007fef379ed78 4001f77 158 System.Int32 1 instance 0 _RequestContinueCount
000007fef379ed78 4001f78 15c System.Int32 1 instance 300000 _ReadWriteTimeout
000007fef3169e88 4001f79 110 …t.CookieContainer 0 instance 0000000000000000 _CookieContainer
000007fef379ed78 4001f7a 160 System.Int32 1 instance 64 _MaximumResponseHeadersLength
000007fef2c0d8b0 4001f7b 118 …onnectionDelegate 0 instance 0000000000000000 _UnlockDelegate
000007fef2c04da0 4001f7c 120 …ificateCollection 0 instance 0000000000000000 _ClientCertificates
000007fef379fb48 4001f48 e80 System.Byte[] 0 shared static HttpBytes
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022befe0 000000001ab979c0:00000000026b6928 <<
000007fef3786130 4001f49 e88 …ding.WaitCallback 0 shared static s_EndWriteHeaders_Part2Callback
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bf000 000000001ab979c0:00000000026b6948 <<
000007fef2c1d668 4001f4a e90 …erThread+Callback 0 shared static s_ContinueTimeoutCallback
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bf040 000000001ab979c0:00000000026b6988 <<
000007fef3172938 4001f4b e98 …TimerThread+Queue 0 shared static s_ContinueTimerQueue
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bf080 000000001ab979c0:00000000026b69c8 <<
000007fef2c1d668 4001f4c ea0 …erThread+Callback 0 shared static s_TimeoutCallback
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bf158 000000001ab979c0:00000000026b6aa0 <<
000007fef3786130 4001f4d ea8 …ding.WaitCallback 0 shared static s_AbortWrapper
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022bf198 000000001ab979c0:00000000026b6ae0 <<
000007fef379ed78 4001f4e 9a8 System.Int32 1 shared static s_UniqueGroupId
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0 000000001ab979c0:0 <<
now we getting somewhere, I see this as an HttpWebRequest Object, and in here I can access the URI:
000007fef2c28bd8 4001f63 a0 System.Uri 0 instance 00000000033d3a50 _Uri
ok, so lets dump this out
0:018> !DumpObj 00000000033d3a50
Name: System.Uri
MethodTable: 000007fef2c28bd8
EEClass: 000007fef29543b8
Size: 72(0x48) bytes
(C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef3797b08 4001b8a 8 System.String 0 instance 00000000025ee6a0 m_String
000007fef3797b08 4001b8b 10 System.String 0 instance 0000000000000000 m_originalUnicodeString
000007fef2c28e60 4001b8c 18 System.UriParser 0 instance 000000000257e0c0 m_Syntax
000007fef3797b08 4001b8d 20 System.String 0 instance 0000000000000000 m_DnsSafeHost
000007fef313f598 4001b8e 30 System.UInt64 1 instance 37615763456 m_Flags
000007fef2c28fd0 4001b8f 28 System.Uri+UriInfo 0 instance 00000000033d3ac0 m_Info
000007fef3796cd8 4001b90 38 System.Boolean 1 instance 0 m_iriParsing
000007fef3797b08 4001b7f c88 System.String 0 shared static UriSchemeFile
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002122830 000000001ab979c0:0000000002122830 <<
000007fef3797b08 4001b80 c90 System.String 0 shared static UriSchemeFtp
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002122810 000000001ab979c0:0000000002122810 <<
000007fef3797b08 4001b81 c98 System.String 0 shared static UriSchemeGopher
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002122858 000000001ab979c0:0000000002122858 <<
000007fef3797b08 4001b82 ca0 System.String 0 shared static UriSchemeHttp
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000021227c0 000000001ab979c0:00000000021227c0 <<
000007fef3797b08 4001b83 ca8 System.String 0 shared static UriSchemeHttps
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000021227e8 000000001ab979c0:00000000021227e8 <<
000007fef3797b08 4001b84 cb0 System.String 0 shared static UriSchemeMailto
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000021228d0 000000001ab979c0:00000000021228d0 <<
000007fef3797b08 4001b85 cb8 System.String 0 shared static UriSchemeNews
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000021228a8 000000001ab979c0:00000000021228a8 <<
000007fef3797b08 4001b86 cc0 System.String 0 shared static UriSchemeNntp
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002122880 000000001ab979c0:0000000002122880 <<
000007fef3797b08 4001b87 cc8 System.String 0 shared static UriSchemeNetTcp
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002122970 000000001ab979c0:0000000002122970 <<
000007fef3797b08 4001b88 cd0 System.String 0 shared static UriSchemeNetPipe
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002122998 000000001ab979c0:0000000002122998 <<
000007fef3797b08 4001b89 cd8 System.String 0 shared static SchemeDelimiter
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000021227a0 000000001ab979c0:00000000021227a0 <<
000007fef31691a8 4001b91 ce0 …etSecurityManager 0 shared static s_ManagerRef
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000000000000 000000001ab979c0:0000000000000000 <<
000007fef37973f8 4001b92 ce8 System.Object 0 shared static s_IntranetLock
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002123388 000000001ab979c0:000000000257e280 <<
000007fef3796cd8 4001b93 970 System.Boolean 1 shared static s_ConfigInitialized
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:1 000000001ab979c0:0 <<
000007fef31403c8 4001b94 974 System.Int32 1 shared static s_IdnScope
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0 000000001ab979c0:0 <<
000007fef3796cd8 4001b95 978 System.Boolean 1 shared static s_IriParsing
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0 000000001ab979c0:0 <<
000007fef37973f8 4001b96 cf0 System.Object 0 shared static s_initLock
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000022ec898 000000001ab979c0:0000000000000000 <<
000007fef3799400 4001b97 cf8 System.Char[] 0 shared static HexUpperChars
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000021233a0 000000001ab979c0:000000000257e298 <<
000007fef3799400 4001b98 d00 System.Char[] 0 shared static HexLowerChars
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:00000000021233d8 000000001ab979c0:000000000257e2d0 <<
000007fef3799400 4001b99 d08 System.Char[] 0 shared static _WSchars
>> Domain:Value 00000000017ea4f0:NotInit 000000001a9d6a70:0000000002123410 000000001ab979c0:000000000257e308 <<
excellent, now lets get the string value
0:018> !DumpObj 00000000025ee6a0
Name: System.String
MethodTable: 000007fef3797b08
EEClass: 000007fef339e550
Size: 126(0x7e) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: http://Romiko.com.au/test.jsp
Fields:
MT Field Offset Type VT Attr Value Name
000007fef379ed78 4000096 8 System.Int32 1 instance 51 m_arrayLength
000007fef379ed78 4000097 c System.Int32 1 instance 50 m_stringLength
000007fef3799550 4000098 10 System.Char 1 instance 68 m_firstChar
000007fef3797b08 4000099 20 System.String 0 shared static Empty
>> Domain:Value 00000000017ea4f0:0000000002101308 000000001a9d6a70:0000000002101308 000000001ab979c0:0000000002101308 <<
000007fef3799400 400009a 28 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 00000000017ea4f0:0000000002101af8 000000001a9d6a70:0000000002109b68 000000001ab979c0:0000000002572798 <<
YAY ! We can see from here that the policy search is calling the test.jsp page, also we notice this happening in the stack from within the Page_Init. Lets go and get this source code my disassembling the module 
We need to tell the customer the exact code causing this problem, the way to do that is to get the source code and look at the Page_Init for the policysearch class.
Lets go back to the stack and find out which dll this code is coming from
0:018> !clrstack
OS Thread Id: 0x121c (18)
Child-SP RetAddr Call Site
000000001e88dbc0 000007fef2b3e817 System.Threading.WaitHandle.WaitOne(Int64, Boolean)
000000001e88dc00 000007fef2b48641 System.Net.LazyAsyncResult.WaitForCompletion(Boolean)
000000001e88dc60 000007fef2b47f34 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)
000000001e88dcf0 000007fef2b474f8 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
000000001e88dd60 000007fef2b4b031 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
000000001e88ddc0 000007ff00705bbb System.Net.HttpWebRequest.GetResponse()
000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)
FOUND IT:
000000001e88de50 000007fef09da30a RomikoCommon.PageBase.OnPreInit(System.EventArgs)
what we need to do now is get this dll file, lets look at what is in the domain
0:018> !DumpDomain
————————————–
System Domain: 000007fef8933f20
LowFrequencyHeap: 000007fef8933f68
HighFrequencyHeap: 000007fef8933ff8
StubHeap: 000007fef8934088
Stage: OPEN
Name: None
————————————–
Shared Domain: 000007fef8934820
LowFrequencyHeap: 000007fef8934868
HighFrequencyHeap: 000007fef89348f8
StubHeap: 000007fef8934988
Stage: OPEN
Name: None
Assembly: 000000000172e0c0
Assembly: 000000001aa8f210
Assembly: 000000001ab529f0
Assembly: 000000001aa247e0
Assembly: 000000001aa9b220
Assembly: 000000001aac5970
Assembly: 000000000172e3c0
Assembly: 000000001aa8ef10
Assembly: 000000001aac5730
Assembly: 000000001ab52c30
Assembly: 000000001ca16590
Assembly: 00000000017e4f30
Assembly: 000000000172e540
Assembly: 000000001aa8f5d0
Assembly: 000000001ca03fd0
Assembly: 000000001aa8df50
Assembly: 00000000017e5230
Assembly: 000000001aa24ba0
Assembly: 00000000017e59b0
Assembly: 000000001aa25560
Assembly: 000000001ca16290
Assembly: 000000001aa253e0
Assembly: 000000001aa9a9e0
Assembly: 000000001aa8e0d0
Assembly: 000000001aa8f450
————————————–
Domain 1: 00000000017ea4f0
LowFrequencyHeap: 00000000017ea538
HighFrequencyHeap: 00000000017ea5c8
StubHeap: 00000000017ea658
Stage: OPEN
SecurityDescriptor: 00000000017c9d60
Name: DefaultDomain
Assembly: 000000000172e0c0 [C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 000000000172e180
SecurityDescriptor: 000000000172df40
Module Name
000007fef3361000 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
000007ff000e2568 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
000007ff000e2020 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp
I cut allot out, but lets look at this, we have our romikocommon.dll further down
Assembly: 00000000017e6c70 [C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\purchasepath\2d20f8c3\9af2cafc\assembly\dl3\455d0c2c\ff027527_32ffcb01\RomikoCommon.DLL]
ClassLoader: 00000000017e6d30
SecurityDescriptor: 00000000017e6bb0
Module Name
000007ff00176880 C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\purchasepath\2d20f8c3\9af2cafc\assembly\dl3\455d0c2c\ff027527_32ffcb01\CommonCommon.DLL
Nice, now, we going to dump this to a file
We will use the savemodule command, but we need the base address, so issue the LM command
0:018> LM
start end module name
00000000`1c170000 00000000`1c197000 System_EnterpriseServices_Wrapper_1c170000 (no symbols)
00000000`1c290000 00000000`1c2b6000 ICSharpCode_SharpZipLib (no symbols)
00000000`1cdf0000 00000000`1d0ef000 System_Data (export symbols) System.Data.dll
00000000`6b570000 00000000`6bb04000 AjaxControlToolkit (no symbols)
00000000`72a00000 00000000`72a18000 RomikoCommon_72a00000 C (no symbols)
00000000`72ee0000 00000000`72ef8000 App_Web_jweapfn4 C (no symbols)
00000000`72f00000 00000000`72f0a000 App_Web_xg2ybqom C (no symbols)
00000000`72f10000 00000000`72f18000 App_Web_1elshlii C (no symbols)
00000000`72f20000 00000000`72f38000 RomikoCommon C (no symbols)
perfect now we export the dll for .net reflector to use or ildasm, grabbing the base address above and dumping the contents as a dll.
0:018> !SaveModule 00000000`72a00000 e:\projects\debug\romikocommom.dll
remember in the stack we had this
RomikoCommon.PageBase.OnPreInit(System.EventArgs)
The savemodule has saved/dumped the dll, in reflector we can see the code causing the problem!
Here is the code
protected override void OnPreInit(EventArgs e) { string str = (this._deployOrDev.ToUpper() == "DEV") ? this._sessionId : base.Request.QueryString["sid"]; if (!string.IsNullOrEmpty(str)) { string str2 = this.Session["sid"] as string; if (!string.IsNullOrEmpty(str2) && (string.Compare(str2, str, true) != 0)) { this.Session.Clear(); } this.Session["sid"] = str; } else { str = this.Session["sid"] as string; } if (string.IsNullOrEmpty(str)) { throw new SecurityException("Session is empty."); } try { using (SecurityServicesProxy proxy = new SecurityServicesProxy()) { UserCredential = proxy.GetUserCredential(str, Encrypt.EncryptData(str)); if (UserCredential == null) { throw new SessionExpiredException("The session is invalid. Please login again."); } } } catch (Exception exception) { HandleException(base.GetType().Name, MethodBase.GetCurrentMethod().Name, "Session Indentifier:" + str, exception, new Action<string>(this.DisplayError)); } this.SetUserCredentialsForSession(); try { WebRequest request = WebRequest.Create(this._keepAliveUrl); request.Headers.Add(HttpRequestHeader.Cookie, "JSESSIONID=" + str); request.GetResponse(); } catch (Exception exception2) { _log.Info(exception2); throw new SessionExpiredException("Connecting Java system failed.", exception2); } }
WOW, look, every PAGE in their system is inheriting a custom pagebase, and this is called a external web page (remember test.jsp. It is doing this to keep the java session alive since the .net session is in an Iframe.
My load test always showed errors Connecting to Java system failed and here we can see why. The code should be changed, instead of all page loads calling java, rather have a background thread run and do it every 30 seconds whatever and we improve the performance of EVERY page.
we can also check that the request was not working and was waiting for the javasystem which crashed by checking cpu time.
0:018> kerneltime
Couldn’t resolve error at ‘erneltime’
0:018> .time
Debug session time: Fri May 6 15:31:14.000 2011 (UTC + 10:00)
System Uptime: 58 days 6:25:30.968
Process Uptime: 0 days 0:28:42.000
Kernel time: 0 days 0:00:01.000
User time: 0 days 0:00:05.000
it only using a fraction of a second, so it was definitely handing and waiting for a response, which of course would timeout at some point.
I hope you enjoyed this debugging session as much as I did, and thank you Tatham Oddie for geeking it up with me on this
Romiko Derbynew
Romiko Derbynew 
He Romiko,
Great post man! Absolutely no idea what you are talking about :). How is everything down there?
Rgds, Douwe.
Australia is great, good surf and good times 🙂 Still listening to Elke Klein 🙂
Hi Romiko, great article and great skill!
We have a crashing w3wp.exe and a full dump of it. Would you be able to assist?
Alex
Hi Alex,
I am definitely able to assist you. In which Time Zone are you located, and what is the scenario regarding the issues you are having?