Tag: .Net

Demystifying .NET Core Memory Leaks: A Debugging Adventure with dotnet-dump

It has been a while since I wrote about memory dump analysis; the last post on the subject was back in 2011. Lets get stuck into the dark arts.

First and foremost, .NET is very different to .NET Core, down to App Domains and how the MSIL is executed. Understanding this is crucial before you kick off a clrstack! or dumpdomain! Make sure you understand the architecture of what you debugging from ASP to console apps. Dumpdomain caught me off guard, as you would use dumpdomain to get the sourcecode and decompile via the PDB files in the past.

Feature	ASP.NET Core	ASP.NET Framework
Cross-Platform Support	Runs on Windows, Linux, and macOS.	Primarily runs on Windows.
Hosting	Can be hosted on Kestrel, IIS, HTTP.sys, Nginx, Apache, and Docker.	Typically hosted on IIS.
Performance	Optimized for high performance and scalability.	Good performance, but generally not as optimized as ASP.NET Core.
Application Model	Unified model for MVC and Web API.	Separate models for MVC and Web API.
Configuration	Uses a lightweight, file-based configuration system (appsettings.json).	Uses web.config for configuration.
Dependency Injection	Built-in support for dependency injection.	Requires third-party libraries for dependency injection.
App Domains	Uses a single app model and does not support app domains.	Supports app domains for isolation between applications.
Runtime Compilation	Supports runtime compilation of Razor views (optional).	Supports runtime compilation of ASPX pages.
Modular HTTP Pipeline	Highly modular and configurable HTTP request pipeline.	Fixed HTTP request pipeline defined by the Global.asax and web.config.
Package Management	Uses NuGet for package management, with an emphasis on minimal dependencies.	Also uses NuGet but tends to have more complex dependency trees.
Framework Versions	Applications target a specific version of .NET Core, which is bundled with the app.	Applications target a version of the .NET Framework installed on the server.
Update Frequency	Rapid release cycle with frequent updates and new features.	Slower release cycle, tied to Windows updates.
Side-by-Side Deployment	Supports running multiple versions of the app or .NET Core side-by-side.	Does not support running multiple versions of the framework side-by-side for the same application.
Open Source	Entire platform is open-source.	Only a portion of the platform is open-source.

So we embark on a quest to uncover hidden memory leaks that lurk within the depths of .NET Core apps, armed with the mighty dotnet-dump utility. This tale of debugging prowess will guide you through collecting and analyzing dump files, uncovering the secrets of memory leaks, and ultimately conquering these elusive beasts.

Preparing for the Hunt: Installing dotnet-dump

Our journey begins with the acquisition of the dotnet-dump tool, a valiant ally in our quest. This tool is a part of the .NET diagnostics toolkit, designed to collect and analyze dumps without requiring native debuggers. It’s a lifesaver on platforms like Alpine Linux, where traditional tools shy away.

To invite dotnet-dump into your arsenal, you have two paths:

The Global Tool Approach: Unleash the command dotnet tool install --global dotnet-dump into your terminal and watch as the latest version of the dotnet-dump NuGet package is summoned.
The Direct Download: Navigate to the mystical lands of the .NET website and download the tool executable that matches your platform’s essence.

The First Step: Collecting the Memory Dump

With dotnet-dump by your side, it’s time to collect a memory dump from the process that has been bewitched by the memory leak. Invoke dotnet-dump collect --process-id <PID>, where <PID> is the identifier of the cursed process. This incantation captures the essence of the process’s memory, storing it in a file for later analysis.

The Analytical Ritual: Unveiling the Mysteries of the Dump

Now, the real magic begins. Use dotnet-dump analyze <dump_path> to enter an interactive realm where the secrets of the dump file are yours to discover. This enchanted shell accepts various SOS commands, granting you the power to scrutinize the managed heap, reveal the relationships between objects, and formulate theories about the source of the memory leak.

Common Spells and Incantations:

clrstack: Summons a stack trace of managed code, revealing the paths through which the code ventured.
dumpheap -stat: Unveils the statistics of the objects residing in the managed heap, highlighting the most common culprits.
gcroot <address>: Traces the lineage of an object back to its roots, uncovering why it remains in memory.

The Final Confrontation: Identifying the Memory Leak

Armed with knowledge and insight from the dotnet-dump analysis, you’re now ready to face the memory leak head-on. By examining the relationships between objects and understanding their roots, you can pinpoint the source of the leak in your code.

Remember, the key to vanquishing memory leaks is patience and perseverance. With dotnet-dump as your guide, you’re well-equipped to navigate the complexities of .NET Core memory management and emerge victorious.

Examine managed memory usage

Before you start collecting diagnostic data to help root cause this scenario, make sure you’re actually seeing a memory leak (growth in memory usage). You can use the dotnet-counters tool to confirm that.

Open a console window and navigate to the directory where you downloaded and unzipped the sample debug target. Run the target:

dotnet run

From a separate console, find the process ID:

dotnet-counters ps

The output should be similar to:

4807 DiagnosticScena /home/user/git/samples/core/diagnostics/DiagnosticScenarios/bin/Debug/netcoreapp3.0/DiagnosticScenarios

Now, check managed memory usage with the dotnet-counters tool. The --refresh-interval specifies the number of seconds between refreshes:

dotnet-counters monitor --refresh-interval 1 -p 4807

The live output should be similar to:

Press p to pause, r to resume, q to quit.
    Status: Running

[System.Runtime]
    # of Assemblies Loaded                           118
    % Time in GC (since last GC)                       0
    Allocation Rate (Bytes / sec)                 37,896
    CPU Usage (%)                                      0
    Exceptions / sec                                   0
    GC Heap Size (MB)                                  4
    Gen 0 GC / sec                                     0
    Gen 0 Size (B)                                     0
    Gen 1 GC / sec                                     0
    Gen 1 Size (B)                                     0
    Gen 2 GC / sec                                     0
    Gen 2 Size (B)                                     0
    LOH Size (B)                                       0
    Monitor Lock Contention Count / sec                0
    Number of Active Timers                            1
    ThreadPool Completed Work Items / sec             10
    ThreadPool Queue Length                            0
    ThreadPool Threads Count                           1
    Working Set (MB)                                  83

Focusing on this line:

    GC Heap Size (MB)                                  4

You can see that the managed heap memory is 4 MB right after startup.

Now, go to the URL https://localhost:5001/api/diagscenario/memleak/20000.

Observe that the memory usage has grown to 30 MB.

GC Heap Size (MB)                                 30


By watching the memory usage, you can safely say that memory is growing or leaking. The next step is to collect the right data for memory analysis.

Generate memory dump

When analyzing possible memory leaks, you need access to the app’s memory heap to analyze the memory contents. Looking at relationships between objects, you create theories as to why memory isn’t being freed. A common diagnostic data source is a memory dump on Windows or the equivalent core dump on Linux. To generate a dump of a .NET application, you can use the dotnet-dump tool.

Using the sample debug target previously started, run the following command to generate a Linux core dump:

dotnet-dump collect -p 4807

The result is a core dump located in the same folder.

Writing minidump with heap to ./core_20190430_185145
Complete

For a comparison over time, let the original process continue running after collecting the first dump and collect a second dump the same way. You would then have two dumps over a period of time that you can compare to see where the memory usage is growing.

Restart the failed process

Once the dump is collected, you should have sufficient information to diagnose the failed process. If the failed process is running on a production server, now it’s the ideal time for short-term remediation by restarting the process.

In this tutorial, you’re now done with the Sample debug target and you can close it. Navigate to the terminal that started the server, and press Ctrl+C.

Analyze the core dump

Now that you have a core dump generated, use the dotnet-dump tool to analyze the dump:

dotnet-dump analyze core_20190430_185145

Where core_20190430_185145 is the name of the core dump you want to analyze.

If you see an error complaining that libdl.so cannot be found, you may have to install the libc6-dev package. For more information, see Prerequisites for .NET on Linux.

You’ll be presented with a prompt where you can enter SOS commands. Commonly, the first thing you want to look at is the overall state of the managed heap:

> dumpheap -stat

Statistics:
              MT    Count    TotalSize Class Name
...
00007f6c1eeefba8      576        59904 System.Reflection.RuntimeMethodInfo
00007f6c1dc021c8     1749        95696 System.SByte[]
00000000008c9db0     3847       116080      Free
00007f6c1e784a18      175       128640 System.Char[]
00007f6c1dbf5510      217       133504 System.Object[]
00007f6c1dc014c0      467       416464 System.Byte[]
00007f6c21625038        6      4063376 testwebapi.Controllers.Customer[]
00007f6c20a67498   200000      4800000 testwebapi.Controllers.Customer
00007f6c1dc00f90   206770     19494060 System.String
Total 428516 objects

Here you can see that most objects are either String or Customer objects.

You can use the dumpheap command again with the method table (MT) to get a list of all the String instances:

> dumpheap -mt 00007f6c1dc00f90

         Address               MT     Size
...
00007f6ad09421f8 00007faddaa50f90       94
...
00007f6ad0965b20 00007f6c1dc00f90       80
00007f6ad0965c10 00007f6c1dc00f90       80
00007f6ad0965d00 00007f6c1dc00f90       80
00007f6ad0965df0 00007f6c1dc00f90       80
00007f6ad0965ee0 00007f6c1dc00f90       80

Statistics:
              MT    Count    TotalSize Class Name
00007f6c1dc00f90   206770     19494060 System.String
Total 206770 objects

You can now use the gcroot command on a System.String instance to see how and why the object is rooted:

> gcroot 00007f6ad09421f8

Thread 3f68:
    00007F6795BB58A0 00007F6C1D7D0745 System.Diagnostics.Tracing.CounterGroup.PollForValues() [/_/src/System.Private.CoreLib/shared/System/Diagnostics/Tracing/CounterGroup.cs @ 260]
        rbx:  (interior)
            ->  00007F6BDFFFF038 System.Object[]
            ->  00007F69D0033570 testwebapi.Controllers.Processor
            ->  00007F69D0033588 testwebapi.Controllers.CustomerCache
            ->  00007F69D00335A0 System.Collections.Generic.List`1[[testwebapi.Controllers.Customer, DiagnosticScenarios]]
            ->  00007F6C000148A0 testwebapi.Controllers.Customer[]
            ->  00007F6AD0942258 testwebapi.Controllers.Customer
            ->  00007F6AD09421F8 System.String

HandleTable:
    00007F6C98BB15F8 (pinned handle)
    -> 00007F6BDFFFF038 System.Object[]
    -> 00007F69D0033570 testwebapi.Controllers.Processor
    -> 00007F69D0033588 testwebapi.Controllers.CustomerCache
    -> 00007F69D00335A0 System.Collections.Generic.List`1[[testwebapi.Controllers.Customer, DiagnosticScenarios]]
    -> 00007F6C000148A0 testwebapi.Controllers.Customer[]
    -> 00007F6AD0942258 testwebapi.Controllers.Customer
    -> 00007F6AD09421F8 System.String

Found 2 roots.

You can see that the String is directly held by the Customer object and indirectly held by a CustomerCache object.

You can continue dumping out objects to see that most String objects follow a similar pattern. At this point, the investigation provided sufficient information to identify the root cause in your code.

This general procedure allows you to identify the source of major memory leaks.

Epilogue: Cleaning Up After the Battle

With the memory leak defeated and peace restored to your application, take a moment to clean up the battlefield. Dispose of the dump files that served you well, and consider restarting your application to ensure it runs free of the burdens of the past.

Embark on this journey with confidence, for with dotnet-dump and the wisdom contained within this guide, you are more than capable of uncovering and addressing the memory leaks that challenge the stability and performance of your .NET Core applications. Happy debugging!

Sources:
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak

NServiceBus – Some Benefits

Hi,

I am not sure why, but in many organisation, there is allot of unnecessary complexity when looking at source code. From Programmers using try/catch blocks all over the show to unnecessary synchronous service calls with multiple threads, causing deadlocks on SQL Server.

I would like to give a simple example, consider this code

static ProcessOrder()
{
    const int workerCount = 4;
    WorkQueue = new ConcurrentQueue();
    Workers = new List();

    for (var index = 0; index; workerCount; index++)
    {
        Worker = new BackgroundWorker();
        Workers.Add(Worker);
        Worker.DoWork += Worker_DoWork;
        Worker.RunWorkerAsync();
    }
}

The above code is extremely dangerous. You have no control over the load it will cause to backend calls, especially when it is calling Stored Procs etc. Secondly, it is not durable.

The above code, can easily be replaced by a NServiceBus Saga or handler, in the above, a Saga is appropriate, as this message is a Root Aggregate, we have an order to process. Sagas will provide an environment to preserve state and all the threading and correlation is all handled for you.

partial void HandleImplementation(ProcessOrder message)
{
    Logger.Info("ProcessOrder received, Saga Instance: {0} for OrderId: {1}", Data.Id, Data.OrderId);
    RunOnlyOncePerInstance(message);         
    AlwaysRunForEveryMessage(message); //counts and extracts new order items.

    if (Data.OrderCount == 1)
    {
        SendSaveOrder(message);
        return;
    }

    if (Data.InitialDelayPassed) //Expired workflow
        SendUpdateOrderItem();
}

From the above, you can see state is preserved for each unique OrderId. This Saga processes multiple order items for an OrderId. Which can be submitted at any time during the day. We do not have to worry about multiple threads, correlation is dealt with automatically, well we set the correlation id to the OrderId, so order items coming in can be matched to the correct Saga instance.

We can now get rid of the unreliable issues with in memory worker threads, and uncontrolled pounding of the database.

By using a MSMQ infrastructure and leveraging NServiceBus, these sort of issues, you find in code, can be easily mitigated.

Heroku Neo4j, App Harbor MVC4, Neo4jClient & Ruby Proxy

Overview

We going to outlay the process to deploy a 4 layers architecture using the above technologies.

Prerequisites

Assumed you created a Heroku app and installed neo4j addon, if not go to blog post:https://romikoderbynew.com/2012/01/25/getting-started-with-heroku-toolbelt-on-windows-2/
Download Proxy Ruby Gem fromhttps://github.com/Romiko/RubyRestProxy

Sinatra and Rest-Client

Sinatra and the Rest-Client libraries are used in the Ruby Gem to proxy all Http requests to Neo4j.

Install Process

Install the Heroku client, run this from the heroku\Ruby\bin folder e.g. D:\Program Files (x86)\Heroku\ruby\bin

gem install heroku

We then need to use the Cedar stack (http://devcenter.heroku.com/articles/cedar) which supports ruby in Heroku, so run:

heroku create –stack cedar

(Ensure git.exe is in your environment variables)

Once this is completed we then deploy our code to our Heroku git repository. You can see your git heroku repository in the app details for the app you created in Heroku:

We will use a public key that we created when first setting up Heroku, check blog post

https://romikoderbynew.com/2012/01/25/getting-started-with-heroku-toolbelt-on-windows-2/

if you changed your public key, then add it to Heroku

heroku keys:add

Ok, now we push the Gem files to our Heroku git repository. Your repsotory will be different to mine!

git push git@heroku.com:neo4jorbust.git master

Lets check the state of the application

E:\Projects\RubyRestProxy>heroku ps –app neo4jorbust
Process State Command
——- ———– ————————————
web.1 idle for 9h thin -p $PORT -e $RACK_ENV -R $HER..

Test HttpClient –> Ruby –> Neo4j Communication

Nice, now we got this GEM file, which is our Ruby http proxy code for gremlin queries. Lets test this baby out with fiddler and run gremlin query to the web app, which will then call neo4j!

Lets browse the Ruby Application on the Heroku server, so we can find out what URL we need to use to send Gremlin queries to Neo4j.

Beautiful, all we need to do now is send an HTTP POST to the added, you can use curl, but here is fiddler.

And the result, music to our eyes

ASP.NET MVC 4 APP with Neo4jClient

This application is hosted at

http://frictionfree.org

Source Code:

https://bitbucket.org/romiko/frictionfree/src

Now we going to fire up a .Net application that will use the Neo4jClient Rest Api to communicate with neo4j in Heroku. You can find out more about Neo4jClient here http://hg.readify.net/neo4jclient/wiki/Home

Clone the git repository to get the source code for our sample Neo4jClient .Net application. This is a MVC4 application and runs on the .Net Framework 4 runtime, so please ensure you have this installed on your windows machine.

About frictionfree.org

The application is used to find other people with similar interests to you and the opportunity to get their details and contact them to perhaps share your love of flying, surfing or whatever it is you like to do!

Bitbucket and AppHarbor integration

If you have your own ASP.NET application, you can store it in bitbucket and then configure bitbucket to auto deploy to AppHarbor when pushing back the repostory.

In the admin settings for your repository just add your appharbor token.

Neo4jClient gremlin Query syntax

Check the source code of the MVC application in the Logic library to see how to run queries to Neo4j. Here is sample code to get administrators of the system

private void CreateAdminAccountIfNoneExist() { var anyAdmins = graphClient .RootNode .In(Administers.TypeKey) .Any(); if (!anyAdmins) { provisionNewUserService.CreateAdmin(); } }

Getting Started with Heroku ToolBelt/Neo4j on Windows

Hi Guys,

To get started with Heroku, you need a public key. If you do not have one in the

C:\Users\<YourUsername>\.ssh folder, then you get this error

D:/Program Files (x86)/Heroku/lib/heroku/auth.rb:195:in `read’: No such file or
directory – C:/Users/Romiko/.ssh/id_rsa.pub (Errno::ENOENT)
from D:/Program Files (x86)/Heroku/lib/heroku/auth.rb:195:in `associate_key’

looking at the source code, line 195 is ssh key generation. So what we need to do is use a utility like GitBash to do this for us:

Instructions

TO get up and running with the Heroku Toolbelt on windows ensure you read:

http://devcenter.heroku.com/articles/quickstart

Once you have read that and installed Heroku toolbelt

http://assets.heroku.com/heroku-toolbelt/heroku-toolbelt.exe

then install Git (Or use the Git bash if already installed)

http://msysgit.googlecode.com/files/Git-1.7.8-preview20111206.exe

Now, what you do is follow all the instructions here, you might as well ensure you have a GitHub account, its nice to put your code in a repository if developing:

http://help.github.com/win-set-up-git/

You should then have these files (you might not have known hosts), if you did not connect into git

The Heroku batch file will look for the id_rsa.pub file and use it to connect to the cloud servers.

Once this is done, you can then go back to the heroku.cmd execution and login

D:\Program Files (x86)\Heroku>heroku login
Enter your Heroku credentials.
Email: ROMxxx@xxx.COM
Password:
Found existing public key: C:/Users/Romiko/.ssh/id_rsa.pub
Uploading ssh public key C:/Users/Romiko/.ssh/id_rsa.pub

That is it, so hope this gets you started on Heroku Smile

Neo4j

Once you ran the above command, then create an application

Create Heroku Applicaton in the Cloud

D:\Program Files (x86)\Heroku>heroku apps:create Neo4jOrBust
Creating neo4jorbust… done, stack is bamboo-mri-1.9.2
http://neo4jorbust.heroku.com/ | git@heroku.com:neo4jorbust.git

Notice the appname is lowercase always!

Install Neo4j Addon

For this to work, you must verify the account and you will need a credit card, it will NOT be billed. Don’t sue me if it does!

Then for Neo4j, just install the addon like so

D:\Program Files (x86)\Heroku>heroku addons:add neo4j –app neo4jorbust
—–> Adding neo4j to neo4jorbust… failed
!    Please verify your account to install this add-on
!    For more information, see http://devcenter.heroku.com/categories/billing
!    Confirm now at https://heroku.com/confirm

Go verify your account and then come back and run the command again

https://api.heroku.com/myapps

D:\Program Files (x86)\Heroku>heroku addons:add neo4j –app neo4jorbust
—–> Adding neo4j to neo4jorbust… done, (free)

Thats it!

You can then go check your addons on the web site at heroku under resources. Click databases and scroll to bottom.

Notice neo4jTest and it is free Smile Lets enable it

Notice, the urls for the webadmin site and rest url, you all set to work with Neo4j now Smile

Thats it, have fun!