In the development of the Neo4jClient, we noticed that all DB queries to Neo4j were taking roughly 150ms longer than on my local development machine. after using Fiddler and ApacheBench, it was clear the performance issue was inside the .Net Code.

What we use in Neo4jClient (http://hg.readify.net/neo4jclient/wiki/Home) is the RestSharp open source library. Fiddler proved to be very unreliable to produce consistent results due to it caching and reuse of connections, even when disabling them, also, when requests went via fiddler, the response time was quick, so queries taking over 150ms without fiddler were taking 10-15ms within fiddler.

Profiling

The solution was to build a custom .Net tool to send Http POSTS, and using the JetBrains DotTrace utility, we were able to see an issue where, if we FIRST bypassed RestSharp and just used HttpWebRequest, the response time was quick, as soon as we used RestSharp it went slow, and even after using it, going back to HttpWebeRequest was then slow.

Somewhere, a Static Class was causing an issue that RestSharp uses.

Notice the 150ms overhead in the above screenshot

Here is a profile where restsharp is bypassed

In the above profile, the http response time is fast.

ServicePointManager

RestSharp sets the ServicePointManager, which is a static class, and what we found was that this algorithm was causing 150ms delays in our Neo4j DB calls.

ServicePointManager.UseNagleAlgorithm

I noticed in wire shark the difference between a fast stream and a slow one, it affected the number of packets which pointed to either

ServicePointManager.Expect100Continue

or

ServicePointManager.UseNagleAlgorithm

we found changing Expect100Continue not to cause any changes, however setting UseNagleAlgorithm to false, solved the issue, and now all DB calls to Neo4j are 150ms quicker!

 

(HttpWebRequest) – SLOW – After a RestSharp call (Poisoned system)	FAST TCP STREAM (HTTPWebRequest) This is due to UseNagleAlgorithm being off
POST /db/data/ext/GremlinPlugin/graphdb/execute_script HTTP/1.1 Content-Type: application/json Host: 10.25.234.67:20001 Content-Length: 65 {"script":"g.v(793).in(‘USER_BELONGS_TO’).drop(0).take(100)._()"}HTTP/1.1 200 OK Content-Length: 5316 Content-Encoding: UTF-8 Content-Type: application/json Access-Control-Allow-Origin: * Server: Jetty(6.1.25)	POST /db/data/ext/GremlinPlugin/graphdb/execute_script HTTP/1.1 Content-Type: application/json Host: 10.25.234.67:20001 Content-Length: 65 Expect: 100-continue HTTP/1.1 100 Continue {"script":"g.v(793).in(‘USER_BELONGS_TO’).drop(0).take(100)._()"}HTTP/1.1 200 OK Content-Length: 5316 Content-Encoding: UTF-8 Content-Type: application/json Access-Control-Allow-Origin: * Server: Jetty(6.1.25)

Before Optimisation in Azure

After Optimisation in Azure

Conclusion

If you doing allot of small HTTP POSTS, it might be a good idea to turn off the Nagle Algorithm.

http://www.winsocketdotnetworkprogramming.com/xmlwebservicesaspnetworkprogramming11e.html

When using the Neo4j REST API(http://docs.neo4j.org/chunked/snapshot/rest-api.html), be aware of the response times, and if you notice response times greater than 150ms for all DB calls, then perhaps you have the same issue with the ServicePointManager confguration.

Nagle Algorithm

The Nagle algorithm increases network efficiency by decreasing the number of packets sent across the network. It accomplishes this by instituting a delay on the client of up to 200 milliseconds when small amounts of data are written to the network. The delay is a wait period for additional data that might be written. New data is added to the same packet. The practice of compiling data is not always ideal for Web services, however, because a POST request often contains a small amount of data in the HTTP headers without the body (because of 100 Continue, as described earlier). The Nagle algorithm can be turned off for Web services calls via the System.Net.ServicePointManager.UseNagleAlgorithm property. Yet disabling the Nagle algorithm might not provide the best performance either, you’ll need to experiment with disabling the algorithm to fine- tune your application. Whether you should disable the algorithm in this case depends on several factors, including the size of the Web services call, the latency of the network, and whether HTTP authentication is involved.