Slow HttpWebRequest.GetResponse()

In the development of the Neo4jClient, we noticed that all DB queries to Neo4j were taking roughly 150ms longer than on my local development machine. after using Fiddler and ApacheBench, it was clear the performance issue was inside the .Net Code.

What we use in Neo4jClient (http://hg.readify.net/neo4jclient/wiki/Home) is the RestSharp open source library. Fiddler proved to be very unreliable to produce consistent results due to it caching and reuse of connections, even when disabling them, also, when requests went via fiddler, the response time was quick, so queries taking over 150ms without fiddler were taking 10-15ms within fiddler.

Profiling

The solution was to build a custom .Net tool to send Http POSTS, and using the JetBrains DotTrace utility, we were able to see an issue where, if we FIRST bypassed RestSharp and just used HttpWebRequest, the response time was quick, as soon as we used RestSharp it went slow, and even after using it, going back to HttpWebeRequest was then slow.

Somewhere, a Static Class was causing an issue that RestSharp uses.

image

Notice the 150ms overhead in the above screenshot

Here is a profile where restsharp is bypassed

image

In the above profile, the http response time is fast.

ServicePointManager

RestSharp sets the ServicePointManager, which is a static class, and what we found was that this algorithm was causing 150ms delays in our Neo4j DB calls.

ServicePointManager.UseNagleAlgorithm

I noticed in wire shark the difference between a fast stream and a slow one, it affected the number of packets which pointed to either

ServicePointManager.Expect100Continue

or

ServicePointManager.UseNagleAlgorithm

we found changing Expect100Continue not to cause any changes, however setting UseNagleAlgorithm to false, solved the issue, and now all DB calls to Neo4j are 150ms quicker!

 

(HttpWebRequest) – SLOW – After a RestSharp call (Poisoned system)

FAST TCP STREAM (HTTPWebRequest)

This is due to UseNagleAlgorithm being off

POST /db/data/ext/GremlinPlugin/graphdb/execute_script HTTP/1.1

Content-Type: application/json

Host: 10.25.234.67:20001

Content-Length: 65

{"script":"g.v(793).in(‘USER_BELONGS_TO’).drop(0).take(100)._()"}HTTP/1.1 200 OK

Content-Length: 5316

Content-Encoding: UTF-8

Content-Type: application/json

Access-Control-Allow-Origin: *

Server: Jetty(6.1.25)

POST /db/data/ext/GremlinPlugin/graphdb/execute_script HTTP/1.1

Content-Type: application/json

Host: 10.25.234.67:20001

Content-Length: 65

Expect: 100-continue

HTTP/1.1 100 Continue

{"script":"g.v(793).in(‘USER_BELONGS_TO’).drop(0).take(100)._()"}HTTP/1.1 200 OK

Content-Length: 5316

Content-Encoding: UTF-8

Content-Type: application/json

Access-Control-Allow-Origin: *

Server: Jetty(6.1.25)

Before Optimisation in Azure

image

After Optimisation in Azure

image

Conclusion

If you doing allot of small HTTP POSTS, it might be a good idea to turn off the Nagle Algorithm.

http://www.winsocketdotnetworkprogramming.com/xmlwebservicesaspnetworkprogramming11e.html

When using the Neo4j REST API(http://docs.neo4j.org/chunked/snapshot/rest-api.html), be aware of the response times, and if you notice response times greater than 150ms for all DB calls, then perhaps you have the same issue with the ServicePointManager confguration.

Nagle Algorithm

The Nagle algorithm increases network efficiency by decreasing the number of packets sent across the network. It accomplishes this by instituting a delay on the client of up to 200 milliseconds when small amounts of data are written to the network. The delay is a wait period for additional data that might be written. New data is added to the same packet. The practice of compiling data is not always ideal for Web services, however, because a POST request often contains a small amount of data in the HTTP headers without the body (because of 100 Continue, as described earlier). The Nagle algorithm can be turned off for Web services calls via the System.Net.ServicePointManager.UseNagleAlgorithm property. Yet disabling the Nagle algorithm might not provide the best performance either, you’ll need to experiment with disabling the algorithm to fine- tune your application. Whether you should disable the algorithm in this case depends on several factors, including the size of the Web services call, the latency of the network, and whether HTTP authentication is involved.

Advertisements

5 thoughts on “Slow HttpWebRequest.GetResponse()

  1. Hey Romiko,

    really enjoy reading your blog posts.

    Just wanted to say thanks for taking care of this in the library and for letting the RestSharp group know about your findings.

  2. Pingback: HttpWebRequest GetResponse Yavas

  3. Pingback: OnBase Performance - Blog

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s