Due to a combination of design errors, bugs, and incorrect documentation, it is surprisingly hard to use .NET's HttpClient correctly. As a result, applications that appear to be working correctly in production can suffer from performance issues and runtime failures under load.
This fact was revealed in an article titled You're using HttpClient wrong and it is destabilizing your software by Simon Timms of ASP.NET Monsters.
Responses to the article vary, but mostly reflect disappointment and frustration:
... am I the only one that gets angry when I read stuff like this? I mean what would happen to any of us if we released code that works like that? We'd be pilloried, of course. But when it's part of the core, we just accept it and make workarounds and write the same articles over and over and over.
That seriously screws with the principle of least astonishment.
I'd say that this makes HttpClient either buggy or badly architected. Can't decide which one. It would be funny if it is the second and it needs to be replaced with yet another way to do http requests.
-- Eirenarch
How C# Developers Are Trained
To understand why we got into this situation, we'll first look at another connection-orientated class, SqlConnection. When first taught how to use IDisposable and the using statement, the vast majority of developers are given examples such as:
using (var con = new SqlConnection(connectionString)) {
con.open();
//use the connection here
} //this closes the connection
While the explanation for this example is incomplete, the pattern is correct and has served developers well over the years. However, if you try to apply this pattern to HttpClient, another IDisposable class, you trip over some rather unexpected problems.
Specifically, it is going to open a lot more sockets than you actually need, putting a lot of load on the server. Furthermore, these sockets won't actually be closed by the using statement. Instead they are closed several minutes after the application has ceased to use them.
Connection Pooling
Going back to the SqlConnection example, most connection-orientated resources are pooled. When you "open" a database connection, it first checks the pool for an available, unused connection. If it finds one, it will reuse it instead of creating a new connection.
Likewise, when you "close" a SqlConnection it simply releases the connection back to the pool. Eventually a separate process may close long unused connections, but in general you can count of it to do the correct thing in terms of balancing performance and server load.
HttpClient doesn't work that way. When you dispose it, it starts the process of closing the socket(s) that it controls. Which means you have to go through an entirely new connection cycle the next time you make a request. This can be especially painful if your network has a high latency or your connection is secured, as the latter requires new round of SSL/TLS negotiation.
Closing a Socket Takes Four Minutes
As mentioned above, closing a socket isn't a fast process. When you "close" the socket, what you are really doing is placing it in the TIME_WAIT state. Windows will leave the socket in this state for a configurable amount of time, four minutes by default, just in case any remaining packets are still in transit.
This makes it much more likely that you'll exhaust the number of available sockets, leading to runtime errors such as "Unable to connect to the remote server. System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted".
Simon Timms writes, "Searching for that in the Googles will give you some terrible advice about decreasing the connection timeout. In fact, decreasing the timeout can lead to other detrimental consequences when applications that properly use HttpClient or similar constructs are run on the server. We need to understand what “properly” means and fix the underlying problem instead of tinkering with machine level variables".
The NET Core Performance Hit
Most developers working exclusively with the full version of the .NET Framework don't notice these problems. However, those using .NET Core have an additional issue that makes the overall problem much more apparent.
Between RC1 and RC2 of .NET Core, a bug was introduced that causes a 1,010 to 1,030 ms delay when calling HttpClient.Dispose. This delay isn't expected to be fixed until version 1.2 of .NET Core.
Broker Classes as a Solution
Though not mentioned anywhere in the HttpClient documentation, there is a pattern described in the Microsoft Patterns & Practices GitHub site. They refer to HttpClient as a "Broker Class" and describe it as such:
These broker classes can be expensive to create. Instead, they are intended to be instantiated once and reused throughout the life of an application. However, it is common to misunderstand how these classes are intended to be used, and instead treat them as resources that should be acquired only as necessary and released quickly […]
Rather than creating and disposing of the HttpClient as necessary, Microsoft P&P recommends that you create one instance, store it in a static field, and share it for the lifetime of the application.
Misleading Documentation
This brings us back to the problem of misleading documentation. Though it is essentially boilerplate, the v118 of the offical documentation (which it currently returned by Google and Bing searches) says that sharing an HttpClient across threads isn't supported.
Any public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe.
And that's pretty much it. Unless of course you look at v110 of the official documentation, which has this helpful statement:
HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors. Below is an example using HttpClient correctly.
It goes on to say these methods are thread safe.
- CancelPendingRequests
- DeleteAsync
- GetAsync
- GetByteArrayAsync
- GetStreamAsync
- GetStringAsync
- PostAsync
- PutAsync
- SendAsync
This seems to be an ongoing problem with MSDN documentation. To get the full story of any class, you can have to check each version of the documentation for important passages that were added or removed.
DNS Bugs
If we follow the advice given thus far, there are other problems that can arise. Ali Kheyrollahi writes,
But it turns out there is a serious issue: DNS changes are NOT honoured and HttpClient (through HttpClientHandler) hogs the connections until socket is closed. Indefinitely. So when does DNS change occur? Everytime you do blue-green deployment (in Azure cloud services when you deploy to staging slot and then swap production/staging slots). Everytime you change settings in your Azure Traffic Manager. Failover scenarios. Internally in a myriad of PaaS offerings.
And this has been going on for more than two years without being reported... makes me wonder what kind of applications we build with .NET?
Now if the reason for DNS change is failover, your connection would have been faulted anyway so this time connection would open against the new server. But if this were the blue-black deployment, you swap the staging and production and your calls would still go to the staging environment - a behaviour we had seen but had fixed it by bouncing the dependent servers thinking possibly this was an Azure oddity. What a fool was I - it was there in the code! Whose code? Well debateable...
Fixing this isn't insurmountable. Theoretically the HttpClient could honor the DNS TTL (Time to Live) value, which defaults to one hour. Each time it expires, the HttpClient could verify that the DNS entry is still valid, creating a new connection to the updated IP address if necessary.
But since that probably isn't going to happen, Kheyrollahi gives us a simpler work-around. By leveraging the ServicePointManager, you can tell HttpClient to automatically recycle connections.
var sp = ServicePointManager.FindServicePoint(new Uri("http://foo.bar"));
sp.ConnectionLeaseTimeout = 60*1000; // 1 minute
So this is something that you would want to do only at the startup of your application, once and for all endpoints your application is going to hit (if endpoints decided at runtime, you would be setting that at the time of discovery). Bear in mind, path and query strings are ignored and only the host, port and schema are important. Depending on your scenario, values of 1-5 minutes probably make sense.
