Reuse HttpClient

[gtoombs-microsaas]

Currently, HtmlWeb does not reuse a client, and spins one up on every call:

      HttpClient httpClient = new HttpClient((HttpMessageHandler) httpClientHandler);

This is in contravention of the recommendations from .NET:

HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors.

[JonathanMagnan]

Hello @gtoombs-microsaas ,

Thank you for reporting, you are right.

However, there is also some customization in the HttpClient such as https://github.com/zzzprojects/html-agility-pack/blob/def26b2d2f5ea8e6a077d1dd8ec145004a6cf424/src/HtmlAgilityPack.Shared/HtmlWeb.cs#L1833 and the handler which currently make it impossible to re-use the same HttpClient in some scenario.

However, we will look at if we can re-use it when none of those setting are configured which is pretty much the case probably 99% of the time.

Best Regards,

Jon

[gtoombs-microsaas]

@JonathanMagnan Good news! The specific line you referenced:

client.DefaultRequestHeaders.Add("User-Agent", UserAgent);

is fairly easy to convert to a per-request setting of headers. There are many convenient ways to do this, including holding those headers in a member variable and applying them en-masse to a request. Said another way: even for the 1% of requests that require customisation, it should still not be necessary to have a separate Client instance if the customisation is only for HTTP headers.

[JonathanMagnan]

Hello @gtoombs-microsaas ,

The v1.11.40 has been released.

The same HttpClient (per UserAgent) should now be re-used whenever is possible.

Let me know if everything works as expected.

Best Regards,

Jon

[gtoombs-microsaas]

Amazing, thanks! I'll check it out.