HttpClient 多執行緒執行出錯案例

2023-11-23 04:59 PM

依先前所學，System.Net.HttpClient 的 GetAsync()/PostAsync()/SendAsync() 等方法為 Thread-Safe，建議做法是只建立一份 HttpClient，以 static 方式共用。

今天踩到雷，以下為重現問題的程式範例。網頁使用 Windows 登入，故傳入 HttpClientHandler 指定 UseDefaultCredentials = true，準備 FormUrlEncodedContent 後，以 Parallel.ForEachAsync() 同步呼叫 HttpClient.PostAsync()：

using System.Diagnostics;
using System.Collections.Concurrent;
var results = new ConcurrentQueue<string>();

var httpClient = new HttpClient(new HttpClientHandler()
{
    UseDefaultCredentials = true
});
var postContent = new FormUrlEncodedContent(new Dictionary<string, string>()
{
    ["key"] = "1234"
});
var paramData = "1,2,3,4,5,6,7,8,9".Split(',');

await Parallel.ForEachAsync(paramData, async (p, cancelToken) =>
{
    var sw = new Stopwatch();
    sw.Start();
    var resp = await httpClient.PostAsync($"http://localhost/aspnet/auth/", postContent);
    var res = "OK";
    try
    {
        resp.EnsureSuccessStatusCode();
    }
    catch (Exception ex)
    {
        res = "ERR:" + ex.Message;
    }
    sw.Stop();
    results.Enqueue($"{p} {sw.ElapsedMilliseconds:n0}ms {res}");
});

Console.WriteLine(string.Join("\n", results.ToArray()));

反覆多執行幾次，有時會彈出以下錯誤：

Unhandled exception. System.ArgumentException: An item with the same key has already been added. Key: System.Net.Http.Headers.HeaderDescriptor
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)    
   at System.Net.Http.Headers.HttpHeaders.GetOrCreateHeaderInfo(HeaderDescriptor descriptor, Boolean parseRawValues)
   at System.Net.Http.Headers.HttpContentHeaders.get_ContentLength()
   at System.Net.Http.SocketsHttpHandler.ValidateAndNormalizeRequest(HttpRequestMessage request)
   at System.Net.Http.SocketsHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<>c__DisplayClass0_0.<<<Main>$>b__0>d.MoveNext() in X:\Lab\http-client-thread-safe\Program.cs:line 22
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Program.<Main>$(String[] args) in X:\Lab\http-client-thread-safe\Program.cs:line 16
   at Program.<Main>(String[] args)

有時是另一種訊息：

Unhandled exception. System.InvalidCastException: Unable to cast object of type 'System.Collections.Generic.List1[System.Object]' to type 'System.Int64'.
   at System.Net.Http.Headers.HttpContentHeaders.get_ContentLength()
   at System.Net.Http.SocketsHttpHandler.ValidateAndNormalizeRequest(HttpRequestMessage request)
   at System.Net.Http.SocketsHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

依老兵征戰沙場多年經驗，An item with the same key has already been added 配上偶爾出現，是平行處理沒處理好 Thread-Safe 的典型特徵。但 HttpClient 不是 Thread Safe 嗎？

Google 爬到 Github .NET Runtime 有篇 Issue 多執行緒設定 HttpClient.DefaultRequestHeaders.Authorization 導致 HttpHeaders.GetOrCreateHeaderInfo 爆出相同錯誤的案例。但該案例是多執行緒同時修改 DefaultRequestHeaders，較容易解釋。我的案例感覺就有點離奇了，但問題來自 HttpClientHandler.SendAsync()，不排除是多執行緒共用同一個 HttpClientHandler 造成問題。

有趣的是，將 .NET 版本由 .NET 6.0 改為 7.0，問題會消失。基於好奇，開了 dotPeek 追原始碼找到差異。(註：.NET 反組譯工具原本還有 JustDecompile、dnSpy 等選項，但二者都已多年未更新，已不適用 .NET 6+)

.NET 6.0 的出錯路徑是 GetOrCreateHeaderInfo() -> CreateAndAddHeaderToStore() -> AddHeaderToStore()，其中有段 new Dictionary<HeaderDescriptor, object>().Add() 非 Thread-Safe：

private void AddHeaderToStore(HeaderDescriptor descriptor, object value)
{
    Debug.Assert(value is string || value is HeaderStoreItemInfo);
    (_headerStore ??= new Dictionary<HeaderDescriptor, object>()).Add(descriptor, value);
}

.NET 7.0 版做了大幅改版，變成 GetOrCreateHeaderInfo() -> CreateAndAddHeaderToStore() -> AddEntryToStore() -> GetValueRefOrAddDefault()，邏輯大幅改寫，_headerStore 為 HeaderEntry[] 時查詢或新增至最後；為 null 時新建 HeaderEntry[] 並指定第一筆；_headerStore 為 Dictionary<HeaderDescriptor, object> 時用 CollectionsMarshal.GetValueRefOrAddDefault() 設定，再追進去，GetValueRefOrAddDefault 等同 TryInsert()，理論上容錯性較高，也或許其他處有修改寫法。總之，改用 .NET 7 後試了多次都不再能重現問題。

private ref object? GetValueRefOrAddDefault(HeaderDescriptor key)
{
    object? store = _headerStore;
    if (store is HeaderEntry[] entries)
    {
        for (int i = 0; i < _count && i < entries.Length; i++)
        {
            if (key.Equals(entries[i].Key))
            {
                return ref entries[i].Value!;
            }
        }

        int count = _count;
        _count++;
        if ((uint)count < (uint)entries.Length)
        {
            entries[count].Key = key;
            return ref entries[count].Value!;
        }

        return ref GrowEntriesAndAddDefault(key);
    }
    else if (store is null)
    {
        _count++;
        entries = new HeaderEntry[InitialCapacity];
        _headerStore = entries;
        ref HeaderEntry firstEntry = ref MemoryMarshal.GetArrayDataReference(entries);
        firstEntry.Key = key;
        return ref firstEntry.Value!;
    }
    else
    {
        return ref DictionaryGetValueRefOrAddDefault(key);
    }

    ref object? GrowEntriesAndAddDefault(HeaderDescriptor key)
    {
        var entries = (HeaderEntry[])_headerStore!;
        if (entries.Length == ArrayThreshold)
        {
            return ref ConvertToDictionaryAndAddDefault(key);
        }
        else
        {
            Array.Resize(ref entries, entries.Length << 1);
            _headerStore = entries;
            ref HeaderEntry firstNewEntry = ref entries[entries.Length >> 1];
            firstNewEntry.Key = key;
            return ref firstNewEntry.Value!;
        }
    }

    ref object? ConvertToDictionaryAndAddDefault(HeaderDescriptor key)
    {
        var entries = (HeaderEntry[])_headerStore!;
        var dictionary = new Dictionary<HeaderDescriptor, object>(ArrayThreshold);
        _headerStore = dictionary;
        foreach (HeaderEntry entry in entries)
        {
            dictionary.Add(entry.Key, entry.Value);
        }
        Debug.Assert(dictionary.Count == _count - 1);
        return ref CollectionsMarshal.GetValueRefOrAddDefault(dictionary, key, out s_dictionaryGetValueRefOrAddDefaultExistsDummy);
    }

    ref object? DictionaryGetValueRefOrAddDefault(HeaderDescriptor key)
    {
        var dictionary = (Dictionary<HeaderDescriptor, object>)_headerStore!;
        ref object? value = ref CollectionsMarshal.GetValueRefOrAddDefault(dictionary, key, out s_dictionaryGetValueRefOrAddDefaultExistsDummy);
        if (value is null)
        {
            _count++;
        }
        return ref value;
    }
}

總之，實測升級 .NET 7 可避開問題，另外還發現如果修改一下程式，將 FormUrlEncodedContent 移入迴圈內建立不共用，反覆測試多次沒再出錯。又花了點時間追 Code 但沒找到共用 HttpContent 會釀禍的證據。

var paramData = "1,2,3,4,5,6,7,8,9".Split(',');

await Parallel.ForEachAsync(paramData, async (p, cancelToken) =>
{
    var postContent = new FormUrlEncodedContent(new Dictionary<string, string>()
    {
        ["key"] = "1234"
    });
    var sw = new Stopwatch();
    sw.Start();
    var resp = await httpClient.PostAsync($"http://localhost/aspnet/auth/", postContent);
    var res = "OK";
    try
    {
        resp.EnsureSuccessStatusCode();
    }
    catch (Exception ex)
    {
        res = "ERR:" + ex.Message;
    }
    sw.Stop();
    results.Enqueue($"{p} {sw.ElapsedMilliseconds:n0}ms {res}");
});

最後，終於文找到 .NET 6 HttpHeaders 非 Thread-Safe 的 Issue 回報以及 .NET 7 做了修正的證據。

結論：HttpClient 原則上是 Thread-Safe 的，但在 .NET 6 若涉及 HttpHeaders 操作在多執行緒環境可能出錯，此問題於 .NET 7 修復。

HttpClient 多執行緒執行出錯案例

Comments

# 2023-11-29 09:11 AM by Jy

Post a comment