The Middleware Bug That Took Down Our API (Twice)

Picture this: Your API is humming along perfectly. 99.9% success rate. Life is good.
Then suddenly – BAM! 🔥
Random 500 errors. Garbled responses. Users complaining about “weird characters” in their data.
And it only happens under load.
The Culprit? Our “innocent” logging middleware.
Here’s what we did WRONG:

//
public async Task InvokeAsync(HttpContext context, RequestDelegate next) {
    await next(context); // Call next middleware

    // Try to log the response
    var body = await context.Response.Body.ReadAsStringAsync();
    Logger.Log(body); // Oops...
}
//

Why This Fails Spectacularly:
1.  The response stream is already consumed and sent to the client
2.  Multiple requests try to read/modify streams concurrently
3.  Streams aren't thread-safe by default
4.  Gzip-compressed responses come back as garbage characters
5.  Everything breaks in weird, unpredictable ways
The Fix That Actually Works:

`
public async Task InvokeAsync(HttpContext context, RequestDelegate next) {
    // Step 1: Enable request buffering
    context.Request.EnableBuffering();

    // Step 2: Swap response stream with a buffered one
    var originalBodyStream = context.Response.Body;
    using var memoryStream = new MemoryStream();
    context.Response.Body = memoryStream;

    try {
        // Step 3: Let the pipeline execute
        await next(context);

        // Step 4: Safely read the buffered response
        memoryStream.Position = 0;

        // Step 5: Handle compression properly
        string responseBody;
        if (context.Response.Headers.ContentEncoding.Contains("gzip")) {
            using var gzipStream = new GZipStream(memoryStream, CompressionMode.Decompress);
            using var reader = new StreamReader(gzipStream, Encoding.UTF8);
            responseBody = await reader.ReadToEndAsync();
        } else {
            using var reader = new StreamReader(memoryStream, Encoding.UTF8);
            responseBody = await reader.ReadToEndAsync();
        }

        // Step 6: Log it
        Logger.Log(responseBody);

        // Step 7: Copy back to original stream
        memoryStream.Position = 0;
        await memoryStream.CopyToAsync(originalBodyStream);
    } finally {
        // Step 8: Restore original stream
        context.Response.Body = originalBodyStream;
    }
}

Enter fullscreen mode Exit fullscreen mode

The Hidden Gotcha Nobody Tells You:
Modern APIs use gzip/deflate compression by default. If you don’t decompress before reading, you get binary garbage that looks like: �H���W�K�M�U��Z

Our Results:
✅ Zero stream-related exceptions
✅ Zero “garbage character” bug reports
✅ Proper logging at 10,000+ req/min
✅ My on-call rotation got way quieter 😴
The Brutal Truth:
Middleware is easy to write. Thread-safe, production-ready middleware? That’s an art form.

Who else has spent hours debugging middleware issues? Let’s commiserate in the comments 👇

dotnet #aspnetcore #middleware #threadsafety #debugging #productionissues #softwareengineering #webdevelopment #csharp #lessonslearned #cloudnative

P.S. Always test middleware under load. Your local machine running 1 request at a time will NEVER expose these bugs.