Skip to content

Close request body async iterator when a write error occurs - #1119

Open
Dextheking1 wants to merge 5 commits into
encode:masterfrom
Dextheking1:fix/3597-writeerror-stream-aclose
Open

Dextheking1 wants to merge 5 commits into
encode:masterfrom
Dextheking1:fix/3597-writeerror-stream-aclose

Conversation

@Dextheking1

@Dextheking1 Dextheking1 commented Sep 23, 2026 •

Copy link
Copy Markdown

Closes encode/httpx#3597.

What

AsyncHTTP11Connection._send_request_body now closes the request body's async iterator in a finally block, so it is cleaned up on both the success and failure paths.

Why

When a WriteError is raised part-way through sending the request body, handle_async_request catches and suppresses it (to allow reading a pre-emptive error response from the server), but the request body's async iterator was left abandoned mid-iteration. It was then garbage collected without ever being exhausted, producing:

ResourceWarning: Async generator '...' was garbage collected before it had been exhausted.
Surround its use in 'async with aclosing(...):' to ensure that it gets cleaned up as soon as you're done using it.

Note: closing the iterator (rather than calling request.stream.aclose()) is what fixes this, e.g. httpx's ByteStream.aclose() is a no-op, while the abandoned object is the generator created by stream.__aiter__(). The aclose call is getattr-guarded so streams whose iterators lack aclose keep working.

Test evidence

New regression test test_http11_write_error_closes_request_body in tests/_async/test_http11.py: a network stream that raises WriteError mid-body, asserting the body iterator receives GeneratorExit while the WriteError stays suppressed and the response remains readable.

Before (on master, trio backend):

ResourceWarning: Async generator '...TrackingBody.__aiter__.<locals>.gen' was garbage collected before it had been exhausted. ...

After: no warning; tests/_async/test_http11.py, 28 passed (asyncio + trio backends). ruff check and ruff format clean.

If a `WriteError` is raised part-way through `_send_request_body`,
the request body's async iterator was abandoned mid-iteration and
later garbage collected without ever being exhausted, triggering
`ResourceWarning: async generator was garbage collected before it
had been exhausted` (encode/httpx#3597).

Close the iterator in a `finally` block so it is cleaned up on both
success and failure paths.
The _sync/http11.py and _sync/test_http11.py files are generated from
their _async counterparts by scripts/unasync.py (enforced by
'scripts/check' in CI). Also annotate the new test's generator so mypy
is clean.
@Dextheking1
Dextheking1 marked this pull request as ready for review September 23, 2026 23:54
@Dextheking1
Dextheking1 force-pushed the fix/3597-writeerror-stream-aclose branch from 6eed176 to d6d35b7 Compare September 28, 2026 11:29
@jacksonsr451

Copy link
Copy Markdown

Hi! I investigated the async request-body iterator cleanup related to encode/httpx#3597 and reviewed this PR.

I found one additional cancellation-safety edge case: when an async generator's finally block contains an await checkpoint, Trio cancellation can interrupt the unshielded aclose() before the generator finishes cleanup.

I reproduced this deterministically against the current PR implementation. The regression test fails under Trio without shielding and passes under both asyncio and Trio when aclose() is wrapped with HTTPCore's existing AsyncShieldCancellation.

I also ran the full HTTPCore and HTTPX test suites, comparing the complementary change against the PR head. The observed failures were reproducible on the unmodified PR implementation.

I have a minimal complementary patch and regression test prepared locally. Would you be interested in incorporating this cancellation-safety improvement into the existing PR?

Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

ResourceWarning in test_write_timeout due to Unclosed Async Generator

2 participants