Repository navigation
Close request body async iterator when a write error occurs - #1119
Dextheking1 wants to merge 5 commits into
Conversation
If a `WriteError` is raised part-way through `_send_request_body`, the request body's async iterator was abandoned mid-iteration and later garbage collected without ever being exhausted, triggering `ResourceWarning: async generator was garbage collected before it had been exhausted` (encode/httpx#3597). Close the iterator in a `finally` block so it is cleaned up on both success and failure paths.
The _sync/http11.py and _sync/test_http11.py files are generated from their _async counterparts by scripts/unasync.py (enforced by 'scripts/check' in CI). Also annotate the new test's generator so mypy is clean.
6eed176 to
d6d35b7
Compare
|
Hi! I investigated the async request-body iterator cleanup related to encode/httpx#3597 and reviewed this PR. I found one additional cancellation-safety edge case: when an async generator's I reproduced this deterministically against the current PR implementation. The regression test fails under Trio without shielding and passes under both asyncio and Trio when I also ran the full HTTPCore and HTTPX test suites, comparing the complementary change against the PR head. The observed failures were reproducible on the unmodified PR implementation. I have a minimal complementary patch and regression test prepared locally. Would you be interested in incorporating this cancellation-safety improvement into the existing PR? Thanks! |
Closes encode/httpx#3597.
What
AsyncHTTP11Connection._send_request_bodynow closes the request body's async iterator in afinallyblock, so it is cleaned up on both the success and failure paths.Why
When a
WriteErroris raised part-way through sending the request body,handle_async_requestcatches and suppresses it (to allow reading a pre-emptive error response from the server), but the request body's async iterator was left abandoned mid-iteration. It was then garbage collected without ever being exhausted, producing:Note: closing the iterator (rather than calling
request.stream.aclose()) is what fixes this, e.g. httpx'sByteStream.aclose()is a no-op, while the abandoned object is the generator created bystream.__aiter__(). Theaclosecall isgetattr-guarded so streams whose iterators lackaclosekeep working.Test evidence
New regression test
test_http11_write_error_closes_request_bodyintests/_async/test_http11.py: a network stream that raisesWriteErrormid-body, asserting the body iterator receivesGeneratorExitwhile theWriteErrorstays suppressed and the response remains readable.Before (on
master, trio backend):After: no warning;
tests/_async/test_http11.py, 28 passed (asyncio + trio backends).ruff checkandruff formatclean.