Repository navigation
Fix requirements files read as absent when not UTF-8 (#1120, #1119) - #1152
Conversation
Assisted-by: Claude Code:claude-opus-5-5
A requirements.txt that pip or uv installs from was read as absent
whenever it was not plain UTF-8:
- A file saved as Latin-1 or cp1252 with a PEP 263 coding line
("# -*- coding: latin-1 -*-") was skipped by lock-only discovery, so
a fresh-checkout scan said "No pypi packages found" and exited 0
while pip installed the unpatched release (#1119).
- A UTF-16 requirements.txt exported beside uv.lock (what
`uv export > requirements.txt` writes in Windows PowerShell 5.1) was
ignored by the vendored routing, `vendor --check` and `vex`. The scan
gave no pypi_multiple_lockfiles warning, the check passed and VEX
attested not_affected while `uv pip install -r requirements.txt`
installed the unpatched release (#1120).
pip's decoder now lives in one place. utils::requirements::decode
follows pip's auto_decode: byte-order marks first, then a PEP 263
coding line in the first two lines (UTF-8, ASCII, Latin-1 and cp1252
and their Python aliases), then UTF-8. A codec it does not model is
reported as unreadable, never guessed.
Every reader that only observes requirements files now goes through
it: the -r include walk, the vendored #612 probe beside a tool lock,
and VEX discovery. The probe also counts a requirements.txt it cannot
decode as one that may pin the package, so the warning fails closed.
The vendored planner still refuses files that are not UTF-8, since it
rewrites them byte for byte.
Fixes #1119
Fixes #1120
Assisted-by: Claude Code:claude-opus-5-5
Adds the CLI-level regression for #1120: a requirements.txt exported beside uv.lock (UTF-8, UTF-16 LE and UTF-16 BE) makes `vendor` warn pypi_multiple_lockfiles and `vendor --check` fail on the contested wiring, and is left byte-for-byte as the user wrote it. Before the decoder change, both UTF-16 cases passed the check as if the file were absent. Refs #1120 Assisted-by: Claude Code:claude-opus-5-5
|
BugBot review Generated by Claude Code |
Resolves the conflict with #1091 in VEX requirements discovery: keep its install_tree grouping of the -r include tree and read each file through the pip-compatible decoder. Assisted-by: Claude Code:claude-opus-5-5
|
BugBot review Generated by Claude Code |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 576706e. Configure here.
|
[agent] CI is green on 576706e. Three jobs failed once in code this PR doesn't touch. Each passed on its single re-run:
Generated by Claude Code |
Mikola Lysenko (mikolalysenko)
added
the
Ready for review
|
Burn-down agent: labeled Ready for review at 576706e.
Generated by Claude Code |
|
[agent] The merge queue dropped this PR at 22:25 UTC because of The failure isn't caused by this PR. The same job passed on this PR's own CI at 576706e. It also passed in the queue runs for #1108 and #1147 during the same window, and this diff doesn't touch sbt or JVM code. There's no fix to port, so I re-queued the PR (22:26 UTC). Generated by Claude Code |
Resolve the pypi_requirements.rs conflict with #1168, which moved the requirements walk onto ProjectView: keep this PR's per-caller decode step (strict UTF-8 for the planner, pip's decoding for the include listing) by reading bytes through view.read_bytes and decoding them, and drop the now-unused read_regular_to_bytes import. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
[final reviewer] The queue dropped this at 23:18 UTC because #1168 merged first and conflicts with it in
Tanmay Singla (@Tanmay182003) this is a merge commit only, but the conflict resolution touches the walk itself, so a quick look at the merge diff of Generated by Claude Code |
|
[agent] The queue dropped this at 00:26 UTC because Generated by Claude Code |
|
[agent] The queue dropped this again at 00:34 UTC on Generated by Claude Code |
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3 participants
LLM Description written by Claude Code:claude-opus-5-5
Fixes #1120
Fixes #1119
Root cause
The requirements-file readers had no single pip-compatible
decode-or-refuse entry point.
utils::requirements::decode(#724)handled byte-order marks only, with no PEP 263
coding:step (#1119).Several call sites skipped it entirely and read strict UTF-8:
vendor/pypi.rsrequirements_pins_target(the #612 probe), theshared
-rinclude walk (requirements_include_names) andvex/discover/pypi_other.rsextract_requirements(#1120). Each sitethen mapped a file it couldn't decode to "absent", so scans,
vendor --checkandvexall wired around a file pip or uv stillinstalls from.
Fix
decodenow follows pip'sauto_decode: BOMs first, then a PEP 263coding line in the first two lines (UTF-8, ASCII, Latin-1, cp1252 and
their Python aliases), then UTF-8. A codec it doesn't model reads as
undecodable and is never guessed. I checked it against pip 24.0's own
auto_decodeon the same byte inputs (Latin-1, ISO-8859-1 on line 2,cp1252 punctuation, undefined cp1252 bytes, a third-line coding,
L1,cp819), and the results match.the include walk, the vendored Vendored mode in a Pipenv project wires only Pipfile.lock and silently leaves a sibling requirements.txt unpatched, and the hosted → vendored takeover reverts that file's hosted pin to plain PyPI #612 probe beside a tool lock, and VEX
discovery (new
DiscoverCtx::read_requirements_text). The Vendored mode in a Pipenv project wires only Pipfile.lock and silently leaves a sibling requirements.txt unpatched, and the hosted → vendored takeover reverts that file's hosted pin to plain PyPI #612 probecounts a file it can't decode as one that may pin the package, so it
fails closed.
files because it rewrites them byte for byte, and hosted mode still
refuses them by name (
candidate_file_unreadable). With the pins nowdiscovered, a lock-only scan of a Latin-1 file reaches that refusal
instead of printing "No pypi packages found" and exiting 0.
No wrapper changes are needed (
npm/,pypi/,gem/only dispatch tothe binary).
Per-issue tests (red without the fix, green with it)
utils::requirements::tests::decode_follows_pips_pep_263_coding_linevendor::lock_inventory::tests::requirements_pep_263_files_are_inventoried(root file and-rinclude, disk and in-memory)scan_requirements_lock_only::lock_only_scan_discovers_pep_263_pins(hosted and vendored lock-only scan, root and include)vendor::pypi::tests::non_utf8_requirements_beside_the_governing_lock_is_a_loud_loser(UTF-16 LE/BE, PEP 263, undecodable)vex::discover::pypi_other::tests::requirements_files_decode_as_pip_does(UTF-16 root and include, PEP 263 include, undecodable diagnosed)vendor::pypi_requirements::tests::requirements_include_names_decodes_as_pip_doesmode_migration_pypi::uv_requirements_export_contests_the_wiring_in_any_encoding(vendorwarnspypi_multiple_lockfiles,vendor --checkfails "wiring contested", for UTF-8, UTF-16 LE and UTF-16 BE)To show red, I temporarily reverted only the non-test fix hunks. All 5
new core tests and both new CLI tests then failed.
Local evidence
cargo clippy --workspace --all-features -- -D warnings: clean(before and after merging main).
cargo test -p socket-patch-core --all-features: 5788 passed, 4failed. The 4 are read-only-permission tests (
relax_loop_must_not_traverse_symlinked_root,an_unremovable_hidden_lock_keeps_every_store_entry,wire_write_failure_maps_error_and_leaves_lock_untouched,wire_failure_rolls_back_already_written_files) that can't fail awrite as root, and this sandbox runs as uid 0. They are unrelated to
this diff and pass in CI.
e2e_vendor_pypi_build -- --include-ignoredwith real uv 0.11.32:36/36 passed.
scan_requirements_lock_only,mode_migration_pypi,scan_vendor_requirements_unwired,in_process_vendor_pypi_takeover,in_process_redirect_pipenv,hosted_superseding_pypi,in_process_get_hosted_ecosystems,hosted_memory_engine,e2e_redirect_uv_build): all green exceptpipenv_hosted_to_vendored_names_the_unpatched_requirements. Thattest fetches
https://pypi.org/pypi/six/1.16.0/jsonlive, and thesandbox's TLS-inspecting proxy breaks that request because the client
trusts only bundled roots (Every HTTPS call fails behind a TLS-inspecting proxy because the clients trust only bundled webpki roots #1107). It passes in CI.
cargo test --workspacedidn't fit in the sandbox's diskallowance (hundreds of CLI test binaries). CI covers it.
CI
in
pypi_other.rs: it keeps Fix requirements -r include tree counted as a competing lock (#1086) #1091'sinstall_treegrouping andreads each file through the new decoder. All 15 workflows passed.
One job needed a re-run:
test (windows-latest, 2)failed once ine2e_redirect_gem_stale_install::gem_hosted_global_gemfile_setting_is_refused,a gem suite this diff doesn't touch. That test passes on Linux
(37/37), passed on Windows for 3095cf5 and on main, and passed on the
re-run.
failed fetching fixtures from Maven Central (HTTP 429, and
commons-text not found).
🤖 Generated with Claude Code
https://claude.ai/code/session_01YTFXaxBRSh6oHYt8J9yegn
Generated by Claude Code