Skip to content

fix: avoid LAPACK deadlock and validate older JAX compatibility - #280

Merged
Jammy2211 merged 4 commits into
mainfrom
feature/restore-dashboard-green
Oct 4, 2026
Merged

Jammy2211 merged 4 commits into
mainfrom
feature/restore-dashboard-green

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Release integration's old <0.11 override selected JAX0.10.2 and the exact-wheel replay captured all four Eigen workers blocked in LAPACK ParallelBatchMap -> CholeskyFactorization -> lapack_dpotrf_ffi. Keep the release baseline at >=0.11.2,<0.12, whose open-source build removes that path. Preserve the 115-pin incident manifest and bounded native diagnostic.

Add an independent compatibility workflow for0.9.2 and0.11.2 using the original TestPyPI wheels, Python3.12.14, exact workspace/Hands commits, three original-likelihood trials, numerical/sampler checks and the historical FFT witness. The original control/candidate comparison remains manual and preserves its inconclusive result. Package policy in companion Nerves/Hands/consumer PRs retains older JAX while excluding affected releases; this validation baseline is not a global user minimum.

API Changes

Validation dependency floor changes; diagnostic workflows and evidence added. No Python public symbols, workspace API migration, timeout increases, quarantines or readiness-rule changes. See full details below.

Test Plan

  • Heart full suite:1222 passed. YAML, tenant firewall and whitespace checks pass.
  • Independent Sol review CLEAN for the eight-repo compatibility change.
  • Exact replay37205459198 captures native LAPACK deadlock after one pass.
  • Local CPU/CUDA numerical, original-likelihood and resolver witnesses described below.
  • Hosted compatibility matrix 37210342253 passes both endpoints at8721c2e: Python3.12.14,113 unchanged non-JAX pins verified, original likelihood3/3 each (0.9.2:17.433/14.117/14.184s;0.11.2:15.323/12.021/12.014s), numerical and20-iteration FFT witnesses pass. Artifacts downloaded and checked.
  • Exact-head Heart unit CI 37210342215 passes Python3.12/3.13 at8721c2e.

Interleaved comparison37206724174 remains inconclusive: three controls and three candidates passed, so it does not measure failure-rate improvement. Previous-head Python3.12/3.13 CI37207278755 passed; cancelled37207278790 contributes no evidence. Original runner image and original wheel/FITS hashes were not retained.

Evidence and limits

Native capture in Heart replay37205459198 shows the 0.10.2 LAPACK/Eigen worker-pool deadlock. Tagged source lacks this path in 0.9.2 and disables it in OSS builds from 0.11.1. Only 0.10.2 was observed deadlocking; other exclusions are source-based precautions.

Original likelihood passes three CPU trials and one actual CUDA trial on each of 0.9.2/0.11.2; independent likelihood, finite-difference gradient, NUFFT, Optax and short BlackJAX execution checks pass on both backends. Historical FFT workaround is retained and its reproducer passes both versions. All 25 wheel resolver cases pass; fresh normal resolution without prereleases also selects protected synthetic stable wheels. Local Python3.12.10 differs from incident3.12.14.

Broader point-source gradient diagnostics hit the unchanged300s cap on BOTH versions after earlier checks pass; cause remains unresolved. Retained range does not certify every older version. No full release validation was rerun. Detailed receipts and compatibility CI: HeartPR280.

Development authorization

Heart remains RED: release validation FAILED (stage integrate). Live human authorization: “I authorize development investigation and repair despite the current RED reason above. Follow start-dev and record the task-specific authorization wherever required.” Compatibility plan approved: “ok yes do it properly then”. Independent Sol review CLEAN; applicable tests and targeted smoke evidence above pass. This authorizes development PRs only. Merge requires explicit human /prm; no release is authorized. Umbrella: #274.

Full API Changes (for automation & release notes)

Changed Behaviour

  • Smoke/integration/notebook recipes require JAX/JAXlib>=0.11.2,<0.12.
  • Compatibility diagnostics exercise older0.9.2 and baseline0.11.2 without clearing release readiness.
  • Original incident comparison is manual-only; its failed control and inconclusive evidence remain preserved.

Migration

  • No workspace API changes. Package release ordering and lockfile migration are documented in the companion PRs and docs/release_validation.md.

Generated by the PyAutoLabs agent workflow.

Coordinated PRs

PyAutoNerves, PyAutoHands, PyAutoFit, PyAutoGalaxy, PyAutoHeart, PyAutoArray, PyAutoLens, PyAutoCTI. Nerves must publish before repaired consumer wheels.

@Jammy2211 Jammy2211 changed the title Avoid JAX LAPACK pool deadlock in release validation fix: avoid LAPACK deadlock and validate older JAX compatibility Oct 4, 2026
@Jammy2211 Jammy2211 added the pending-release Awaiting a published release label Oct 4, 2026
@Jammy2211
Jammy2211 merged commit 4920ab9 into main Oct 4, 2026
4 checks passed
@Jammy2211
Jammy2211 deleted the feature/restore-dashboard-green branch October 4, 2026 15:50
@Jammy2211 Jammy2211 removed the pending-release Awaiting a published release label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant