feat: opt-in quadrature pooling of unequal real/imag sigma on the sparse path

[Jammy2211]

Summary

The sparse interferometer path refuses any noise map whose real and imaginary sigmas differ beyond rtol=1e-5. Noise estimated by differencing adjacent visibilities typically shows 1-2 % real/imag scatter, so every such chunk raised. This adds an opt-in pool_noise_map=True that pools the two sigmas in quadrature, sigma^2 = (sigma_re^2 + sigma_im^2) / 2 (which preserves the total variance), and logs how large the difference was. The default (False) is unchanged and still raises.

Requested by @HRSAstro (pyuvimage) in https://github.com/orgs/PyAutoLabs/discussions/13 (item 2, "Unequal real and imaginary sigma"). Closes #617.

• sparse_terms_from_chunks(..., pool_noise_map=True) pools each chunk's noise map before every term (W~, dirty image, dirty beam, sum_weights, data_term, noise_normalization). The asymmetry is accumulated over the whole stream and logged once per call: nothing when the sigmas are already equal (within rtol=1e-5), INFO with the median and max difference when the median is at most 25 %, WARNING above 25 % (the difference may be real). The warning names the dense InversionInterferometerMapping path as the exact one.
• Interferometer.from_stream(..., pool_noise_map=True) passes the option through.
• Interferometer.apply_sparse_operator(..., pool_noise_map=True) builds every term from the pooled noise map and returns the dataset carrying the pooled noise_map, so the dense residual/chi-squared maps agree with the cached sparse data_term / noise_normalization.
• apply_sparse_operator_from_chunks(pool_noise_map=True) is refused with a DatasetException, as phase_centre already is: it keeps this dataset's unpooled noise_map. The message says to pool the dataset first with noise_map_pooled_from, or to use from_stream(..., pool_noise_map=True).
• Docstrings and the existing error text now cover four things:
  • the equal-sigma assumption and how to pool;
  • the approximation: the exact curvature is sum wbar cos(a-b) + dw cos(a+b), and W~ can only hold the Toeplitz cos(a-b) part;
  • with pooling, data_term is exactly phase-invariant;
  • log-evidences of a pooled fit and an unpooled fit should not be compared.

API Changes

Added only. Two new helpers and a new keyword argument pool_noise_map=False on three existing entry points. A fourth entry point refuses that keyword when it is True. Default behaviour is unchanged bit for bit.
See full details below.

Test Plan

• Witness: sparse_terms_from_chunks(chunks, pool_noise_map=True) on chunks with 2 % asymmetry equals the default call on the same chunks pooled beforehand with noise_map_pooled_from. The check is field by field and bit-identical. The default call on the unpooled chunks still raises. The test fails on unfixed main (unknown kwarg).
• The existing raise tests are unchanged and pass under the default: test_dataset.py test__apply_sparse_operator__unequal_real_imag_noise__raises_exception and test__apply_sparse_operator_from_chunks__unequal_real_imag_noise__raises; util test__check_noise_map_real_imag_equal__tiny_sigmas_use_a_relative_tolerance and test__sparse_terms_from_chunks__unequal_real_imag_noise_in_a_later_chunk__raises.
• Helpers: the pooled map keeps re^2 + im^2 and has equal parts. Already-equal entries come back bit for bit, so pooling is idempotent. All three input forms are accepted. The asymmetry helper gives the expected values on a known 2 % map.
• apply_sparse_operator(pool_noise_map=True): the returned noise_map is pooled. The sparse log_evidence equals the mapping-path log_evidence on the pooled dataset to rel=1e-8 (measured ~1e-15). On equal sigmas the result is unchanged and nothing is logged.
• from_stream(pool_noise_map=True) equals from_stream on pre-pooled chunks, bit for bit.
• caplog: one INFO line at 2 % ("median difference 2.00 %"), one WARNING at 30 % naming InversionInterferometerMapping, and nothing at equality. Each case logs once per call across 3 chunks.
• apply_sparse_operator_from_chunks(pool_noise_map=True) raises.
• Documentation pin of the approximation size: the pooled-sparse fit is compared with the exact unpooled-mapping fit at 2 % asymmetry on 40 visibilities. Δ log_evidence = 9.9e-3 nats (7.7e-5 relative). Redrawing which visibilities have the larger imaginary sigma (seeds 0-7) gives 0.010-0.096 nats, at most 7.4e-4 relative. Most of this is chi-squared: the weighting error is first order in the asymmetry and random in sign. The noise_normalization shift is fixed and second order (8.2e-3 nats). The test asserts 1e-3 < |Δ| < 1e-1.
• test_dataset.py + test_inversion_interferometer_util.py: 83 passed. Full test_autoarray: 1991 passed, 4 xfailed.

Full API Changes (for automation & release notes)

Added

• autoarray.inversion.inversion.interferometer.inversion_interferometer_util.noise_map_pooled_from(noise_map) -> np.ndarray: complex s + 1j*s with s = sqrt((re^2 + im^2) / 2). It accepts a VisibilitiesNoiseMap, a complex (K,) array or a real (K, 2) array, and returns exactly-equal entries unchanged.
• inversion_interferometer_util.noise_map_real_imag_asymmetry_from(noise_map) -> (median, max): the fractional |re - im| / max(re, im).
• inversion_interferometer_util.NOISE_MAP_POOLING_WARNING_FRACTION = 0.25.

Changed Signature

• inversion_interferometer_util.sparse_terms_from_chunks(..., pool_noise_map: bool = False)
• Interferometer.from_stream(..., pool_noise_map: bool = False)
• Interferometer.apply_sparse_operator(..., pool_noise_map: bool = False)

Changed Behaviour

• Interferometer.apply_sparse_operator_from_chunks(..., pool_noise_map=True) raises DatasetException. Before this change it would have raised TypeError from the accumulator.
• The check_noise_map_real_imag_equal error message now explains pooling (pool_noise_map=True, noise_map_pooled_from) and the cos(a+b) caveat. The raising condition is unchanged.

Migration

None needed (opt-in). To use it:

• Before: aa.Interferometer.from_stream(chunks, mask) raises on 1-2 % sigma scatter.
• After: aa.Interferometer.from_stream(chunks, mask, pool_noise_map=True). Or pool upstream with noise_map_pooled_from(noise_map).

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code