Skip to content

feat: SparseTerms oversample=q fine precision-operator and dirty-image grids - #621

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/sparse-terms-oversampled-fine-grids
Oct 7, 2026
Merged

Jammy2211 merged 2 commits into
mainfrom
feature/sparse-terms-oversampled-fine-grids

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

Adds an optional oversample=q to sparse_terms_from_chunks. It was requested by @HRSAstro (pyuvimage) in Discussion #13, comment item 1. With it set, the same streaming pass also accumulates two grids q times finer than the image pixel:

  • precision_operator_fine: K(lag) = Σ w cos(2π u·lag) over the full native shape plus a ceil(oversample_pad · max(N)) lag pad (default 25 %), in the same wraparound order as W~, with the Nyquist row/column kept.
  • dirty_image_fine: D(x) = Re Σ c exp(2πi u·x) from the same phase-shifted, noise-weighted data as dirty_image_native. It covers twice the field, centred, in native orientation.

For a unit point at p these give every sparse-likelihood term of a uv-analytic component: the cross term K(x_i − p), the point-point term K(p − p') and the data term D(p). pyuvimage can then drop its own pass. The column-terms helper (review item B2) is a deferred follow-up and is not in this PR.

The type-1 core of nufft_precision_operator_via_nufft_from is lifted into _type1_real_grid_from. A new test pins W~ bit for bit against a verbatim copy of the old inline loop. The fine grids are preallocated and added in place per chunk (quadrant-wise ifftshift add), so no full-size grid is allocated per chunk. They inherit the equal-sigma check and pool_noise_map=True pooling from #619. One info line logs a memory estimate.

The dependency sparse_noise_map_pooling_option is merged as #619 (2c5cb69).

Measured cost (warm call, CPU, 100×100 native, 2 × 1e5-visibility chunks, fresh process each):

oversample wall peak RSS fine grids
None 2.1 s 2016 MB —
4 5.2 s 2159 MB K 1000², D 800²
8 5.0 s 2533 MB K 2000², D 1600²

At 400 px and q=8, K is 512 MB and D 328 MB, plus about 4 GB of nufftax work grid per call (5-6 GB peak). The cost is documented, with the advice to use chunks of at least ~1e6 visibilities with oversample.

Closes #620 (on merge of this PR and PyAutoLabs/PyAutoGalaxy's companion PR).

API Changes

Additive only. Defaults are unchanged and W~ is bit-for-bit unchanged.

  • sparse_terms_from_chunks(..., oversample=None, oversample_pad=0.25); Interferometer.from_stream(..., oversample=None, oversample_pad=0.25); apply_sparse_operator_from_chunks(..., oversample=...) via its accumulator kwargs.
  • SparseTerms gains optional precision_operator_fine, dirty_image_fine, oversample, oversample_pad, and a has_fine_grids property.
  • SparseTerms.__add__ now also checks oversample/oversample_pad provenance, and refuses when exactly one side carries fine grids.
  • oversample must be a positive even integer, and the transformer a TransformerNUFFT. Otherwise it raises ValueError / InversionException.
    See full details below.

Test Plan

  • Witness (i): fine K at lags (iq, jq) == nufft_precision_operator[i, j] within the masked extent (mixed rtol/atol 1e-10), and fine K == brute force everywhere, pad and Nyquist included.
  • Witness (ii): fine D at native pixel centres == dirty_image_native on unmasked pixels, and fine D == brute-force DFT everywhere.
  • Chunked == one-shot for both grids; phase_centre shifts D by exactly the shift and leaves K bit-identical; pooled == pre-pooled bit for bit.
  • __add__ sums fine grids and refuses mismatched oversample/oversample_pad, fine-on-one-side and fine-shape mismatch; odd/zero/negative/float/bool q, negative pad and the DFT transformer raise; memory log line; pass-through on from_stream / apply_sparse_operator_from_chunks.
  • W~ refactor bit-for-bit pin (7x7 and 16x16 fixtures, one-shot and chunked).
  • Witness tests run red on unfixed source first (22 failed, 1 reference passed).
  • Full test_autoarray: 2014 passed, 4 xfailed. Full test_autogalaxy (with the companion PR): 1357 passed.
  • Smoke: autolens_workspace/scripts/interferometer/features/datacube/modeling_array_free.py (from_stream / summed SparseTerms / phase_centre) passes on this branch.
Full API Changes (for automation & release notes)

Added

  • autoarray.inversion.inversion.interferometer.inversion_interferometer_util.sparse_terms_from_chunks(..., oversample: Optional[int] = None, oversample_pad: float = 0.25) — accumulates precision_operator_fine / dirty_image_fine.
  • aa.Interferometer.from_stream(..., oversample=None, oversample_pad=0.25) — pass-through.
  • aa.SparseTerms.precision_operator_fine, .dirty_image_fine, .oversample, .oversample_pad (all default None), .has_fine_grids.
  • inversion_interferometer_util.fine_grid_shapes_from(shape_native, oversample, oversample_pad=0.25) — the two fine-grid shapes.
  • inversion_interferometer_util._type1_real_grid_from(...) (private) — reusable type-1 NUFFT real-grid core with in-place accumulation.

Changed

  • aa.SparseTerms.__add__ — oversample / oversample_pad join the provenance check; terms where exactly one side carries fine grids now raise InversionException (previously no such fields existed).

Migration

  • None: every new argument and field is optional, with the previous behaviour as default.

Requested by @HRSAstro — https://github.com/orgs/PyAutoLabs/discussions/13 (https://github.com/orgs/PyAutoLabs/discussions/13#discussioncomment-18741903).

Tier: judge — merge mode: human /prm. Heart readiness at ship: YELLOW (manifest drift: shared-standards blocks; release validation stale), acknowledged.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

…e grids

`sparse_terms_from_chunks(..., oversample=q, oversample_pad=0.25)` accumulates two
grids q times finer than the image pixel in the same streaming pass, for components
analytic in the uv-plane (points, small Gaussians; Discussion #13 item 1, @HRSAstro):

- precision_operator_fine: K(lag) = sum w cos(2 pi u.lag), full native shape plus a
  ceil(pad * max(N)) lag pad, W~'s wraparound order, Nyquist kept;
- dirty_image_fine: D(x) from the phase-shifted noise-weighted data, twice the
  field, centred, native orientation.

The type-1 core of nufft_precision_operator_via_nufft_from is lifted into
_type1_real_grid_from (W~ pinned bit for bit against the old inline loop). The fine
grids are preallocated and added in place per chunk; q must be even and the
transformer TransformerNUFFT. SparseTerms gains precision_operator_fine,
dirty_image_fine, oversample, oversample_pad; __add__ checks oversample provenance
and refuses fine grids on one side only. Pass-through on Interferometer.from_stream
and apply_sparse_operator_from_chunks; one memory-estimate log line.

Refs #620

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The two oversample ValueError tests build their inputs via
_streaming_inputs(), which constructs a TransformerNUFFT and so needs
nufftax; they failed (7 cases) on the unittest-nojax CI leg. Guard them
with pytest.importorskip("nufftax") like their siblings.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211
Jammy2211 merged commit 8a293d2 into main Oct 7, 2026
3 checks passed
@Jammy2211
Jammy2211 deleted the feature/sparse-terms-oversampled-fine-grids branch October 7, 2026 10:55
@Jammy2211 Jammy2211 removed the pending-release PR queued for the next release build label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: SparseTerms oversample=q fine precision-operator and dirty-image grids

1 participant