Repository navigation
perf(v3): reuse rooted metadata deltas and bound streamed snapshot bodies - #54
Merged
Merged
Conversation
Ivanbeethoven
marked this pull request as draft
October 7, 2026 01:54
Ivanbeethoven
marked this pull request as ready for review
October 7, 2026 03:58
Ivanbeethoven
marked this pull request as draft
October 7, 2026 06:28
Ivanbeethoven
marked this pull request as ready for review
October 7, 2026 07:56
Rebase the complete PR #54 code and test delta onto current main as one verified signed commit. Preserve the exact previously reviewed source tree, including fixed-content metadata optimizations and controlled native initialization.
Ivanbeethoven
force-pushed
the
maint/mst2-native-head-with-projection
branch
from
October 7, 2026 08:44
e941a56 to
acf4e12
Compare
Git SinglePut preserves existing immutable objects. In the isolated test fixture, remove the owned object before fault injection or repair, read back the exact bytes, and exclude setup reads from HTTP counters. Preserve all corruption, missing-body, retry and cache assertions.
…56) Use primary PostgreSQL authority for fixed native snapshot sources and per-lease certificates. Atomically hand off installed metadata protection to durable sessions and leases, retain only roots on warm resolve, and retire session pins with the last lease. Recheck current lease, source and captured primary scope before each actual TreeFrame or shared raw block. Add real HTTP/PostgreSQL regressions; native validation and performance remain pending.
Resolve native all-target Clippy result_large_err by returning SnapshotError from authentication. Construct the identical HTTP error envelope at the two existing middleware exits inside the same request-ID scope. Preserve bearer and lease checks, codes, messages and every regression assertion; do not suppress the lint.
Ivanbeethoven
changed the title
…#57) Persist complete metadata lifetime bindings and private storage seals before payload installation. Verify exact generations at observed-DAG finalization and recovery; preserve existing v3 HTTP sessions and all original regression assertions. Isolate this repository until generation-aware collection and session adoption are implemented. Native Linux gates and performance remain pending.
Fix native all-target/all-feature Clippy chunks_exact_to_as_chunks. Keep the prior exact length/count guard, canonical re-encoding, per-entry parsing and every generation regression unchanged; no lint suppression. Native gate run 37610549234 confirmed the single failing lint.
Add explicit service init --yes --commit-time UNIX_SECONDS to reproduce initial root and materialized path parents for independent backends. Preserve current-time defaults and existing initialization persistence. Under the original advisory lock, reject fixed-input root commit/tree mismatches before object/ref changes. Frozen five-path source independently reviewed; nightly formatting and locked offline metadata passed. Native/PG/network push not run locally.
…receipts (#59) Add composite immutable lifetime history, a separate current watermark, persisted domain-bound preparation seals, explicit abort/coverage retirement and same-primary terminal replay. Preserve existing v3 serving, G1 NULL-domain receipts, original guards and bootstrap C1 blobs. No payload deletion, qualified collector, generation replacement or production adoption is enabled. Twelve frozen code/test paths independently source-reviewed; native validation and performance remain unmeasured.
Fix the two native batch root inserts to supply PostgreSQL now() for the existing required created_at column. Preserve graph locks, identities, ON CONFLICT semantics and all HTTP/generation assertions. Keep the reproducible bootstrap fixture consistent with the validated default import directory. Native run 37615218353 exposed 29 missing-timestamp failures and three invalid-fixture failures; source review, nightly formatting and locked offline metadata passed, native retest pending.
Add permanent lifetime/domain fences, a generation-qualified graph, bounded discovery and persisted exact GC operations. Physical payload removal, edges/counters, immutable history and APPLIED receipt commit together; fresh preparation requires exact prior removal proof and protects against ABA. Preserve current v3 serving, upstream updates and native timestamp/bootstrap fixes. Nineteen real PostgreSQL regressions added; final frozen source reviewed and formatting/offline metadata passed, native validation and performance pending. Qualified production sessions, collector and generic-to-qualified adoption remain closed.
Observe pg_locks through the transaction already holding the retention lock, and clear its statistics snapshot before each poll. Keep all original expiry, root-release and byte-count assertions. Stream repository test diagnostics before a possible CI cancellation. Native regression validation is pending.
This was referenced Oct 7, 2026
…verage (#63) Use one fixed-path metadata walk and current verified size/digest facts for lookup without whole-content reads. Preserve the original zero-body-read assertions and HEAD behavior. Correct three observed native fixture failures without changing production publication or payload guards. Native execution of this combined source remains pending.
Mint an opaque capability from complete immutable preparation evidence and freeze its durable identity and membership. PREPARING batches use bounded exact-member checks while preserving current primary, scope, state, physical lifetime and payload validation. COMMITTED replay and finalize retain their complete oracle. Native validation and measured performance remain pending.
…#64) Repair exactly diagnosed native capability regressions: mutate verification revision rather than a legal no-op; count the actual canonical radix DAG and batch partitions; inject one handoff corruption with the exact prepare guard restored transactionally; prove both legitimate generic/qualified refusal paths with complete state rollback and retain a real private physical-checker rejection. Production guards and original HTTP integrity/zero-read assertions remain unchanged. Native rerun is separate from source review.
Deduplicate concurrent same-digest cold projection loads, retain verified results for active participants after cache eviction, and preserve full SHA/map/slice validation. Bound registry keys and clean up exact last-owner gates without dropping large results under the global lock. Add ten failure/cancellation/eviction/capacity regressions. Native and performance evidence remain separate.
Move the unchanged cfg(test) SQL qualification regression module to the end of native_chunk_map.rs. This fixes the exact #78 all-target/all-feature Clippy items_after_test_module failure without a lint allow or any production or test assertion change. The raw streaming implementation merged in #79 is retained. Nightly formatting, diff and locked offline metadata checks pass; corrected native build/tests remain pending.
Change RawBlobReader access validation to borrow its exclusive owner. The byte producer is Send and deliberately need not be Sync; retaining a shared reference to the whole reader across an await made the actual Axum body future non-Send. This fixes exact #80 native compilation at snapshot_raw_blob.rs189/235 without adding a lock or weakening any first-poll, post-await, EOF or per-delivery access check. All13HTTP regression assertions remain. Formatting, diff and locked offline metadata pass; corrected native execution remains pending.
Sync latest main into v3 performance integration
Commit publication builds canonical local directory proofs and reuses unchanged fixed-root metadata. Default qualified-family serving reads bounded DIR windows and selected LOOKUP routes with current source revision facts; warm body callers avoid Git-tree projection. Permanent snapshot routes, exact publication and lease identities, source mutation invalidation, root ownership and bounded background recovery/GC remain enforced. The authority catalog validates exact registered physical families, full qualified shape and source revision triggers, including precise treatment of legitimate qualified foreign-key RI triggers. Direct lease lock contention maps to retryable HTTP 503. Publication-disabled formal MST/2 sessions retain fixed-root HEAD/body/OBJECT/CHUNK service. Migration 000200 and actual PostgreSQL/HTTP regression sources are integrated without raising existing projection limits. Independent review covers all 46 paths, all four corrected findings and every changed assertion site. Format, diff and locked offline metadata pass; native PostgreSQL execution and performance measurement remain separate.
#84) The exact native suite exposed two test accounting errors. Cold raw opens one source to earn its receipt and one for authenticated delivery; its revocation regression now requires both passes while retaining rejected-tail, zero range IO and exact byte accounting. The admission/cancellation source-builder regression now increments its existing loader counter on its real successful retry, preserving the required two source opens, credit refund, producer drop and chunk verification. Production behavior is unchanged.
…ma (#85) A TEMP table with the same name could poison native metadata repository initialization. Capture the actual permanent schema through pg_catalog, qualify the storage identity read during initialization and mutation/recovery barriers, and continue observing and comparing the actual current database/schema/OIDs/server/replica identity. A matching fake TEMP UUID cannot authorize another primary schema. Add a single-connection regression with TEMP present before construction, primary schema switching and restoration. Runtime verification query counts and existing integrity assertions are preserved.
The exact rooted metadata native check exposed unused private type reexports, a u64/u32 END assertion and two strict Clippy findings. Remove unused exports, compare the same exact count after lossless conversion, return existing typed SnapshotError from JSON/cursor helpers and name the unchanged proof page type. Response conversion keeps the same MST/2 HTTP error mapper at the caller boundary; proof bounds and validation are unchanged.
Sync upstream 0.42.24 into v3 performance integration
The exact native check exposed the remaining unused directory-window and lookup-batch reexports in the intermediate rooted module after the outer family reexports were removed. Remove those two unused names and retain the used lookup status reexport. Reader type definitions, methods, proof bounds, HTTP behavior and every test assertion are unchanged.
Strict native Clippy identified an unnecessary cloned single-item slice in the rooted metadata donor fixture. Pass the same immutable payload with std::slice::from_ref. Preserve every donor certificate, source verification, delta reuse and fixed-root integrity assertion; production code is unchanged.
Ivanbeethoven
changed the title
Sync upstream 0.42.25 into v3 performance integration
…actual deadline (#91) Bound persisted v3 chunk maps and exact receipt generations with primary database owners, capacity admission, resumable collection and final transport-clone retention. Pending cold and warm backend opens, input polls and receipt creates now terminate under the actual remaining owner deadline; empty fragments grant no renewal. Add actual HTTP/storage/database test sources for expiry, withheld final bytes, refund and safe replay. Owned nightly format, canonical candidate whitespace and SQL lexical checks pass; native build/tests, database execution and commit-update performance measurements remain unrun. Truthful backend capabilities are a separately reviewed source-only follow-up dependency.
…es (#92) Missing rooted blob facts now hash a complete source stream under fixed consumer credit, removing full-file materialization while preserving exact size, digest and EOF checks before persistence. Local and memory backend streams split visible items lazily without copying; capability discovery reports raw, chunk and full hydration support from the actual backend without IO. Eleven added test sources retain previous assertions and the actual merged chunk-owner deadline work. Seven owned Rust format/parse and canonical whitespace checks pass; native/database/performance execution remains unrun.
…ding (#93) Rooted cold and zero-delta preparation referenced value from unnamed unnest outputs in both graph-member checks, causing the exact native column-value failures. Explicitly bind the four array output columns. MTP2 integer serialization also divided numeric values before truncation, rounding u64MAX into wrong bytes; use the exact integer quotient instead. Enhance existing real database test sources for persisted rooted plan/payload reuse and independent Rust byte encoding at integer boundaries, retaining all prior full-byte/page-id and rejection assertions. Owned nightly format/parse, canonical whitespace and exact SQL lexical preservation checks pass; PostgreSQL/native/performance execution remains pending. Other CI failure groups are separate.
Sync accepted latest main ancestry into v3 integration
…iguration (#95) Use production rooted metadata provisioning for ordinary snapshot, view operations, and native push fixtures. Retain schema ownership, generic history behavior, native queue timing, and all existing test assertions. Validation is source-only: owned nightly format checks and Git whitespace checks; native builds, database tests, and performance runs are deferred.
) The UN30 facade fixture now supplies its actual per-test database config to full storage assembly, so the rooted Q pool shares the correct database. The forged namespace insertion must match its exact BEFORE INSERT registration guard and roll back, preserving every route/source state assertion. The durable scope-drift test now expects the shared family-selection contract: 502 INTEGRITY_ERROR with retryable=false for both descriptor and renew, before a G/Q repository is selected. It retains stream error/no-END, scope restoration and no-IO assertions. Three owned nightly format/parse and whitespace source checks pass; native execution remains unrun.
The exact PR91 repository gate stopped at compilation: common IoOrbitError conversion missed the two retention variants, fourteen test connection borrows outlived temporary MonoStorage owners, and one spawned install borrowed fixture storage. Explicitly convert both concrete retention errors through existing MegaError::ObjStorage without changing their specialized Snapshot classifications, bind each connection owner, and move an owned Storage clone sharing the original Arc<OnceCell> repository cache into the install task before borrowing that same cached repository. All original assertions, deadlines, cancellation and final Bytes retention checks remain byte-for-byte preserved outside these lifetime edits. Owned nightly format and exact source/tree checks pass; native build, Clippy and repository tests remain pending.
Ivanbeethoven
changed the title
The exact PR97 and integrated c1ff repository gates both stop at two Clippy collapsible_if errors. Collapse only those nested conditions: nonempty raw bytes followed by an existing chunk map, and a successful cancelled-install mutex lock followed by its strict 64-entry cap. Preserve short-circuit order, lock scope, ownership, progress error propagation and every cancellation/stream assertion. Two owned nightly format/parse checks, exact whole-file forward/inverse reconstruction and the complete unchanged-outside-paths tree proof pass. Native Clippy, build and repository tests remain unrun for this candidate.
) Database bootstrap currently fails with PostgreSQL 42601 at Original(10355): the first unparenthesized CASE in a PL/pgSQL IF exposes its internal THEN as the end of the IF condition. Parenthesize that CASE and the identical later map guard expression. Preserve both 512 MiB capacity checks, state-dependent 256 KiB/1 MiB reserves, error messages and every other migration byte. Exact original source-offset and whole-file inverse proofs plus whitespace and unchanged-tree checks pass. The cancelled native run has no completed test summary; this candidate has not executed SQL or native tests.
…100) Qualified metadata requests previously retained every completed reader, so the 65,536-owner history cap eventually blocked new requests. Reclaim at most 64 previously committed terminal owners without roots before admission, and fence every page/source/finish operation with UUID plus a monotonically issued generation stored in one high-water row per Q. Same-transaction terminal owners remain until their deferred completion commits. Add a trusted transactional forward migration, remove UUID-only function overloads, and preserve permanent SID, source, certificate and lease history. Nine default database regression test sources cover fresh/old-Q/Q-less/tampered upgrades, real HTTP requests, stale actual-reader replay, rollback issuance, maximum issuance and the 64-owner deletion budget; a new ignored capacity soak issues 65,537 real HTTP requests. Owned nightly formatting, complete candidate whitespace, patch checks and offline source proofs pass. Native build/tests/Clippy, PostgreSQL execution and performance measurements remain pending; this package claims source validation only.
…des (#101) Native rooted plan admission failed because PL/pgSQL parent and child locals conflicted with unqualified graph-query columns. Qualify both columns and repair the migration shape sampler to deparse under pg_catalog,pg_temp with parameterized caller-path restoration. Share trusted forward migration logic between 400 and new 500: authenticate exact 6d/900e/current implementation, captured historical templates, authority/full physical catalogs, scope and canonical namespace identity; replace the decoder on old 6d and the three changed functions on 900e while preserving its relations, function OIDs, reader high-water and retained ownership/history. Admit the exact known Q-visible legacy deparse shape only for the authenticated 900e policy with no actual Q. Handle the legitimate 900e state with a missing 400 ledger without replaying reader DDL. Move the old reader capture into migration verification and reuse the forward-only template renderer in owned fixtures. Add six PostgreSQL regression sources for active/terminal readers, nonzero issuance, all anchors/OIDs, old SID lookup, Q-less canonical/known legacy policies, missing ledger, tampering and idempotence. Add test-only underlying install error output; the production 503 remains unchanged. Owned nightly formatting, candidate whitespace, historical-source equivalence and patch checks pass. Native build/tests/Clippy, PostgreSQL execution and performance measurements remain pending; this is a source-only submission.
Cache the immutable compiled qualified-family implementation fingerprint in a process-local OnceLock<[u8; 32]>, preserving the existing Vec return API. Keep all 15 source components, domain separator, length prefixes and digest bytes unchanged; every database catalog, shape, policy, identity and physical-stamp check still runs. Owned nightly formatting, offline source equivalence and candidate/patch checks pass. Native typecheck, Clippy, PostgreSQL tests and performance measurements remain pending.
Rooted PostgreSQL resolve failed its exact descriptor handoff because SQL encoded descriptor u16 fields in big endian while mst2-codec 0.3.1 writes little endian. Encode version, flags and UTF-8 byte length with the locked codec order. Add independent transactional migration600; freeze400/500 at authenticated87b so ledger replay cannot downgrade the repaired family. Authenticate unchanged core/Q catalogs, complete physical shape, scope and exact identities before every no-op or forward repair. Preserve existing OIDs, issuance, anchors, readers, source roots and canonical sessions. Add four real PostgreSQL regression sources for bytewise root/ASCII/256-byte/UTF-8 descriptors, authenticated87b upgrades and missing ledgers, Q-less provisioning, idempotence and descriptor tampering; verify compiled-family400/500 replay without a downgrade in the same strong fixture. Correct the complete20-name migration-order assertion while preserving alias DB behavior; historical900e replay includes600. Add a focused24-test native gate with exact inventory presence checks before the retained full repository suite. Formatting, historical-source equivalence, exact function replacement sets, candidate whitespace and bidirectional patch checks pass. Native compile, PostgreSQL regression execution and performance measurements for this candidate remain pending.
) The native 64-owner backlog regression failed at its first bulk ACTIVE-owner commit, before any explicit prune assertion, because the shared deferred ownership guard rejected the final state. Source inspection supports the 60-second reader deadline as the most likely explanation; the runtime log does not identify which guard predicate failed. Complete each of the 65 admitted readers immediately in the same transaction, preserving the terminal transaction fence and deferred validation. After its first successful commit, assert 65 FINISHED owners, no ACTIVE owners or REQUEST/READER anchors, and an issuance high-water advance of 65. Preserve the same-transaction prune=0 and later prune=64 then1, with retained-row counts1 then0 and all existing tests. Production SQL, deadlines, ownership guards, implementation fingerprints and the focused24-test workflow remain unchanged. Nightly formatting, scoped source checks, candidate whitespace and bidirectional patch checks pass; native execution of this new fixture remains pending.
genedna
approved these changes
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed-SID ScorpioFS / Worktree v3 sessions keep their published source while later commits reuse certified unchanged Git subtrees and install bounded metadata deltas. Sparse directory and lookup windows, on-demand body/range streams, bounded caches and generation GC preserve source, size, chunk, whole-SHA, actual-EOF, ownership and deadline checks. Reader admission reclaims at most 64 committed terminal owners while preserving active/deferred owners, persistent issuance and permanent source/certificate history. The compiled implementation fingerprint is cached once per process; dynamic database catalog, physical shape and identity checks remain.
PostgreSQL MSD2 descriptors now match the locked Rust codec's little-endian integers and UTF-8 scope byte lengths. Authenticated transactional migration600 repairs captured v3 families without rewriting session, reader, anchor, issuance, source history or function identities. Earlier migrations authenticate upgraded families without rolling them back during missing-ledger replay; tampered families are rejected.
The previous exact source tree's native gate completed with 23 passed and one failed. Formatting, all-target/all-feature Clippy, build and build-tests passed. All descriptor, historical-upgrade, alias, source-reuse and incremental-cost regressions passed. The sole failure occurred at the backlog fixture's first bulk ACTIVE-reader commit, before its prune assertions; the deferred ownership guard rejected the transaction. The final fixture correction completes each admitted reader immediately in the same transaction, then checks 65 committed FINISHED owners, no ACTIVE owners or reader roots, and issuance+65. It preserves the same-transaction prune=0, subsequent prune=64 then1, and remaining-row counts1 then0. Production SQL, deadlines and integrity guards are unchanged.
Delivery: signed merge
5ff2db2888c6f9fb775ba2e1bfb091c057681e1a, tree82efe82de463ba17e72a08fcc4518633859fcdd3, includes checked maind572485aac5f6e3e79d9fdca2d7809e2cebf99b9(59 ahead, zero behind). All added integration commits have valid GitHub signatures. All 180 changed paths are source, necessary tests or the existing workflow. The final single-file fixture passed independent exact-source review and nightly formatting; the automatic native run is validating the current source. Its current exact-tree native gate terminated with22 passed and2 failed. The backlog correction passed. Remaining failures are a4-second initial admission-barrier timeout before stale-UUID assertions and a PostgreSQL40P01 lock cycle during compiled-family missing400/500 migration replay. Their final assertions were not reached; minimal corrections are in progress. Previous results are not transferred to the new tree. The full repository suite remains enabled after the focused gate, with isolated dependencies and cleanup.Client runtime evidence separately covers the five necessary performance functions, including a real kernel mount regression. Large-scale performance testing has not started, and no superiority over Git is claimed. Plans, logs and evidence remain outside the repositories. Final main merge requires approval from someone other than the last pusher; no force push or rule bypass is used.