fix(snapshot): prevent lost OCI index updates during concurrent saves

[suhasagg]

Related Issue

Fixes #1917

This PR addresses the concurrent Snapshot::save() lost-update race reported in issue #1917.

The fix introduces exclusive filesystem locking around OCI snapshot persistence to prevent concurrent writers from silently overwriting each other's snapshot references.

Summary

Fixes a potential lost-update race condition when multiple threads or processes concurrently save snapshots to the same OCI layout directory.

Previously, Snapshot::save() could read the existing index.json, independently modify its manifest list, and atomically replace the index without coordinating with other writers.

Although the file replacement was atomic, the complete read-modify-write transaction was not synchronized.

Consequently, concurrent snapshot saves could overwrite one another's index updates, causing successfully saved snapshot tags to disappear from index.json.

Root Cause

Atomic file replacement protects against partially written files but does not prevent lost updates.

Example race scenario:

1. Writer A reads the existing OCI index.

2. Writer B reads the same index.

3. Writer A adds its snapshot descriptor and replaces index.json.

4. Writer B adds its descriptor to the previously read index and replaces index.json.

5. Writer A's descriptor is lost, even though both save operations may have completed successfully.

Root cause: Missing synchronization around the OCI index read-modify-write transaction.

Implementation

1. Add Exclusive Filesystem Locking

• Introduce fs2 = "0.4" for filesystem locking.

• Create a stable .hyperlight-index.lock file inside the OCI layout directory.

• Acquire an exclusive lock before OCI layout validation and index modification.

• Hold the lock throughout snapshot persistence and atomic index replacement.

• Release the lock automatically through RAII, including error paths.

Why use a separate lock file?

index.json is atomically replaced during snapshot saves. Locking the index file itself would not provide a stable synchronization target across replacements.

The dedicated lock file provides a consistent filesystem object for cooperating writers.

2. Preserve Existing Snapshot Semantics

The implementation preserves:

• Atomic index.json replacement

• Tag-based manifest replacement

• OCI layout marker validation

• Marker-before-index write ordering

• Independent locking for separate OCI layout directories

The lock serializes writers targeting the same OCI layout without introducing a global lock.

3. Add a Concurrency Regression Test

Test: concurrent_snapshot_saves_preserve_all_tags

The test:

1. Creates a shared snapshot and OCI layout directory.

2. Starts 8 concurrent writer threads synchronized using a barrier.

3. Saves snapshots using unique tags.

4. Waits for all writer threads to finish.

5. Reads the resulting index.json.

6. Verifies that all 8 tags are preserved.

This regression test detects missing manifest entries after concurrent saves.

Validation

The following checks were executed successfully on Ubuntu:

Validation | Result -- | -- cargo check -p hyperlight-host | PASSED cargo test -p hyperlight-host --lib | 485 passed, 0 failed, 15 ignored Snapshot file test module | 167 passed, 0 failed Eight-writer concurrency regression test | PASSED Concurrency stress test | 20/20 runs passed git diff --check | PASSED

Reviewer Notes

I would appreciate maintainer feedback on:

• Filesystem locking strategy and cross-platform considerations.

• Lock-file naming and lifecycle.

• Whether additional independent-process regression coverage should be included.

• Whether the lock should be held across the complete snapshot persistence operation or narrowed to the index transaction.

The implementation is intentionally focused on preventing lost OCI index updates while preserving existing snapshot persistence behavior.

Thank you for reviewing!