feat(divan): run each benchmark in its own process

[not-matthias]

TLDR: Benchmarks in one bench binary share a process, so one benchmark can change the allocator state for every benchmark after it. With CODSPEED_PROCESS_ISOLATION=1, divan_compat now re-executes the binary once per benchmark (simulation mode only).

• The parent keeps CLI filtering and URI generation, then spawns current_exe() with __CODSPEED_DIVAN_BENCH_ID=<entry>[:<arg>].
• The child runs only that entry; its hooks and callgrind dump use the child's PID.
• No changes to cargo-codspeed or the runner: they rely on the existing --trace-children=yes.
• New allocator_ripple bench: three identical calloc(4 MiB) benchmarks. The first one frees an mmapped chunk, which raises glibc's mmap threshold, so the later ones get heap memory that must be cleared with memset.

Measured (allocator_ripple, callgrind Ir, same on reruns):

Bench	Shared process	Isolated
zeroed_buffer[1]	821	543
zeroed_buffer[2]	128,070	543
zeroed_buffer[3]	229,939	543

Variance:

executor	set	runs	benches	median CV %	p90 CV %	max CV %	CV > 1%	CV > 0
valgrind	before (prod, last 10)	10	215	0.256	6.593	14.381	67	158
valgrind	isolated (staging, 10)	10	218	0.000	0.000	0.188	0	7
memory	before (prod, last 10)	10	215	0.000	0.000	0.000	0	0
memory	isolated (staging, 10)	10	218	0.000	0.000	0.000	0	0

Review notes

• Opt-in via env var for now. Each benchmark pays a process start-up under valgrind (about 0.28 s each on a small binary).
• Untested: walltime (divan_fork), memory-mode tracking of the child processes, criterion and bencher.
• Doesn't help with binary layout or code-generation shifts.

Refs COD-3799

[codspeed]

Merging this PR will degrade performance by 6.69%

⚠️ 59 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 31 improved benchmarks
❌ 37 regressed benchmarks
✅ 526 untouched benchmarks
🆕 12 new benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Memory	fib_in_thread[32]	424 B	760 B	-44.21%
❌	Memory	fib_in_thread[30]	424 B	760 B	-44.21%
❌	Memory	fib_in_thread[31]	424 B	760 B	-44.21%
❌	Memory	fib_in_thread_bench_local[31]	424 B	760 B	-44.21%
❌	Memory	fib_in_thread_bench[32]	424 B	760 B	-44.21%
❌	Memory	fib_in_thread_bench_local[32]	424 B	760 B	-44.21%
❌	Memory	fib_in_thread_bench[30]	424 B	760 B	-44.21%
❌	Memory	fib_in_thread_bench[31]	424 B	760 B	-44.21%
❌	WallTime	mul	2 ns	3 ns	-33.33%
❌	Simulation	Iterative	121.5 ns	177.1 ns	-31.38%
❌	Simulation	Iterative[20]	121.5 ns	177.1 ns	-31.38%
❌	Simulation	Iterative[21]	122.8 ns	178.4 ns	-31.15%
❌	Simulation	generate_combinations[7]	21.4 µs	27.6 µs	-22.35%
❌	Simulation	recursive_memoized[HashMap<u64, u64>, 10]	8.6 µs	10.8 µs	-19.74%
❌	Simulation	add_two_integers[(42, 13)]	233.8 ns	288 ns	-18.81%
❌	Simulation	add_two_integers[(255, 255)]	233.8 ns	288 ns	-18.81%
❌	Simulation	add_two_integers[(1024, -1024)]	295.2 ns	349.4 ns	-15.5%
❌	Simulation	add_two_integers[(65535, -65535)]	307 ns	361.1 ns	-15%
❌	Simulation	generate_combinations[8]	28.8 µs	33.9 µs	-14.97%
❌	Simulation	generate_combinations[9]	35.7 µs	41.8 µs	-14.51%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-3799-explore-process-per-benchmark-in-codspeed-rust (304fa86) with main (832f252)}

[avalanche-staging]

Congrats! CodSpeed is installed 🎉

🆕 444 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

⚠️ This run is associated to a commit that was force-pushed

It won't be associated with its branch history. Be careful when comparing its performance to other runs.

Detected benchmarks

• with_input[5] (WallTime): < 1 ns
• with_input[5] (WallTime): < 1 ns
• with_input[5] (WallTime): < 1 ns
• bench_explicit_measurement (WallTime): < 1 ns
• sum_fold (WallTime): 307 ns
• build_vec (WallTime): 213 ns
• print_env_hello (Simulation): 15.2 µs
• permutations[3] (Simulation): 12 µs
• n_queens_solver[8] (Simulation): 1.1 ms
• iterative[0] (Simulation): 224.6 ns
• iterative[10] (Simulation): 245.4 ns
• recursive[10] (Simulation): 1.7 µs
• permutations[5] (Simulation): 71.5 µs
• iterative[5] (Simulation): 234.1 ns
• iterative[30] (Simulation): 290.4 ns
• count_set_bits[65535] (Simulation): 185.1 ns
• find_highest_set_bit[0] (Simulation): 172.3 ns
• recursive[20] (Simulation): 139.9 µs
• permutations[6] (Simulation): 399.7 µs
• div (Simulation): 178.1 ns
• ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

---

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Adds process isolation option for benchmark runs.

Fix the hang with existing Tokio worker pools and satisfy the logging rule before merging.

Findings

1.  Existing async workers disappear ▶
2.  Warning bypasses required logging ▶
3.  Isolated runs are never tested ▶

Fix with agent prompt

### Issue 1
crates/codspeed/src/isolation.rs:78-79
With `CODSPEED_PROCESS_ISOLATION=1`, a Criterion benchmark that creates a multithreaded Tokio `Runtime` before `bench_function` can hang. `fork()` copies the runtime but not its worker threads. A closure using `b.to_async(&runtime)` and awaiting a spawned task then waits for workers that no longer exist. The parent also waits forever in `waitpid`.

Start a fresh executable for each benchmark, or keep this path disabled for benchmarks with existing worker threads.

### Issue 2
crates/codspeed/src/isolation.rs:39-42
The new Windows warning uses `eprintln!`. The repository rule says never to use `println!` or `eprintln!` in Rust and to use tracing macros instead. Replace this warning with `warn!` to satisfy the requirement before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 3
crates/codspeed/src/isolation.rs:75
No automated test or CI run enables `CODSPEED_PROCESS_ISOLATION`, so the new child path is never exercised. Adding `allocator_ripple` only tests the shared-process default. Changes that lose child results or hide child failures can therefore pass CI.

Add an isolated analysis run that checks all selected cases report results and that a failing child makes the parent fail.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds opt-in, fork-per-benchmark execution to the Divan and Criterion analysis adapters. The child emits measurement metadata, runs one benchmark, and returns its exit status to the parent.

• Adds allocator_ripple to demonstrate allocator effects between benchmarks.
• Sends CI uploads to staging.
• Existing Tokio worker pools can leave isolated benchmarks hanging.
• The new warning violates the logging rule, and CI never enables isolation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Parent selects benchmark and builds URI] --> B{Isolation enabled?}
  B -->|No| C[Run benchmark in parent]
  B -->|Yes| D[Fork child]
  D --> E[Child emits metadata]
  E --> F[Child runs benchmark]
  F --> G[Child exits with success or failure]
  D --> H[Parent waits]
  G --> H
  H --> I{Child succeeded?}
  I -->|Yes| A
  I -->|No| J[Parent exits with child status]

Loading

_{Reviews (1) · Last reviewed commit: 3a215de · Reviewed by Greptile}