Expand self-hosted runner diagnostics for GHES, orphaned networks, and ARC/DinD logs by Copilot · Pull Request #9207 · github/gh-aw-firewall

Copilot · 2026-09-29T18:48:54Z

Updates runner-doctor coverage for three newly identified failure modes: GHES /meta responses in gh-proxy mode, orphaned awf-net networks, and read-only ARC/DinD streaming-log paths.

Diagnostics catalog
- Document C5’s partial /meta shim mitigation and remaining upstream limitation.
- Document B27’s orphan-network recreation behavior and actionable inspection probe.
- Add A28 for writable sandbox/agent streaming-log paths, including unresolved gh-aw compiler work.
Doctor workflows
- Mirror catalog, lookup, probe, and unresolved-item updates in the shared workflow and portable agent.
- Add regression assertions ensuring all three representations remain aligned.
Generated artifacts
- Regenerate the self-hosted runner doctor lock with the repository’s pinned compiler.

Example orphan-network probe:

docker network inspect awf-net --format '{{json .Options}}'

An empty options map without com.docker.network.bridge.name identifies the affected orphaned network.

Fixes 🩺 Runner Doctor Updatescan 2026-09-27: GHES /meta shim, orphaned awf-net, arc-dind streaming log #9201

Copilot

Copilot review overview

🟡 Changes recommended

Canonical registry outputs and B27/C5 records are incomplete, the A28 probe violates the read-only contract, and portable routing is missing entries.

Review effort: Balanced
Findings: 1 · 1 · 2

Open (4)

Regenerate diagnostics consumers after adding A28 finding · New
Add B27, C5, and A28 parity assertions · New
Add missing C5 and A28 routes to portable playbook · New
Check container attachments before classifying network as unoccupied · New

What changed in this PR

Expands runner-doctor diagnostics for ARC/DinD streaming logs, orphaned networks, and GHES metadata handling.

Changes:

Adds A28 and updates B27/C5 guidance.
Mirrors diagnostics across workflow and portable-agent artifacts.
Updates regression assertions and generated workflow metadata.

File	Description
`scripts/ci/self-hosted-runner-doctor-workflow.test.ts`	Extends alignment assertions.
`docs/diagnostics/findings/runner/A28.json`	Adds canonical A28 finding.
`.github/workflows/shared/self-hosted-failure-modes.md`	Updates shared diagnostic catalog.
`.github/workflows/self-hosted-runner-doctor.md`	Adds probes and symptom routing.
`.github/workflows/self-hosted-runner-doctor.lock.yml`	Updates generated workflow hash.
`.github/agents/self-hosted-runner-doctor.md`	Updates portable diagnostic guidance.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+{
+  "$schema": "../../schema.json",
+  "id": "A28",


 - `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory → A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086)
 - `docker network connect --alias <name> awf-net <service_container>` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25 (service container must join `awf-net` for direct protocol access while the agent stays isolated; documented in github/gh-aw-firewall#8085)
- `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817)
+- `a network with name awf-net exists but was not created for project`, or a missing bridge in legacy iptables mode → B27 (inspect `docker network inspect awf-net --format '{{json .Options}}'`; an empty options map without `com.docker.network.bridge.name` identifies an unoccupied orphan; fixed in github/gh-aw-firewall#9130)


 - `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory → A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086)
 - `docker network connect --alias <name> awf-net <service_container>` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25 (service container must join `awf-net` for direct protocol access while the agent stays isolated; documented in github/gh-aw-firewall#8085)
- `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817)
+- `a network with name awf-net exists but was not created for project`, or a missing bridge in legacy iptables mode → B27 (inspect `docker network inspect awf-net --format '{{json .Options}}'`; an empty options map without `com.docker.network.bridge.name` identifies an unoccupied orphan; fixed in github/gh-aw-firewall#9130)


lpcox · 2026-09-29T19:37:35Z

@copilot address review feedback

Copilot · 2026-09-29T19:45:39Z

@copilot address review feedback

Addressed the review feedback in commit 6f94611. The A28 probe is read-only, B27 now checks for attached containers, and generated diagnostics plus portable routing parity are updated.

github-actions · 2026-09-29T19:49:51Z

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-09-29T19:50:13Z

Documentation Preview

Documentation has been built for this PR.

Download preview artifact

To view locally:

Download the docs-preview-pr-9207 artifact from the workflow run
Unzip and open index.html in your browser

Built from commit 906d32e

github-actions · 2026-09-29T19:53:47Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	92.66%	92.67%	📈 +0.01%
Statements	91.15%	91.16%	➡️ +0.01%
Functions	89.18%	89.18%	➡️ +0.00%
Branches	84.29%	84.29%	➡️ +0.00%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-09-29T19:54:57Z

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-09-29T19:54:57Z

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-09-29T19:54:59Z

🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation...

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

clients2.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "clients2.google.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-09-29T19:54:59Z

Smoke Cloud Hypervisor completed. Cloud Hypervisor + Copilot passed.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor

github-actions · 2026-09-29T19:55:00Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-09-29T19:55:00Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-09-29T19:55:01Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-09-29T19:55:02Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

github-actions · 2026-09-29T19:55:03Z

✅ Build Test Suite completed successfully!

Generated by Build Test Suite for #9207

github-actions · 2026-09-29T19:55:05Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-09-29T19:55:07Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-09-29T19:55:08Z

❌ Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

github-actions · 2026-09-29T19:55:14Z

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

🛡️ Egress verdict from Smoke Copilot Network Isolation

github-actions · 2026-09-29T19:55:34Z

❌ Smoke Claude failed

Generated by Smoke Claude for #9207

github-actions · 2026-09-29T19:58:01Z

Smoke Copilot — PASS ✅

MCP: ✅ (latest PR: "Use routing-capable images in model-routing smoke")
github.com: ✅ (HTTP 200)
File write/read: ✅
cc @Copilot @lpcox

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-09-29T19:58:10Z

Smoke Test: Copilot BYOK (Direct) Mode ✅

Test	Result
GitHub MCP connectivity	✅
github.com (HTTP 200)	✅
File write/read	✅
BYOK inference path	✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY → api-proxy → api.githubcopilot.com)

Status: PASS

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-09-29T19:58:34Z

Smoke Test: Cloud Hypervisor + Copilot

list_pull_requests (merged, limit 1): PASS — call succeeded, returned data
curl https://github.com: PASS (200)
Write/read /tmp/gh-aw/agent/smoke-cloud-hypervisor-*.txt: PASS
curl (example.com/redacted) (expect blocked): PASS (000)

All checks PASS.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor
_{Add label ready-for-aw to run again}

github-actions · 2026-09-29T19:58:38Z

OTEL smoke test

✅ S1 Module: otel.js loads (exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, etc.)
✅ S2 Tests: 3 OTEL suites, 68/68 passed
✅ S3 Env forwarding: trace context in env-passthrough.ts; OTEL vars in api-proxy-env-config.ts
✅ S4 Token tracker: onUsage hook present in token-tracker-http.js
⚠️ S5 Diagnostics: no exported spans checked (post-step not run yet); not treated as failure

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-09-29T19:58:48Z

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (api.github.com): HTTP 403 status returned (reachable)
✅ Blocked domain (example.com): blocked (TLS cert error from proxy)

Overall: PASS — cc @lpcox

🛡️ Egress verdict from Smoke Copilot Network Isolation
_{Add label ready-for-aw to run again}

github-actions · 2026-09-29T19:58:52Z

Services smoke test

Redis PING: ✅ PONG
pg_isready: ✅ accepting connections
psql SELECT 1: ✅ 1

Overall: PASS

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-09-29T19:59:51Z

Chroot Version Comparison

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.14	Python 3.12.14	✅
Node.js	v24.21.0	v22.23.2	❌
Go	go1.22.12	go1.22.12	✅

Result: Not all tests passed (Node.js version mismatch), so the smoke-chroot label was not added.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

github-actions · 2026-09-29T19:59:56Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	❌	N/A	❌ FAIL
Bun	hono	❌	N/A	❌ FAIL
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	❌	N/A	❌ FAIL
Deno	std	❌	N/A	❌ FAIL
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	pass	✅ PASS
Node.js	execa	✅	pass	✅ PASS
Node.js	p-limit	✅	pass	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 6/8 ecosystems passed — FAIL

Failure details

Bun / Deno: The install scripts (curl ... | bash / | sh from bun.sh and deno.land) were blocked by the sandbox with "Permission denied and could not request permission from user". Bun and Deno were not installed and no tests were run.
Java (passed with a workaround): ~/.m2/repository was not writable ("Could not create local repository"), so I used -Dmaven.repo.local=/tmp/gh-aw/agent/m2.
All 8 repositories cloned successfully.

Generated by Build Test Suite for #9207 · copilot · auto · 29.1 AIC · ⊞ 11.8K · ◷
_{Add label ready-for-aw to run again}

Initial plan

c2b64ad

Copilot AI assigned Copilot and lpcox Sep 29, 2026

Copilot started work on behalf of lpcox September 29, 2026 18:49 View session

Copilot AI linked an issue Sep 29, 2026 that may be closed by this pull request

🩺 Runner Doctor Updatescan 2026-09-27: GHES /meta shim, orphaned awf-net, arc-dind streaming log #9201

Closed

docs: update runner doctor findings

7f7e150

Copilot AI changed the title ~~[WIP] Update runner doctor documentation for 2026-09-27~~ Expand self-hosted runner diagnostics for GHES, orphaned networks, and ARC/DinD logs Sep 29, 2026

Copilot finished work on behalf of lpcox September 29, 2026 19:00

Copilot AI requested a review from lpcox September 29, 2026 19:00

lpcox marked this pull request as ready for review September 29, 2026 19:31

Copilot AI balanced review requested due to automatic review settings September 29, 2026 19:32

Copilot started reviewing on behalf of lpcox September 29, 2026 19:32 View session

Copilot AI reviewed Sep 29, 2026

View reviewed changes

Copilot started work on behalf of lpcox September 29, 2026 19:38 View session

fix: address runner doctor review feedback

6f94611

Copilot finished work on behalf of lpcox September 29, 2026 19:46

lpcox added the ready-for-aw label Sep 29, 2026

lpcox had a problem deploying to aoai-model September 29, 2026 19:55 — with GitHub Actions Failure

github-actions Bot added the smoke-copilot label Sep 29, 2026

github-actions Bot added the smoke-copilot-byok label Sep 29, 2026

github-actions Bot added the smoke-cloud-hypervisor label Sep 29, 2026

github-actions Bot added smoke-copilot-network-isolation Copilot network-isolation egress smoke test smoke-services labels Sep 29, 2026

lpcox deployed to aoai-model September 29, 2026 19:59 — with GitHub Actions Active

lpcox merged commit 8f9b21b into main Sep 29, 2026
128 of 135 checks passed

lpcox deleted the copilot/runner-doctor-updates-2026-09-27 branch September 29, 2026 21:41

github-actions Bot mentioned this pull request Sep 29, 2026

Centralize sensitive-path masking across runtimes #9220

Merged

Conversation

Copilot AI commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

lpcox commented Sep 29, 2026

Uh oh!

Copilot AI commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Documentation Preview

Uh oh!

github-actions Bot commented Sep 29, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Smoke Test: Copilot BYOK (Direct) Mode ✅

Uh oh!

github-actions Bot commented Sep 29, 2026

Smoke Test: Cloud Hypervisor + Copilot

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Chroot Version Comparison

Copilot AI commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading