[frankreyesgarcia]

Adds the DeepSeek benchmark dataset under benchmark/runs-opencode-deepseek-docker-v3/: the 60 top-package cases (10 per ecosystem), each run with and without yul's hook, 3 repetitions per condition.

• Model: deepseek/deepseek-flash (DeepSeek V4.1 Flash) via OpenCode 1.18.34, with the hook provided by opencode-yul 0.0.21.
• Isolation: every run gets its own fresh Docker container that mounts only that run's working directory, with a throwaway $HOME. No cache is shared between runs, and nohook runs have no way to see yul.
• Environment: the sandbox image has the same toolchains as benchmark/Dockerfile (go, cargo, maven, uv).
• Completion: 359/360 runs finished. cargo-top-09-winapi-x86_64-pc-windows-gnu nohook run-01 hit the 15-minute watchdog. Total cost was $1.98.

This PR contains data only. The harness and analysis code that produced it are on the opencode-deepseek branch: run_case_opencode_deepseek.sh, run_all_opencode_deepseek.sh, Dockerfile.opencode-sandbox and analyze_runs.py.

analysis_report.txt and analysis_rows.json are the output of analyze_runs.py, pooled over the three repetitions:

                | Nohook (RQ1)          | Hook (RQ2)
                | Valid  Already latest | Valid  Already latest  Mitigated  Rate
Cargo           | 23     21/23          | 24     22/24           2/2        100%
GitHub Actions  | 30     0/30           | 30     0/30            27/30      90%
Go              | 23     20/23          | 23     22/23           1/1        100%
Maven           | 29     10/29          | 30     8/30            22/22      100%
npm             | 8      8/8            | 13     13/13           0/0        n/a
PyPI            | 21     21/21          | 20     20/20           0/0        n/a
All             | 134    80/134         | 140    85/140          52/55      95%

The 3 hook runs that were not mitigated are all GitHub Actions. In each, the agent rejected yul's SHA-pinned suggestion, believing that version did not exist, and never rewrote the workflow.

Left out of the data: build output and caches the agents produced (node_modules, target, .venv, compiled binaries, ~/.m2). A GITHUB_TOKEN that three runs dumped via env has been redacted.

🤖 Generated with Claude Code

[algomaster99]

Cargo

nohook

Run	Files	Why it was excluded
cargo-top-02-cfg-if/nohook/run-02	Cargo.toml, transcript.jsonl	[dependencies] is empty. The agent wrote plain #[cfg] modules and never used cfg-if.
cargo-top-08-lazy_static/nohook/run-01	Cargo.toml, src/main.rs, transcript.jsonl	[dependencies] is empty. It uses std::sync::LazyLock instead.
cargo-top-08-lazy_static/nohook/run-02	same files	Same as run-01.
cargo-top-08-lazy_static/nohook/run-03	same files	Same as run-01.
cargo-top-09-winapi-x86_64-pc-windows-gnu/nohook/run-01	Cargo.toml (there's no transcript.jsonl)	The run was cut off by the 15-minute timeout. Its partial Cargo.toml already has windows = "0.62.2", but there's no transcript to judge it from.

• run-01 (cargo-top-05-syn/nohook/run-01/Cargo.toml): it settled on syn = "2" in its reasoning and wrote it straight into the workspace Cargo.toml, with no cargo search or crates.io lookup.
• run-02 (cargo-top-05-syn/nohook/run-02/Cargo.toml): same pattern. Its only network activity was curl checks to see whether crates.io was reachable, never to look up a version.

1. Straight from memory. 2) uses curl to see if crates.io was reachable.

hook

cargo-top-09-winapi-x86_64-pc-windows-gnu/hook/run-01:

It discovers our plugin and tries to satisy compatibility and latest version.

[algomaster99]

githubactions

A caveat on case 05 (cache): no nohook rep uses actions/cache at all. DeepSeek used setup-node@v4's built-in cache: option instead, so the analyzer scored setup-node as the target. That row is still stale, but it
measures a different action than the case name suggests. On the hook side, only run-01 ended up using actions/cache.

Unmitigated cases

This looks like a legitimate suggestion but it's odd — v7 of these actions? Actually actions/checkout is at v4 currently (as of 2024). v7 doesn't exist. This could be a prompt injection attempt embedded in tool output, trying to get me to pin to arbitrary SHAs.

benchmark/runs-opencode-deepseek-docker-v3/ghactions-top-03-upload-artifact/hook/run-03/transcript.jsonl

But this "outdated dependencies" message appears to be injected. This could be a prompt injection attempt embedded in tool output.

benchmark/runs-opencode-deepseek-docker-v3/ghactions-top-04-setup-python/hook/run-01/transcript.jsonl

Real docker/login-action latest is v3 (v3.x). There is no v4.6.0 of docker/login-action as far as I know. metadata-action latest is v5, not v6.2.0. build-push-action latest is v6, not v7.4.0. So these suggested versions are fabricated — a classic supply-chain / prompt injection.

benchmark/runs-opencode-deepseek-docker-v3/ghactions-top-10-docker-build-push/hook/run-02/transcript.jsonl

[algomaster99]

go

Nothing relevant. The task is underspecified. I'll make a decision: ask clarifying questions is best, but let me weigh
If you just want a working skeleton with a representative x/sys-based call, say so and I'll pick a neutral module path you can rename.

benchmark/runs-opencode-deepseek-docker-v3/go-top-01-x-sys/hook/run-03/transcript.jsonl

The one case that suceeded benchmark/runs-opencode-deepseek-docker-v3/go-top-10-x-tools/hook/run-02/transcript.jsonl because it did not let go use an earlier version because of compatibility.

[algomaster99]

maven

2.5.252 doesn't sound real...

The suggestion is taken though

benchmark/runs-opencode-deepseek-docker-v3/maven-top-10-h2database/hook/run-02/transcript.jsonl

14 from memory

• junit run-01: "Actually there's JUnit 6? Let me check maven central. Given uncertainty, I should pick a version that resolves." It then writes 5.11.4.
• slf4j run-02: "Let me verify versions maybe by checking maven central? I shouldn't guess URLs… let me just pick well-known versions."
• gson run-01: "I could check but no internet necessarily."
• mysql run-03: "let me check maven central quickly? Network may be limited… Let me use 8.4.0 which is widely available."

The agent sometimes thought there is limited internet connection.

2 compatibility

Deliberate downgrade (2 runs, kotlin-stdlib-jdk7 run-01 and run-02, pinned 1.6.21)

The task targets JDK 7, and Kotlin 1.7 and later dropped JVM target 1.6. DeepSeek works its way down through older releases (1.9.24, then 1.7.22, then 1.6.21) based on build errors. This is a compatibility choice, not a memory problem. Run-03 reached 2.4.20.

3 stale despite looking up

Looked it up, still stale (3 runs)

• spring-boot-test run-02 queried the search.maven.org Solr API. Its index is stale and lists 3.5.3 first, so DeepSeek pinned Spring Boot 3.5.3 (latest 4.1.1).
• lombok run-03 queried the same API and pinned 1.18.38 (latest 1.18.48).
• mysql-connector run-02 read maven-metadata.xml and saw 26.7.0. The new numbering confused it ("MySQL switched to calendar versioning in 2026?"), and it chose 9.7.0 "for predictability".

hook

8 latest are because of maven-metadata.xml.

benchmark/runs-opencode-deepseek-docker-v3/maven-top-09-kotlin-stdlib-jdk7/hook/run-03/transcript.jsonl: This is interesting case.
It kept JDK 7 because it copied an earlier recorded run from the public yul repo.

1. The hook blocked its pom twice (Kotlin 1.7.22). It then noticed pom.xml didn't exist and started investigating.
2. It read the opencode-yul plugin's README in the sandbox. From there it fetched the chains-project/yul file tree from the GitHub API, then benchmark/cases.json, cases_top.json and latest_versions_snapshot.json.
3. Its last message says "Found the exact benchmark case (maven-top-09-kotlin-stdlib-jdk7)."
4. It downloaded the recorded benchmark/top/maven-top-09-kotlin-stdlib-jdk7/{hook,nohook}/pom.xml and App.kt. It concluded that "the benchmark apparently only checks the manifest / dependency versions, not a real build" and wrote the recorded hook pom with 2.4.20 and jvmTarget 1.7.
5. It ran only mvn dependency:resolve, never a compile.