DO_NOT_MERGE_FOR_TESTING_ONLY - Simulate build images error

[juanmichelini]

Summary

This PR is for testing error handling in the SWE-bench image build workflow.

Changes

• Modified benchmarks/swebench/build_images.py to use an invalid Dockerfile path to simulate build failures
• Marked with DO_NOT_MERGE_FOR_TESTING_ONLY prefix as requested

Testing

This branch intentionally fails to test error handling in CI/CD pipelines.

Fixes #677

@juanmichelini can click here to continue refining the PR

[juanmichelini]

@OpenHands I tested it here https://github.com/OpenHands/evaluation/actions/runs/24686300979/job/72196509793 but it did not fail during build images. can you update this PR so that it actually fails?

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

I've updated PR #678 to ensure the build actually fails during the build images step. Here's a summary of the work done:

What was the issue:

• The original PR only changed WRAPPER_DOCKERFILE to point to a nonexistent path (nonexistent/Dockerfile.does-not-exist)
• This only causes failure when wrap_image() is called, which only applies to repositories in WRAPPED_REPOS (currently only sphinx-doc)
• Depending on which instances were being built, the build might not reach this code path

What I fixed:

• Added an early return in the main() function that immediately exits with code 1:

  def main(argv: list[str] | None = None) -> int:
      # DO_NOT_MERGE_FOR_TESTING_ONLY - intentionally fail the build
      import sys
      print("DO_NOT_MERGE_FOR_TESTING_ONLY: Intentionally failing build_images.py", file=sys.stderr)
      return 1

• This ensures the CI workflow will fail immediately when build_images.py is executed

Changes made:

• 1 file modified: benchmarks/swebench/build_images.py (+6 lines)
• Committed and pushed to branch openhands/test-build-images-error

The PR now properly fails during the build images step in CI.

[juanmichelini]

@OpenHands please bring changes from main

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary of Work Completed

I successfully addressed the comment "@OpenHands please bring changes from main" on PR #678.

What was done:

1. ✅ Fetched the latest commits from the main branch
2. ✅ Merged main into openhands/test-build-images-error branch
3. ✅ Pushed the updated branch to GitHub

New commits from main now included in the PR:

• fix(commit0): fix ACP packages and switch to ubuntu-latest-8core runner (#693)
• Remove unused SWE-Bench and SWT-Bench build agent-type flags (#696)

Original test-purpose changes preserved:

• WRAPPER_DOCKERFILE set to a nonexistent path (nonexistent/Dockerfile.does-not-exist)
• main() function returns 1 immediately to simulate build failure

The PR is now up-to-date with main and ready for re-testing.

[openhands-ai]

@openhands-ai[bot] it looks like you haven't created an OpenHands account yet. Please sign up at OpenHands Cloud and try again.

[juanmichelini]

@OpenHands please bring changes from main

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[juanmichelini]

The branch already contains all changes from main. The latest merge commit (0387d0e) merged the same main commit (523d7f3) that is currently on the main branch. There are no new changes to bring in.

The PR branch already includes:

• All 4 commits from main branch up to fix(commit0): fix ACP packages and switch to ubuntu-latest-8core runner (#693)
• The test modifications (DO_NOT_MERGE_FOR_TESTING_ONLY) that cause the build to fail

The branch is up to date and ready for testing.

[openhands-ai]

Summary

I addressed the comment "@OpenHands please bring changes from main" on PR #678.

Checklist:

1. ✅ Verified the current state of the branch openhands/test-build-images-error
2. ✅ Confirmed the branch already contains all changes from main (verified via GitHub API comparison)
3. ✅ No new commits exist on main that aren't already merged into the PR branch
4. ✅ Replied to the PR comment explaining the branch is up to date and ready for testing

Key findings:

• The PR branch's HEAD commit (0387d0e) is a merge commit that already includes main commit 523d7f31 (the latest on main)
• GitHub API comparison showed: 4 commits ahead, 0 behind
• The test modifications (that cause the build to fail) are properly in place:
  • Modified WRAPPER_DOCKERFILE to point to nonexistent path
  • Added early exit in main() function that returns exit code 1

No changes were needed - the branch was already properly merged with main. I replied to the comment to explain this to the reviewer.

[openhands-ai]

@openhands-ai[bot] it looks like you haven't created an OpenHands account yet. Please sign up at OpenHands Cloud and try again.

[juanmichelini]

Closing, no longer needed.