Skip to content

Disallow further step() calls after episode completion - #91

Open
pangwangshu wants to merge 1 commit into
allenai:mainfrom
pangwangshu:wangshu/issue-31
Open

Disallow further step() calls after episode completion#91
pangwangshu wants to merge 1 commit into
allenai:mainfrom
pangwangshu:wangshu/issue-31

Conversation

@pangwangshu

Copy link
Copy Markdown

Closes #31.

Problem

step() recomputes isCompleted on every call (success, failure via negative score, or step-limit reached), but never actually stops processing new actions once it's set. An agent that doesn't check the returned flag can keep calling step() after the episode is "done," and those actions still reach the simulator and can change the score — exactly the score-gaming risk described in the issue, especially for forced-choice tasks.

Fix

ScienceWorldEnv.step() now short-circuits once self.isCompleted is True: it skips self.server.step(...) entirely and returns the cached terminal observation/infos with reward=0, instead of letting the action through. The flag is reset to False in load() and reset() so a new episode isn't blocked by a stale value from the previous one.

This follows the same pattern already used a few lines above for the step-limit and negative-score checks (see the existing # New: Handle this in the API rather than the agent comment) — completion handling lives in this Python wrapper, not the Scala simulator, so no simulator/JAR changes are needed here.

Scope

The issue thread also touched on two related-but-separate things that this PR intentionally does not address, to keep it focused:

  • An opt-in "continue after failure" mode (requested by @yuchenlin for RL-from-failure use cases) — discussed but never reached consensus (@MarcCote questioned the need), so no API was agreed on.
  • A possible false-failure bug where focusing on an object already in inventory ends the episode early — flagged by @MarcCote but never reproduced/confirmed, and it's about why isCompleted gets set rather than what happens after.

Testing

🤖 Generated with Claude Code

Prevents an agent from continuing to act (and inflating its score)
after isCompleted is set, by ignoring further step() calls and
returning the cached terminal state instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Disallow new steps/actions if the isCompleted flag is set

1 participant