Skip to content

Migrate to JDI execution control - #137

Draft
m-dzianishchyts wants to merge 16 commits into
dflib:mainfrom
m-dzianishchyts:feat-jdi-execution-control
Draft

m-dzianishchyts wants to merge 16 commits into
dflib:mainfrom
m-dzianishchyts:feat-jdi-execution-control

Conversation

@m-dzianishchyts

Copy link
Copy Markdown
Contributor

What does this PR do?

Moves notebook execution into a separate persistent executor JVM and separates what that JVM computes from what the managing JVM shows in Jupyter. The branch carries both steps:

  • Executor JVM: cells run in a persistent executor JVM controlled through JDI.
  • Output separation: the display path between the two JVMs is replaced by a transport-independent protocol.

Java 21 is now the minimum version.

Details

Execution

  • A cell's result is now a plain-text evaluation result. It no longer carries Jupyter display data.
  • The old in-process class loading, execution control, telemetry collectors and the standalone launcher are removed.
  • The %time magic is simplified accordingly.
  • Standard input for JShell now comes from the Jupyter input stream.

Display and update between the JVMs

  • Explicit display and updateDisplay calls are encoded as validated, transferable requests. Cyclic structures and non-finite numbers inside bundles or containers are rejected before any connection is opened.
  • Delivery is one request and one response per connection over a Unix-domain socket, with a bounded wait and no retries.
  • Failures are reported as unavailable, rejected or uncertain. The uncertain case never claims that nothing was shown, and a healthy executor stays usable afterwards.

Helper routing

  • The managing JVM and the executor JVM now use separate helper classes, and which one runs is fixed by what each JVM imports.
  • Executor printf, local render, display and update are supported. Nested evaluation, programmatic magics and kernel access in the executor fail with operation-specific messages.

Compatibility

  • Java 11 and 17 are no longer supported. Build and CI target 21 and 25.

Testing

  • the protocol codec and validation;
  • delivery and endpoint behavior, including the timeout bound;
  • helper routing and render characterization;
  • magic dispatch;
  • dependency boundaries between the evaluator, protocol and Jupyter layers.
  • the display round trip;
  • failures leaving the next cell runnable;
  • a stalled manager;
  • endpoint lifecycle.

Launch the managing kernel directly and remove the launcher JVM.
Keep user state in a separate executor across cells, imports and
redefinitions.

- Pass JJAVA_JVM_OPTS to the executor JVM.
- Handle magics and extension lifecycle in the managing kernel.
- Return ordinary expression results as text/plain.
- Transfer portable display bundles over a loopback socket.
- Report timeouts, interrupts and executor termination explicitly.
- Wire executor stdout/stderr directly to JupyterIO.
- Add regression coverage for state, recovery and display transport.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant