[GR-40566] Bytecode interpreter as default

PullRequest: graalpython/2409

Author: timfel

CHANGELOG.md

@@ -5,6 +5,9 @@ language runtime. The main focus is on user-observable behavior of the engine.
 
 ## Version 22.3.0
 * Rename GraalPython to GraalPy. This change also updates the launchers we ship to include symlinks from `python` and `python3` to `graalpy`, for better integration with other tools.
+* New interpreter backend based on interpreting bytecode. This change should bring better startup performance and memory footprint while retaining good JIT-compiled performance. There is no support for GraalVM instrumentation tools on the bytecode backend yet, so using one of the instrumentation options (e.g. `--inspect`) falls back on the AST backend.
+* New parser generated from CPython's new PEG grammar definition. It brings better compatibility and enables us to implement the `ast` module.
+* Added support for tracing API (`sys.settrace`) which makes `pdb` and related tools work on GraalPy.
 * Updated our pip support to automatically choose the best version for known packages. You can use `pip install pandas`, and pip will select the versions of pandas and numpy that we test in the GraalPy CI.
 
 ## Version 22.2.0

ci.jsonnet

@@ -1 +1 @@
-{ "overlay": "31221c98c6c5240fb2cb8f22be51dea1b325f108" }
+{ "overlay": "55b5a0614a8d46864bb0197b92d2e20033e58ed7" }

docs/contributor/IMPLEMENTATION_DETAILS.md

@@ -1,70 +1,5 @@
 # Implementation Details
 
-## Abstract Operations on Python Objects
-
-Many generic operations on Python objects in CPython are defined in the header
-files `object.h` and `abstract.h`. These operations are widely used and their
-interplay and intricacies are the cause for the conversion, error message, and
-control flow bugs when not mimicked correctly. Our current approach is to
-provide many of these abstract operations as part of the `PythonObjectLibrary`.
-
-### Common operations in the PythonObjectLibrary
-
-The code has evolved over time, so not all built-in nodes are prime examples of
-messages that should be used from the PythonObjectLibrary. We are refactoring
-this as we go, but here are a few examples for things you can (or should soon be
-able to) use the PythonObjectLibrary for:
-
- - casting and coercion to `java.lang.String`, array-sized Java `int`, Python
-   index, fileno, `double`, filesystem path, iterator, and more
- - reading the class of an object
- - accessing the `__dict__` attribute of an object
- - hashing objects and testing for equality
- - testing for truthy-ness
- - getting the length
- - testing for abstract types such as `mapping`, `sequence`, `callable`
- - invoking methods or executing callables
- - access objects through the buffer protocol
-
-### PythonObjectLibrary functions with and without state
-
-Usually, there are at least two messages for each operation - one that takes a
-`ThreadState` argument, and one that doesn't. The intent is to allow passing of
-exception state and caller information similar to how we do it with the `PFrame`
-argument even across library messages, which cannot take a VirtualFrame.
-
-All nodes that are used in message implementations must allow uncached
-usage. Often (e.g. in the case of the generic `CallNode`) they offer execute
-methods with and without frames. If a `ThreadState` was passed to the message, a
-frame to pass to the node can be reconstructed using
-`PArguments.frameForCall(threadState)`. Here's an example:
-
-```java
-@ExportMessage
-long messageWithState(ThreadState state,
-        @Cached CallNode callNode) {
-    Object callable = ...
-
-    if (state != null) {
-        return callNode.execute(PArguments.frameForCall(state), callable, arguments);
-    } else {
-        return callNode.execute(callable, arguments);
-    }
-}
-```
-
-*Note*: It is **always** preferable to call an `execute` method with a
-`VirtualFrame` when both one with and without exist! The reason is that this
-avoids materialization of the frame state in more cases, as described on the
-section on Python's global thread state above.
-
-### Other libraries in the codebase
-
-Accessing hashing storages (the storage for `dict`, `set`, and `frozenset`)
-should be done via the `HashingStorageLibrary`. We are in the process of
-creating a `SequenceStorageLibrary` for sequence types (`tuple`, `list`) to
-replace the `SequenceStorageNodes` collection of classes.
-
 ## Python Global Thread State
 
 In CPython, each stack frame is allocated on the heap, and there's a global

docs/contributor/MISSING.md

@@ -36,7 +36,6 @@ This is just a snapshot as of 2021-07-29.
 
 #### These we should re-implement
  * **_codecs_cn, _codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw, _multibytecodec**:  We can just use our own codecs
- * **_ctypes, _ctypes_test**:  Work in progress
  * **_string**: Empty right now, but its only two methods that we can re-implement
  * **_tracemalloc**:  Memory allocation tracing, we should substitute with the Truffle instrument.
  * **_uuid**: Can be implemented ourselves, is just 1 function
@@ -47,25 +46,20 @@ This is just a snapshot as of 2021-07-29.
  * **parser**:  We need to implement this for our parser
 
 ### Incompleteness on our part:
- * **_ast**: Used in various places, including the help system. Would be nice to support, ours is an empty shell
- * **_contextvars**: Very incomplete
- * **_multiprocessing**:  Work in progress
+ * **_contextvars**: Work in progress
  * **_signal**: Work in progress
  * **mmap**:  We use this as a mixture from the C module, Python, and Java code. Needs major optimizations.
  * **resource**:  This is about resources, there should be Truffle APIs for this (there are issues open)
  * **unicodedata**: A bit incomplete, but not difficult. Maybe should use a Java ICU library
 
 ### Basically complete or easy to make so
- * **_collections**: We've mostly implemented this in Python (a lot is taken from PyPy), but should intrinsify the module in Java for better performance
  * **_md5**:  We use the Python impl from PyPy, but should intrinsify as Java code for performance
  * **_random**
  * **_sha1**:  We use the Python impl from PyPy, but should intrinsify as Java code for performance
  * **_sha256**:  We use the Python impl from PyPy, but should intrinsify as Java code for performance
  * **_sha512**:  We use the Python impl from PyPy, but should intrinsify as Java code for performance
  * **binascii**: Just missing a few methods
- * **codecs**
  * **functools**: Missing a few functions, we mostly implemented it in Python, but should intrinsify the module in Java for better performance
  * **itertools**: We mostly just implement all this in Python (a lot is taken from PyPy), but should intrinsify the module in Java for better performance
  * **locale**: Partially Truffle APIs, should probably use more to play nice for embedders
  * **readline**: We re-implemented this in terms of JLine used in our launcher
- * **zipimport**: We have reimplemented this, but Python 3.8 is moving to a pure-Python impl that we can use

docs/user/ParserDetails.md

@@ -8,30 +8,6 @@ permalink: /reference-manual/python/ParserDetails/
 
 This guide elaborates on how Python files are parsed on the GraalVM Python runtime.
 
-## Parser Performance
-
-#### Loading code from serialized `.pyc` files is faster than parsing the `.py` file using ANTLR.
-
-Creating the abstract syntax tree (AST) for a Python source has two phases.
-The first one creates a simple syntax tree (SST) and a scope tree.
-The second phase transforms the SST to the [Truffle Language Implementation framework](https://github.com/oracle/graal/blob/master/truffle/docs/README.md) tree.
-
-For the transformation, the scope tree it needed.
-The scope tree contains scope locations for variable and function definitions, and information about scopes.
-The simple syntax tree contains nodes mirroring the source.
-Comparing the SST and the Language Implementation framework tree, the SST is much smaller.
-It contains just the nodes representing the source in a simple way.
-One SST node is usually translated to many the Language Implementation framework nodes.
-
-The simple syntax tree can be created in two ways: with ANTLR parsing, or deserialization from an appropriate `*.pyc` file.
-If there is no appropriate `.pyc` file for a source, then the source is parsed with ANTLR.
-If the Python standard import logic finds an appropriate `.pyc` file, it will just trigger deserialization of the SST and scope tree from it.
-
-The deserialization is much faster than source parsing with ANTLR and needs only roughly 30% of the time that ANTLR needs.
-Of course, the first import of a new file is a little bit slower -- besides parsing with ANTLR, the Python standard library import logic serializes the resulting code object to a `.pyc` file, which in our case means
-the SST and scope tree are serialized such a file.
-
-
 ## Creating and Managing pyc Files
 
 #### `.pyc` files are created automatically by the GraalVM Python runtime when no or an invalid `.pyc` file is found matching the desired `.py` file.
@@ -48,7 +24,7 @@ The hashcode is generated only based on the Python source by calling `source.has
 The `.pyc` files are also regenerated if a magic number in the Python parser is changed.
 The magic number is hard-coded in the Python source and can not be changed by the user (unless of course that user has access to the bytecode of Python).
 
-The developers of GraalVM's Python runtime change the magic number when the format of SST or scope tree binary data is altered.
+The developers of GraalVM's Python runtime change the magic number when the bytecode format changes.
 This is an implementation detail, so the magic number does not have to correspond to the version of GraalVM's Python runtime (just like in CPython).
 The magic number of pyc is a function of the concrete Python runtime Java code that is running.
 
@@ -76,29 +52,7 @@ top_folder
 By default the `__pycache__` directory is created on the same directory level as a source code file and in this directory all `.pyc` files from the same directory are stored.
 This folder may store `.pyc` files created with different versions of Python (including, e.g., CPython), so the user may see files ending in `*.cpython3-6.pyc` for example.
 
-The current implementation also includes a copy of the original source text in the `.pyc` file.
-This is a minor performance optimization so you can create a `Source` object with the path to the original source file, but you do not need to read the original `*.py` file, which speeds up the process obtaining the Language Implementation framework tree (just one file is read).
-The structure of a `.graalpy.pyc` file is this:
-```python
-MAGIC_NUMBER
-source text
-binary data - scope tree
-binary data - simple syntax tree
-```
-
-Note that the `.pyc` files are not an effective means to hide Python library source code from guest code, since the original source can still be recovered.
-Even if the source were omitted, the syntax tree contains enough information to decompile into source code easily.
-
-The serialized SST and scope tree are stored in a Python `code` object as well, as the content of the attribute `co_code` (which contains bytecode on CPython). For example:
-```python
->>> def add(x, y):
-...   return x+y
-...
->>> add.__code__.co_code
-b'\x01\x00\x00\x02[]K\xbf\xd1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ...'
-```
-
-#### `.pyc` files are largely managed automatically by the runtime in a manner compatible to CPython. Like on CPython there are options to specify their location, and if they should be written at all, and both of these options can be changed by guest code.
+`.pyc` files are largely managed automatically by the runtime in a manner compatible to CPython. Like on CPython there are options to specify their location, and if they should be written at all, and both of these options can be changed by guest code.
 
 The creation of `*.pyc` files can be controlled in the same ways as on CPython
 (c.f. https://docs.python.org/3/using/cmdline.html):
@@ -134,10 +88,6 @@ files must be removed by the embedder as required.
 
 ## Security Considerations
 
-The serialization of SST and scope tree is hand-written and during deserialization, it is not possible to load classes other than SST Nodes.
-Java serialization or other frameworks are not used to serialize Java objects.
-The main reason is performance, but this has the effect that no class loading can be forced by a maliciously crafted `.pyc` file.
-
 All file operations (obtaining the data, timestamps, and writing `pyc` files)
 are done through the [FileSystem API](https://www.graalvm.org/sdk/javadoc/org/graalvm/polyglot/io/FileSystem.html). Embedders can modify all of these operations by means of custom (e.g., read-only) `FileSystem` implementations.
 The embedder can also effectively disable the creation of `.pyc` files by disabling I/O permissions for GraalVM's Python runtime.

docs/user/Tooling.md

@@ -5,25 +5,18 @@ link_title: Tooling Support for Python
 permalink: /reference-manual/python/Tooling/
 ---
 # Tooling Support for Python
-
-GraalVM's Python runtime is incomplete and cannot launch the standard Python debugger `pdb`.
-However, it can run the tools that GraalVM provides.
-The `graalpy --help:tools` command will give you more information about tools currently supported on Python.
+GraalVM Python runtime can run many standard Python tools as well as tools from the GraalVM ecosystem.
+The `graalpy --help:tools` command will give you more information about GraalVM tools currently supported on Python.
 
 ## Debugger
+The built-in `breakpoint()` function will use `pdb` by default.
 
-To enable debugging, pass the `--inspect` option to the `graalpy` launcher.
-For example:
-```shell
-graalpy --inspect -c "breakpoint(); import os; os.exit()"
-Debugger listening on port 9229.
-To start debugging, open the following URL in Chrome:
-    chrome-devtools://devtools/bundled/js_app.html?ws=127.0.1.1:9229/76fcb6dd-35267eb09c3
-```
+### PDB
+The standard python debugger `pdb` is supported on GraalVM. Refer to the offical [PDB documentation](https://docs.python.org/3/library/pdb.html) for usage.
 
-The standard Python built-in `breakpoint()` will work using the [GraalVM's Chrome Inspector](https://github.com/oracle/graal/blob/master/docs/tools/chrome-debugger.md) implementation.
-You can inspect variables, set watch expressions, interactively evaluate code snippets, etc.
-However, this only works if you pass `--inspect` or some other inspect option. Otherwise, `pdb` is triggered as on CPython (and does not currently work).
+### Chrome Inspector
+To enable [GraalVM's Chrome Inspector](https://github.com/oracle/graal/blob/master/docs/tools/chrome-debugger.md) debugger, pass the `--inspect` option to the `graalpy` launcher.
+The built-in `breakpoint()` function will work using the Chrome Inspector implementation when `--inspect` is passed.
 
 ## Code Coverage
 

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/PythonTests.java

@@ -112,12 +112,7 @@ public static Context enterContext(Map<String, String> options, String[] args) {
         PythonTests.outArray.reset();
         PythonTests.errArray.reset();
         Context prevContext = context;
-        Context.Builder builder = Context.newBuilder().engine(engine).allowExperimentalOptions(true).allowAllAccess(true).options(options).arguments("python", args).option("python.Executable",
-                        executable);
-        if (usingBytecodeCompiler()) {
-            builder.option("python.EnableBytecodeInterpreter", "true").option("python.DisableFrozenModules", "true");
-        }
-        context = builder.build();
+        context = Context.newBuilder().engine(engine).allowExperimentalOptions(true).allowAllAccess(true).options(options).arguments("python", args).option("python.Executable", executable).build();
         context.initialize("python");
         assert prevContext == null;
         context.enter();
@@ -143,6 +138,10 @@ public static void skipOnBytecodeInterpreter() {
         Assume.assumeFalse(PythonOptions.EnableBytecodeInterpreter.getDefaultValue());
     }
 
+    public static void skipOnLegacyASTInterpreter() {
+        Assume.assumeTrue(PythonOptions.EnableBytecodeInterpreter.getDefaultValue());
+    }
+
     public static void assertBenchNoError(Path scriptName, String[] args) {
         final ByteArrayOutputStream byteArrayErr = new ByteArrayOutputStream();
         final ByteArrayOutputStream byteArrayOut = new ByteArrayOutputStream();
@@ -345,23 +344,12 @@ public static File getTestFile(Path filename) {
         }
     }
 
-    public static boolean usingBytecodeCompiler() {
-        return System.getProperty("useBytecodeCompiler") != null;
-    }
-
-    private static org.graalvm.polyglot.Source.Builder configureBuilder(org.graalvm.polyglot.Source.Builder builder) {
-        if (usingBytecodeCompiler()) {
-            return builder.mimeType(PythonLanguage.MIME_TYPE_SOURCE_FOR_BYTECODE);
-        }
-        return builder;
-    }
-
     public static org.graalvm.polyglot.Source createSource(String source) {
-        return configureBuilder(org.graalvm.polyglot.Source.newBuilder("python", source, "Unnamed")).buildLiteral();
+        return org.graalvm.polyglot.Source.newBuilder("python", source, "Unnamed").buildLiteral();
     }
 
     public static org.graalvm.polyglot.Source createSource(File path) throws IOException {
-        return configureBuilder(org.graalvm.polyglot.Source.newBuilder("python", path)).build();
+        return org.graalvm.polyglot.Source.newBuilder("python", path).build();
     }
 
     public static Value runScript(String[] args, File path, OutputStream out, OutputStream err) {

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/grammar/ArgumentsTests.java

@@ -27,11 +27,11 @@
 
 import static com.oracle.graal.python.test.PythonTests.assertLastLineErrorContains;
 import static com.oracle.graal.python.test.PythonTests.assertPrints;
-import static com.oracle.graal.python.test.PythonTests.usingBytecodeCompiler;
 
-import org.junit.Assume;
 import org.junit.Test;
 
+import com.oracle.graal.python.test.PythonTests;
+
 public class ArgumentsTests {
 
     @Test
@@ -146,7 +146,7 @@ public void KwArgs2() {
     @Test
     public void kwargsMerge() {
         // TODO AST interpreter doesn't maintain the order correctly
-        Assume.assumeTrue(usingBytecodeCompiler());
+        PythonTests.skipOnLegacyASTInterpreter();
         assertPrints("{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}\n", "\n" +
                         "def foo(**kwargs):\n" +
                         "  print(kwargs)\n" +

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/runtime/TracingTests.java

@@ -40,16 +40,16 @@
  */
 package com.oracle.graal.python.test.runtime;
 
-import com.oracle.graal.python.test.PythonTests;
-import org.junit.Assume;
 import org.junit.Before;
 import org.junit.Test;
 
+import com.oracle.graal.python.test.PythonTests;
+
 public class TracingTests {
 
     @Before
     public void ensureBytecode() {
-        Assume.assumeTrue(PythonTests.usingBytecodeCompiler());
+        PythonTests.skipOnLegacyASTInterpreter();
     }
 
     @Test

graalpython/com.oracle.graal.python.test/src/tests/test_sys_settrace.py

@@ -155,6 +155,7 @@ def test_case(self):
     return test_case
 
 @unittest.skipIf(skip, 'sys.settrace only works in the bytecode interpreter')
+@unittest.skipIf(True, 'Disabled due to GR-40754')
 class TraceTests(unittest.TestCase):
     def trace(self, frame, event, arg):
         code = frame.f_code