build(kernel): add optional [kernel] extra for use_kernel=True

databricks-sql-kernel is now published to PyPI, so the kernel backend
can ship as an optional dependency instead of a local-dev-only build.

- pyproject: declare databricks-sql-kernel as an optional dependency
  gated to python>=3.10 (the wheel is cp310-abi3, Requires-Python
  >=3.10), and add the `[kernel]` extra. The extra also lists pyarrow:
  the kernel result path (backend/kernel/result_set.py) imports it
  unconditionally to wrap the Arrow batches the kernel returns. pyarrow
  is already pulled transitively via the kernel wheel's
  pyarrow>=23.0.1,<24, but naming it makes the connector-side
  requirement explicit and lets pip co-resolve both constraints at
  install time.
- backend/kernel/_errors.py: update the use_kernel=True ImportError to
  point at `pip install "databricks-sql-connector[kernel]"` and note the
  python>=3.10 requirement (was the obsolete "not yet published, build
  locally" hint).
- README: document the [kernel] extra, use_kernel=True usage, and the
  python>=3.10 / pyarrow notes.

On python<3.10 the `[kernel]` extra resolves to nothing and
use_kernel=True raises the friendly ImportError at runtime; the
connector's own python floor (3.8) is unchanged.

Verified locally (kernel served from a locally-built cp310-abi3 wheel,
since the published package isn't yet mirrored on the dev proxy):
- pip install "databricks-sql-connector[kernel]" -> connector + kernel
  + pyarrow all install; use_kernel=True runs a live query end-to-end
  (backend KernelDatabricksClient).
- plain install -> use_kernel=True raises the friendly ImportError.

NOTE: `poetry lock` still needs to be run to refresh poetry.lock with
the databricks-sql-kernel entry; it is intentionally NOT included here
because it requires the kernel to be resolvable on the index poetry/CI
use (the JFrog db-pypi proxy). Confirm the package resolves there before
merging.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

Author: vikrantpuppala

README.md

@@ -30,6 +30,30 @@ Install using `pip install databricks-sql-connector`
 ### Installing the core library with PyArrow
 Install using `pip install databricks-sql-connector[pyarrow]`
 
+### Installing with the Rust kernel backend (`use_kernel=True`)
+Install using `pip install databricks-sql-connector[kernel]`
+
+This adds the optional [`databricks-sql-kernel`](https://pypi.org/project/databricks-sql-kernel/)
+extension (a native Rust client core, exposed via PyO3). Pass
+`use_kernel=True` to `sql.connect(...)` to route the connection through it
+instead of the default Thrift backend:
+
+```python
+connection = sql.connect(
+  server_hostname=host,
+  http_path=http_path,
+  access_token=token,
+  use_kernel=True,
+)
+```
+
+Notes:
+- Requires **Python >= 3.10** (the kernel wheel is published as
+  `cp310-abi3`). On older interpreters the `[kernel]` extra installs
+  nothing and `use_kernel=True` raises an `ImportError`.
+- The extra also pulls in PyArrow, which the kernel result path requires.
+- Authentication supports PAT (`access_token`), OAuth M2M, and OAuth U2M.
+
 
 ```bash
 export DATABRICKS_HOST=********.databricks.com

pyproject.toml

@@ -32,24 +32,30 @@ pyarrow = [
 pyjwt = "^2.0.0"
 pybreaker = "^1.0.0"
 requests-kerberos = {version = "^0.15.0", optional = true}
+# Optional Rust kernel backend for ``use_kernel=True`` (PyO3 wheel).
+# Pulled in only via the ``[kernel]`` extra below. The published wheel
+# is ``abi3`` with ``Requires-Python: >=3.10`` (built ``abi3-py310``),
+# so the dependency is gated to Python >= 3.10: on 3.8/3.9 the
+# ``[kernel]`` extra resolves to nothing and ``use_kernel=True`` raises
+# a clear ImportError at runtime (see backend/kernel/_errors.py).
+databricks-sql-kernel = {version = "^0.1.0", optional = true, python = ">=3.10"}
 
 
 [tool.poetry.extras]
 pyarrow = ["pyarrow"]
-# `[kernel]` extra is intentionally not declared here yet.
-# `databricks-sql-kernel` is built from the databricks-sql-kernel
-# repo and not yet published to PyPI; declaring it as a poetry dep
-# breaks `poetry lock` for every CI job. Once the wheel is on PyPI
-# the extra will be added back here:
+# ``pip install databricks-sql-connector[kernel]`` adds the Rust kernel
+# backend so ``use_kernel=True`` works. No-op on Python < 3.10 (the
+# wheel's floor) — those users get a runtime ImportError if they pass
+# ``use_kernel=True``.
 #
-#     databricks-sql-kernel = {version = "^0.1.0", optional = true}
-#     [tool.poetry.extras]
-#     kernel = ["databricks-sql-kernel"]
-#
-# Until then, the wheel is not on PyPI and the only supported
-# install path is local dev:
-#     cd databricks-sql-kernel/pyo3 && maturin develop --release
-# (into the same venv as databricks-sql-connector).
+# pyarrow is listed explicitly: the kernel result path
+# (``backend/kernel/result_set.py``) imports pyarrow unconditionally to
+# wrap the Arrow batches the kernel returns. The kernel wheel already
+# declares ``pyarrow>=23.0.1,<24`` so it's pulled transitively, but
+# naming it here documents the connector-side requirement and makes pip
+# co-resolve both constraints at install time rather than failing at
+# ``use_kernel=True`` runtime if the kernel ever drops its pyarrow dep.
+kernel = ["databricks-sql-kernel", "pyarrow"]
 
 [tool.poetry.group.dev.dependencies]
 pytest = "^7.1.2"

src/databricks/sql/backend/kernel/_errors.py

@@ -52,11 +52,13 @@
     import databricks_sql_kernel as _kernel  # type: ignore[import-not-found]
 except ImportError as exc:  # pragma: no cover - same hint as client.py
     raise ImportError(
-        "use_kernel=True requires the databricks-sql-kernel extension, which "
-        "is not yet published on PyPI. Build and install it locally from the "
-        "databricks-sql-kernel repo:\n"
-        "  cd databricks-sql-kernel/pyo3 && maturin develop --release\n"
-        "(into the same venv as databricks-sql-connector)."
+        "use_kernel=True requires the optional databricks-sql-kernel "
+        "extension, which is not installed. Install it with:\n"
+        '  pip install "databricks-sql-connector[kernel]"\n'
+        "The kernel wheel requires Python >= 3.10; on older interpreters "
+        "use_kernel is unavailable. For local kernel development you can "
+        "instead build it from the databricks-sql-kernel repo:\n"
+        "  cd databricks-sql-kernel/pyo3 && maturin develop --release"
     ) from exc
 
 # Route the kernel's Rust-side logs into Python's ``logging`` as soon as