test(kernel): exercise use_kernel=True through the real wheel; no silent skips

Ensure every CI job that's meant to cover the kernel actually drives
the use_kernel=True path through the REAL databricks-sql-kernel wheel,
and fails loudly if it can't (rather than silently skipping / passing
on the Thrift path).

Problem this fixes:
- The kernel unit tests inject a fake databricks_sql_kernel into
  sys.modules. In a shared `pytest tests/unit tests/e2e` session (the
  coverage job, which installs --all-extras so the real wheel IS
  present) that fake shadowed the real wheel, so the kernel e2e tests
  silently skipped — the coverage job looked like it exercised the
  kernel but didn't.

Changes:
- tests/e2e/test_kernel_backend.py + test_kernel_tls.py: replace the
  silent `__file__`-based skip with a three-state guard keyed on
  importlib.metadata (the on-disk dist DB, which a sys.modules stub
  can't fake): skip only when the wheel is genuinely absent; FAIL
  LOUDLY when it's installed-but-shadowed. The `conn` fixture now also
  asserts conn.session.backend is KernelDatabricksClient, so a
  use_kernel=True connection that fell back to Thrift fails the test.
- tests/unit/test_session.py: add TestUseKernelRoutesThroughRealWheel
  (marked `realkernel`) — a no-network proof that
  sql.connect(use_kernel=True) instantiates the REAL
  KernelDatabricksClient (mocks only open_session; does not fake the
  wheel). Skips if the wheel is absent; fails if it's shadowed.
- pyproject.toml: register the `realkernel` marker. Tests so marked
  need an unpolluted sys.modules and must run in a separate pytest
  invocation from the fake-injecting unit tests.
- tests/unit/test_kernel_client.py: document that its session-global
  fake mandates the separate-invocation rule for real-wheel tests.
- code-quality-checks.yml: the Unit Tests + Kernel matrix now asserts
  the real wheel, runs `tests/unit -m "not realkernel"`, then runs the
  real-wheel routing test as its own invocation
  (`pytest tests/unit/test_session.py -m realkernel`). All three unit
  matrices gained `-m "not realkernel"`.
- code-coverage.yml: --ignore the kernel e2e files and add
  `-m "not realkernel"` so the shared --all-extras session doesn't trip
  the new loud guards; the real live kernel e2e stays in kernel-e2e.yml
  (isolated session, real wheel, live warehouse).

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

Author: vikrantpuppala

.github/workflows/code-coverage.yml

@@ -63,8 +63,21 @@ jobs:
           install-args: "--all-extras"
       - name: Run all tests with coverage
         continue-on-error: false
+        # This job installs --all-extras, so the REAL databricks-sql-kernel
+        # wheel is present. The unit suite fakes databricks_sql_kernel in
+        # sys.modules, which would shadow the real wheel in this shared
+        # session — so the kernel-backed suites that need the real wheel
+        # are excluded here and covered by the dedicated kernel-e2e.yml
+        # (isolated session, real wheel, live warehouse):
+        #   - --ignore the kernel e2e files (they assert the real wheel and
+        #     now FAIL LOUDLY rather than silently skip if shadowed), and
+        #   - -m "not realkernel" deselects the no-network real-wheel
+        #     routing test for the same reason.
         run: |
           poetry run pytest tests/unit tests/e2e \
+            --ignore=tests/e2e/test_kernel_backend.py \
+            --ignore=tests/e2e/test_kernel_tls.py \
+            -m "not realkernel" \
             -n 4 \
             --dist=loadgroup \
             --cov=src \

.github/workflows/code-quality-checks.yml

@@ -48,7 +48,7 @@ jobs:
           echo "=== Dependency Version: ${{ matrix.dependency-version }} ==="
           poetry run pip list
       - name: Run tests
-        run: poetry run python -m pytest tests/unit
+        run: poetry run python -m pytest tests/unit -m "not realkernel"
 
   run-unit-tests-with-arrow:
     runs-on:
@@ -96,7 +96,7 @@ jobs:
           echo "=== Dependency Version: ${{ matrix.dependency-version }} with PyArrow ==="
           poetry run pip list
       - name: Run tests
-        run: poetry run python -m pytest tests/unit
+        run: poetry run python -m pytest tests/unit -m "not realkernel"
 
   run-unit-tests-with-kernel:
     runs-on:
@@ -129,17 +129,22 @@ jobs:
         run: |
           echo "=== with databricks-sql-kernel ==="
           poetry run pip list
-      - name: Verify the kernel extra is installed and importable
+      - name: Assert the real kernel wheel is installed (not a stub)
         run: |
-          poetry run python -c "import databricks_sql_kernel as k; assert k.__file__, 'kernel wheel missing __file__'; print('kernel ok:', k.__file__)"
-      - name: Verify use_kernel backend wiring loads against the real wheel
-        run: |
-          # Import the kernel-backed client path against the *real* wheel
-          # (the unit tests use a fake module, so this is the only check
-          # that the published extra actually loads end to end).
-          poetry run python -c "from databricks.sql.backend.kernel.client import KernelDatabricksClient; from databricks.sql.backend.kernel.result_set import KernelResultSet; print('use_kernel backend import ok')"
-      - name: Run tests
-        run: poetry run python -m pytest tests/unit
+          poetry run python -c "import databricks_sql_kernel as k; assert k.__file__, 'kernel wheel missing __file__ — not the real wheel'; print('real kernel wheel:', k.__file__)"
+      - name: Unit tests (kernel wheel present, realkernel deselected)
+        # The bulk of tests/unit fakes databricks_sql_kernel in
+        # sys.modules, so the real-wheel routing test is deselected here
+        # and run on its own below (a shared session would shadow the
+        # real wheel — both real-wheel tests fail loudly if that happens).
+        run: poetry run python -m pytest tests/unit -m "not realkernel"
+      - name: Drive use_kernel=True through the REAL wheel (routing)
+        # Separate invocation, explicit file path: never collects the
+        # fake-module test file, so sys.modules stays unpolluted. This is
+        # the no-network proof that sql.connect(use_kernel=True) actually
+        # instantiates the real KernelDatabricksClient (not a stub, not a
+        # Thrift fallback). Fails loudly if the real wheel is shadowed.
+        run: poetry run python -m pytest tests/unit/test_session.py -m realkernel -v
 
   check-linting:
     runs-on:

pyproject.toml

@@ -100,7 +100,8 @@ exclude = '/(\.eggs|\.git|\.hg|\.mypy_cache|\.nox|\.tox|\.venv|\.svn|_build|buck
 [tool.pytest.ini_options]
 markers = [
     "reviewed: Test case has been reviewed by Databricks",
-    "serial: Tests that must run serially (not parallelized)"
+    "serial: Tests that must run serially (not parallelized)",
+    "realkernel: Requires the real databricks-sql-kernel wheel and an unpolluted sys.modules (no fake kernel stub); must run in a separate pytest invocation from tests that fake databricks_sql_kernel (deselect with -m 'not realkernel', run alone with -m realkernel).",
 ]
 minversion = "6.0"
 log_cli = "false"

tests/e2e/test_kernel_backend.py

@@ -21,6 +21,7 @@
 
 from __future__ import annotations
 
+import sys
 from uuid import uuid4
 
 import pytest
@@ -34,24 +35,59 @@
     ServerOperationError,
 )
 
-# Skip the whole module unless the kernel wheel is genuinely installed.
-# ``pytest.importorskip`` alone isn't enough: the kernel unit tests inject a
-# fake ``databricks_sql_kernel`` ModuleType into ``sys.modules`` so the
-# connector's import-time ``import databricks_sql_kernel`` succeeds without
-# the Rust extension. In the same pytest session that fake module is still
-# in ``sys.modules`` when this e2e file is collected, and importorskip
-# happily returns it. A real wheel exposes ``__file__`` (the compiled
-# extension on disk); the fake ModuleType does not.
-_kernel_mod = pytest.importorskip(
-    "databricks_sql_kernel",
-    reason="use_kernel=True requires the databricks-sql-kernel package",
-)
-if not getattr(_kernel_mod, "__file__", None):
+# These tests must run against the REAL databricks-sql-kernel wheel and
+# must NOT silently pass when it's absent or shadowed. We distinguish
+# three states explicitly so a misconfigured CI job can't look green:
+#
+#   1. Wheel genuinely not installed  -> legitimate skip.
+#   2. Wheel installed in the env but ``sys.modules`` currently holds a
+#      stub (the kernel UNIT tests inject a fake ``databricks_sql_kernel``
+#      ModuleType; in a shared ``pytest tests/unit tests/e2e`` session
+#      that fake can still be resident when this file is collected) ->
+#      FAIL loudly. Silently skipping here is what made the coverage job
+#      look like it exercised the kernel when it didn't.
+#   3. Wheel installed and importable for real -> run.
+#
+# "Installed in the env" is decided via importlib.metadata (the dist
+# database on disk), which a ``sys.modules`` stub can't fake. The
+# ``__file__`` check then tells a real compiled extension from a stub
+# ModuleType.
+import importlib.metadata as _ilm
+
+try:
+    _ilm.version("databricks-sql-kernel")
+    _kernel_installed = True
+except _ilm.PackageNotFoundError:
+    _kernel_installed = False
+
+_kernel_mod = sys.modules.get("databricks_sql_kernel")
+if _kernel_mod is None:
+    try:
+        import databricks_sql_kernel as _kernel_mod  # type: ignore[import-not-found]
+    except ImportError:
+        _kernel_mod = None
+
+_kernel_is_real = _kernel_mod is not None and getattr(_kernel_mod, "__file__", None)
+
+if not _kernel_installed:
+    # State 1: nothing to test against.
     pytest.skip(
-        "databricks_sql_kernel is a test stub (no __file__); "
-        "install the real wheel to run kernel e2e tests",
+        "databricks-sql-kernel is not installed; "
+        "install the real wheel (pip install 'databricks-sql-connector[kernel]') "
+        "to run kernel e2e tests",
         allow_module_level=True,
     )
+elif not _kernel_is_real:
+    # State 2: the wheel IS installed but a stub is shadowing it. Do NOT
+    # skip — that would hide the fact that the kernel path never ran.
+    raise RuntimeError(
+        "databricks-sql-kernel is installed in this environment but "
+        "sys.modules holds a stub (no __file__) — the kernel e2e tests "
+        "would not actually exercise the real wheel. This usually means a "
+        "unit test's fake databricks_sql_kernel module is shadowing the "
+        "real one in a shared pytest session. Run the kernel e2e tests in "
+        "isolation (separate pytest invocation) so the real wheel loads."
+    )
 
 
 @pytest.fixture(scope="module")
@@ -80,9 +116,21 @@ def kernel_conn_params(connection_details):
 @pytest.fixture
 def conn(kernel_conn_params):
     """One-shot connection per test (the simple_test pattern the
-    existing e2e suite uses for cursor-level tests)."""
+    existing e2e suite uses for cursor-level tests).
+
+    Asserts the connection actually routed through the kernel backend —
+    if ``use_kernel=True`` silently fell back to Thrift (e.g. a wiring
+    regression), these tests must fail rather than pass against the
+    wrong backend.
+    """
+    from databricks.sql.backend.kernel.client import KernelDatabricksClient
+
     c = sql.connect(**kernel_conn_params)
     try:
+        assert isinstance(c.session.backend, KernelDatabricksClient), (
+            "use_kernel=True did not route through KernelDatabricksClient; "
+            f"got {type(c.session.backend).__name__}"
+        )
         yield c
     finally:
         c.close()

tests/e2e/test_kernel_tls.py

@@ -30,25 +30,46 @@
 from __future__ import annotations
 
 import os
+import sys
 
 import pytest
 
 import databricks.sql as sql
 from databricks.sql.exc import Error as DatabricksSqlError
 
-# Same real-wheel guard as test_kernel_backend.py: a fake
-# ``databricks_sql_kernel`` ModuleType injected by the unit tests has no
-# ``__file__``; only a compiled wheel does.
-_kernel_mod = pytest.importorskip(
-    "databricks_sql_kernel",
-    reason="use_kernel=True requires the databricks-sql-kernel package",
-)
-if not getattr(_kernel_mod, "__file__", None):
+# Same real-wheel guard as test_kernel_backend.py — see the detailed
+# rationale there. Skip only when the wheel is genuinely not installed;
+# FAIL LOUDLY if it's installed but shadowed by a stub (so a misconfigured
+# shared pytest session can't silently pass as covering the kernel).
+import importlib.metadata as _ilm
+
+try:
+    _ilm.version("databricks-sql-kernel")
+    _kernel_installed = True
+except _ilm.PackageNotFoundError:
+    _kernel_installed = False
+
+_kernel_mod = sys.modules.get("databricks_sql_kernel")
+if _kernel_mod is None:
+    try:
+        import databricks_sql_kernel as _kernel_mod  # type: ignore[import-not-found]
+    except ImportError:
+        _kernel_mod = None
+_kernel_is_real = _kernel_mod is not None and getattr(_kernel_mod, "__file__", None)
+
+if not _kernel_installed:
     pytest.skip(
-        "databricks_sql_kernel is a test stub (no __file__); "
-        "install the real wheel to run kernel TLS e2e tests",
+        "databricks-sql-kernel is not installed; install the real wheel "
+        "to run kernel TLS e2e tests",
         allow_module_level=True,
     )
+elif not _kernel_is_real:
+    raise RuntimeError(
+        "databricks-sql-kernel is installed but sys.modules holds a stub "
+        "(no __file__) — the kernel TLS e2e tests would not exercise the "
+        "real wheel. Run them in isolation (separate pytest invocation) so "
+        "a unit-test fake module doesn't shadow the real one."
+    )
 
 _MITM_CA = os.getenv("MITMPROXY_CA_CERT")
 if not _MITM_CA:

tests/unit/test_kernel_client.py

@@ -61,6 +61,21 @@ def __init__(
         self.error_details_json = error_details_json
 
 
+# These unit tests exercise the connector's error-mapping / wiring logic
+# and need a *controllable* fake ``KernelError`` (to simulate arbitrary
+# kernel error codes), so they install a fake ``databricks_sql_kernel``
+# into ``sys.modules`` unconditionally.
+#
+# IMPORTANT: this fake is session-global and shadows a real wheel if one
+# is installed. Tests that need the REAL wheel (the use_kernel routing
+# test in test_session.py, and the e2e suite in
+# tests/e2e/test_kernel_backend.py) MUST be run in a SEPARATE pytest
+# invocation from this file — never `pytest tests/unit tests/e2e` in one
+# session when the real wheel is installed. Both of those real-wheel
+# tests detect the shadowing (real wheel present but sys.modules holds a
+# stub) and FAIL LOUDLY rather than silently skipping, so a CI job that
+# accidentally mixes them will go red instead of falsely green. The
+# kernel CI matrix runs the real-wheel tests as their own step.
 _fake_kernel_module = types.ModuleType("databricks_sql_kernel")
 _fake_kernel_module.KernelError = _FakeKernelError  # type: ignore[attr-defined]
 _fake_kernel_module.Session = MagicMock()  # type: ignore[attr-defined]

tests/unit/test_session.py

@@ -525,3 +525,65 @@ def test_user_agent_entry_reaches_kernel_client_http_headers(self):
                 assert "my-partner-app" in ua, f"UA was {ua!r}"
             finally:
                 conn.close()
+
+
+@pytest.mark.realkernel
+class TestUseKernelRoutesThroughRealWheel:
+    """No-network proof that ``sql.connect(use_kernel=True)`` actually
+    routes through the REAL databricks-sql-kernel wheel — not a stub and
+    not a fallback to Thrift.
+
+    This is the unit-side complement to the live e2e suite: it does not
+    need a warehouse (only the network boundary ``open_session`` is
+    mocked), but unlike the other kernel unit tests it does NOT fake the
+    wheel — the real ``KernelDatabricksClient`` is instantiated and its
+    ``_kernel_session`` is built from the real ``databricks_sql_kernel``
+    ``Session``. Skips only when the real wheel is genuinely absent
+    (e.g. the no-kernel CI tier); it must never silently pass when the
+    wheel is present.
+    """
+
+    def _real_kernel_or_skip(self):
+        import importlib.metadata as ilm
+
+        try:
+            ilm.version("databricks-sql-kernel")
+        except ilm.PackageNotFoundError:
+            pytest.skip("databricks-sql-kernel wheel not installed")
+        mod = __import__("databricks_sql_kernel")
+        if not getattr(mod, "__file__", None):
+            pytest.fail(
+                "databricks-sql-kernel is installed but sys.modules holds a "
+                "stub (no __file__) — a unit-test fake is shadowing the real "
+                "wheel; this routing test would not exercise the real kernel."
+            )
+
+    def test_connect_use_kernel_instantiates_real_kernel_backend(self):
+        self._real_kernel_or_skip()
+
+        from databricks.sql.backend.kernel.client import KernelDatabricksClient
+
+        # Mock only the network boundary: the real KernelDatabricksClient
+        # is constructed (building a real databricks_sql_kernel Session),
+        # but open_session() doesn't hit the wire.
+        with patch.object(
+            KernelDatabricksClient,
+            "open_session",
+            return_value=SessionId(BackendType.SEA, "sess-id", None),
+        ):
+            conn = databricks.sql.connect(
+                server_hostname="foo.cloud.databricks.com",
+                http_path="/sql/1.0/warehouses/abc",
+                use_kernel=True,
+                access_token="dapi-xyz",
+                enable_telemetry=False,
+            )
+            try:
+                # The active backend is the REAL kernel client class.
+                assert isinstance(conn.session.backend, KernelDatabricksClient), (
+                    "use_kernel=True did not route through the real "
+                    f"KernelDatabricksClient; got "
+                    f"{type(conn.session.backend).__name__}"
+                )
+            finally:
+                conn.close()