[HOTFIX] Use importlib.util.find_spec for pluggable worker discovery

[chandrasekharan-zipstack]

What

Replaces the filesystem worker.py existence check in WorkerBuilder._verify_pluggable_worker_exists() with importlib.util.find_spec().

Why

The pre-import check at workers/shared/infrastructure/config/builder.py:338 hardcoded worker.py as the file to look for. This breaks any deployment where the pluggable worker has been compiled to an extension module (e.g. Nuitka or Cython produces a worker.cpython-312-x86_64-linux-gnu.so that replaces worker.py) — the module is perfectly importable, but the hard .py existence check rejects the worker before importlib.import_module() ever runs.

Observed error pattern (paraphrased):

ERROR: Pluggable worker file not found: /app/pluggable_worker/<name>/worker.py
ImportError: Pluggable worker '<name>' not found.
Expected module: pluggable_worker.<name>.worker

This is not specific to any one obfuscation tool — the check would also reject plugins distributed as Cython .so, bytecode-only .pyc, namespace packages, or any future loader that doesn't materialize a .py file on disk.

How

Use Python's standard "is this module importable?" primitive instead of filesystem introspection:

module_path = f"pluggable_worker.{worker_type.value}.worker"

if importlib.util.find_spec(module_path) is None:
    logger.error(f"Pluggable worker module not importable: {module_path}")
    return False

importlib.import_module(module_path)

find_spec() consults every registered finder (source .py, compiled .so, bytecode .pyc, namespace packages, zipimports) and returns None cleanly if nothing can produce the module. The subsequent import_module() still catches findable-but-broken modules (e.g. ImportError inside the loaded module).

The only semantic change: when a parent package exists but an intermediate subpackage is missing (pluggable_worker/ exists but pluggable_worker/<name>/ does not), find_spec raises ModuleNotFoundError. That's a subclass of ImportError and is already handled by the existing except ImportError block, so the function correctly returns False just like before.

Can this PR break any existing features?

No. Behavior preserved across all deployment scenarios, verified via a synthetic fixture that compiles a test worker with Nuitka and runs _verify_pluggable_worker_exists() against it:

Scenario	.py present	.so present	Before	After
No subpackage (pluggable workers disabled)	no	no	False (file not found)	False (ImportError)
Source available	yes	no	True	True
Compiled extension module	no	yes	False ← bug	True ← fixed

Relevant Docs

• Python docs: importlib.util.find_spec

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a6536540-703e-4261-95e3-0bc96ef664e1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/pluggable-worker-find-spec-v0.163.2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Greptile Summary

This hotfix replaces the hardcoded worker.py filesystem existence check in _verify_pluggable_worker_exists() with importlib.util.find_spec(), allowing pluggable workers compiled to .so (Nuitka/Cython) or other non-.py forms to be correctly discovered. A secondary fix in worker.py corrects the core-worker directory derivation for api-deployment, whose on-disk path uses hyphens — the old value.replace("-", "_") was a no-op on already-underscored enum values and would have missed the actual directory.

Confidence Score: 5/5

Safe to merge — both changes are targeted correctness fixes with no regressions on the unaffected paths.

The builder.py change is correct and the previously flagged ValueError is now handled. The worker.py fix properly resolves the api-deployment directory mismatch. No new P0/P1 issues found; all remaining notes from prior review threads have been addressed.

No files require special attention.

Important Files Changed

Filename	Overview
workers/shared/infrastructure/config/builder.py	Replaces filesystem .py check with importlib.util.find_spec(); ValueError now caught alongside ImportError; logic and exception handling are correct.
workers/worker.py	Core-worker directory derivation now uses to_import_path().rsplit() instead of a blind replace, correctly resolving api-deployment's hyphenated on-disk path.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["_verify_pluggable_worker_exists(worker_type)"] --> B{is_pluggable?}
    B -- No --> C[return True]
    B -- Yes --> D["module_path = 'pluggable_worker.{worker_type}.worker'"]
    D --> E["importlib.util.find_spec(module_path)"]
    E -- "returns None" --> F["logger.error\nreturn False"]
    E -- "raises ValueError" --> G["except ImportError | ValueError\nreturn False"]
    E -- "spec found" --> H["importlib.import_module(module_path)"]
    H -- "ImportError / ValueError" --> G
    H -- "OSError / AttributeError" --> I["except OSError | AttributeError\nreturn False"]
    H -- success --> J["logger.debug\nreturn True"]

Loading

_{Reviews (4): Last reviewed commit: "[FIX] Resolve api-deployment worker dire..." | Re-trigger Greptile}

[muhammad-ali-e]

added a comment

[muhammad-ali-e]

Hope this not break running worker locally insated of docker

[chandrasekharan-zipstack]

Tested locally — doesn't break local runs. find_spec() is actually a superset of the previous filesystem check.

Local runs go through workers/run-worker.sh, which sets export PYTHONPATH="$WORKERS_DIR:${PYTHONPATH:-}" (line 702) before launching celery. So import pluggable_worker.<name>.worker resolves the same way the launcher itself uses the dotted module name at line 444 (celery_app_module="pluggable_worker.${worker_type}.worker").

Coverage comparison vs old code:

Scenario	Old (Path(...)/worker.py)	New (find_spec)
Local .py via run-worker.sh	✅	✅ (PYTHONPATH set by script)
Docker with uncompiled plugin	✅	✅
Docker with compiled .so (Cython/Nuitka)	❌ (checks .py)	✅
OSS with no pluggable plugins	n/a — is_pluggable() returns False, early return	same
Bare python worker.py from repo root, no PYTHONPATH	❌	❌ (and would fail at actual import too)

The only way someone could get a false negative from the new code but not the old is if they ran a pluggable worker outside run-worker.sh without any PYTHONPATH setup — in that case the old check returned True but celery would have crashed on the real import a few lines later. So net change: we fail earlier with a cleaner log instead of getting a stack trace.

Also confirmed on snapshot.180420262 locally: pluggable workers (agentic_callback + bulk_download worker.so) register and run fine — previously they crash-looped because Path(...)/worker.py returned False for compiled modules.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud