feat(auditor): reattach job-submit through local config and pause/resume#758
Open
parkanzky wants to merge 4 commits into
Open
feat(auditor): reattach job-submit through local config and pause/resume#758parkanzky wants to merge 4 commits into
parkanzky wants to merge 4 commits into
Conversation
ironcommit
approved these changes
Jul 17, 2026
ironcommit
left a comment
ironcommit
left a comment
Contributor
There was a problem hiding this comment.
LGTM. We can back out of this once, jobs is further along.
Contributor
|
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe auditor plugin adds workspace-scoped job APIs and Docker-backed task launching, then executes audits independently per probe with persistent state, retries, resume handling, signal-aware aggregation, and combined artifacts. Tests and dependency metadata are updated accordingly. ChangesAuditor jobs
Dependency metadata refresh
Sequence Diagram(s)sequenceDiagram
participant Client
participant AuditorPluginService
participant AuditJob
participant PersistentJobStorage
participant Garak
Client->>AuditorPluginService: Submit audit job
AuditorPluginService->>AuditJob: Dispatch workspace-scoped job
AuditJob->>PersistentJobStorage: Create per-probe state
AuditJob->>Garak: Execute probes with retries
Garak-->>PersistentJobStorage: Store probe reports
AuditJob-->>Client: Return status and aggregated results
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py (1)
395-439: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
PlatformJobSpecas the return type.compile()already constructs that spec;-> objecthides the contract. ImportPlatformJobSpecat module scope and annotate the method directly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py` around lines 395 - 439, Update the Audit job classmethod compile to return PlatformJobSpec instead of object. Move the PlatformJobSpec import from inside compile to module scope, then annotate the method directly while preserving its existing PlatformJobSpec construction.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py`:
- Around line 491-501: Update the initialization flow around todo_dir and
_divide_and_write_confs so todo_dir is not treated as initialized until all
probe configuration files are successfully written. Build the split
configuration in a temporary directory, then atomically rename or publish it as
todo_dir after completion, preserving safe resume behavior when interrupted.
- Around line 261-310: Harden _aggregate_reports against path traversal from the
user-controlled report_dir_name and report_prefix: reject absolute values and
any path separators or parent-traversal components, then resolve the constructed
report paths and verify they remain contained within persistent before globbing,
creating, writing, or registering artifacts. Preserve the existing aggregation
behavior for valid names.
- Around line 545-585: The retry loop in the probe execution flow must isolate
generated outputs for each attempt. Update the setup around retry_n and
probe_env so every retry uses a fresh per-attempt XDG_DATA_HOME, or reliably
clears report_marker and partial JSONL outputs before running; ensure only
artifacts produced by the current successful attempt are moved into
complete_dir.
- Around line 471-489: Move SIGTERM registration out of AuditJob.run() in
plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py:471-489; expose the
partial-result save behavior as callback/state passed into the job, and register
the handler only from the main-thread caller in
plugins/nemo-auditor/src/nemo_auditor/tasks/audit.py:31-39 so the
asyncio.to_thread path remains valid. Update
plugins/nemo-auditor/tests/test_audit_job.py:662-689 to trigger the real SIGTERM
dispatch path rather than invoking the former closure directly.
In `@plugins/nemo-auditor/src/nemo_auditor/service.py`:
- Around line 60-65: Expose pause and resume operations for auditor jobs across
all affected sites: update the AuditJob routes in
plugins/nemo-auditor/src/nemo_auditor/service.py at lines 60-65, add the
corresponding /pause and /resume paths and schemas to
plugins/nemo-auditor/openapi/openapi.yaml at lines 453-592, and extend the route
coverage in plugins/nemo-auditor/tests/test_service.py at lines 14-25 to verify
both endpoints.
---
Nitpick comments:
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py`:
- Around line 395-439: Update the Audit job classmethod compile to return
PlatformJobSpec instead of object. Move the PlatformJobSpec import from inside
compile to module scope, then annotate the method directly while preserving its
existing PlatformJobSpec construction.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 39b5e776-e874-4bf0-828d-77823299193d
📒 Files selected for processing (8)
packages/nmp_platform/config/local.yamlplugins/nemo-auditor/openapi/openapi.yamlplugins/nemo-auditor/src/nemo_auditor/jobs/audit.pyplugins/nemo-auditor/src/nemo_auditor/service.pyplugins/nemo-auditor/src/nemo_auditor/tasks/audit.pyplugins/nemo-auditor/tests/test_audit_job.pyplugins/nemo-auditor/tests/test_service.pyservices/core/jobs/tests/test_config.py
parkanzky
force-pushed
the
AALGO-224/auditor-pause-resume-pparkanzky
branch
from
July 17, 2026 19:41
419a066 to
4e0fd93
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-auditor/openapi/openapi.yaml`:
- Around line 273-281: Constrain the pagination parameters in the OpenAPI
definitions for page_size and limit, including the additional occurrences, with
a finite maximum and valid positive lower bound. Apply the same bounds to the
corresponding job and log pagination schemas, and ensure response array limits
are capped consistently so requests cannot produce arbitrarily large result
sets.
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py`:
- Around line 6-8: Correct the audit command documentation to state the actual
default garak interpreter path used by the configuration at line 47,
/app/.garak_venv/bin/python, while retaining the NEMO_AUDITOR_GARAK_PYTHON
override description.
- Around line 574-600: Update the terminal probe handling around the completed
and retries-exhausted branches to unlink probe_yaml before moving probe_dir into
complete_dir or failed_dir, and remove the later unconditional unlink. Ensure
the marker is deleted before either terminal move or the fail-job raise, while
preserving recovery through running/config.yaml if a move fails.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 03930fbe-4e8f-45c4-b6b9-d98cc1acc313
📒 Files selected for processing (8)
packages/nmp_platform/config/local.yamlplugins/nemo-auditor/openapi/openapi.yamlplugins/nemo-auditor/src/nemo_auditor/jobs/audit.pyplugins/nemo-auditor/src/nemo_auditor/service.pyplugins/nemo-auditor/src/nemo_auditor/tasks/audit.pyplugins/nemo-auditor/tests/test_audit_job.pyplugins/nemo-auditor/tests/test_service.pyservices/core/jobs/tests/test_config.py
🚧 Files skipped from review as they are similar to previous changes (6)
- services/core/jobs/tests/test_config.py
- packages/nmp_platform/config/local.yaml
- plugins/nemo-auditor/src/nemo_auditor/tasks/audit.py
- plugins/nemo-auditor/tests/test_service.py
- plugins/nemo-auditor/src/nemo_auditor/service.py
- plugins/nemo-auditor/tests/test_audit_job.py
parkanzky
force-pushed
the
AALGO-224/auditor-pause-resume-pparkanzky
branch
4 times, most recently
from
July 17, 2026 21:43
cd95ccd to
e575461
Compare
parkanzky
enabled auto-merge
July 17, 2026 21:46
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-auditor/openapi/openapi.yaml`:
- Around line 453-484: Add POST OpenAPI operations for
/apis/auditor/v2/workspaces/{workspace}/jobs/audit/{name}/pause and /resume
alongside
cancel_job_apis_auditor_v2_workspaces__workspace__jobs_audit__name__cancel_post.
Include the same required workspace and name path parameters, return AuditJob on
success, and document validation errors consistently with the cancellation
operation.
In `@third_party/osv-licenses.json`:
- Around line 1603-1691: Update the mcp dependency pin from 1.26.0 to version
1.28.1 or later, then regenerate the dependency lockfile and third-party license
artifacts so they reflect the upgraded package and resolved transitive versions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a6502532-f08e-4ace-83cf-2e79deea1684
📒 Files selected for processing (10)
packages/nmp_platform/config/local.yamlplugins/nemo-auditor/openapi/openapi.yamlplugins/nemo-auditor/src/nemo_auditor/jobs/audit.pyplugins/nemo-auditor/src/nemo_auditor/service.pyplugins/nemo-auditor/src/nemo_auditor/tasks/audit.pyplugins/nemo-auditor/tests/test_audit_job.pyplugins/nemo-auditor/tests/test_service.pyservices/core/jobs/tests/test_config.pythird_party/osv-licenses.jsonthird_party/requirements-main.txt
🚧 Files skipped from review as they are similar to previous changes (5)
- packages/nmp_platform/config/local.yaml
- plugins/nemo-auditor/tests/test_service.py
- plugins/nemo-auditor/src/nemo_auditor/service.py
- plugins/nemo-auditor/src/nemo_auditor/tasks/audit.py
- plugins/nemo-auditor/tests/test_audit_job.py
parkanzky
disabled auto-merge
July 17, 2026 21:47
parkanzky
enabled auto-merge
July 17, 2026 21:52
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py (1)
395-415: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse the concrete compile return type.
compile()always returnsPlatformJobSpec;-> objectremoves useful type checking. Move that import to module scope and annotate the return unless a verified import cycle prevents it.As per coding guidelines, prefer concrete type hints and import those types normally when possible.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py` around lines 395 - 415, Update the classmethod compile to return PlatformJobSpec instead of object, and move the PlatformJobSpec import from the method body to module scope using a normal import. Preserve the existing behavior and retain the local import only if a verified import cycle makes a module-level import impossible.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py`:
- Around line 304-309: Update the hitlog aggregation flow around agg_hitlog so
it first collects or checks matching hitlog paths and only creates and writes
agg_hitlog when at least one hitlog exists. Preserve the existing concatenation
behavior for non-empty matches, and ensure _collect_report_artifacts() cannot
register an empty artifact.
---
Nitpick comments:
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py`:
- Around line 395-415: Update the classmethod compile to return PlatformJobSpec
instead of object, and move the PlatformJobSpec import from the method body to
module scope using a normal import. Preserve the existing behavior and retain
the local import only if a verified import cycle makes a module-level import
impossible.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9a4d2a33-42b6-471b-b06e-711e978b4d0a
📒 Files selected for processing (10)
packages/nmp_platform/config/local.yamlplugins/nemo-auditor/openapi/openapi.yamlplugins/nemo-auditor/src/nemo_auditor/jobs/audit.pyplugins/nemo-auditor/src/nemo_auditor/service.pyplugins/nemo-auditor/src/nemo_auditor/tasks/audit.pyplugins/nemo-auditor/tests/test_audit_job.pyplugins/nemo-auditor/tests/test_service.pyservices/core/jobs/tests/test_config.pythird_party/osv-licenses.jsonthird_party/requirements-main.txt
🚧 Files skipped from review as they are similar to previous changes (6)
- plugins/nemo-auditor/tests/test_service.py
- packages/nmp_platform/config/local.yaml
- plugins/nemo-auditor/src/nemo_auditor/tasks/audit.py
- services/core/jobs/tests/test_config.py
- third_party/requirements-main.txt
- plugins/nemo-auditor/tests/test_audit_job.py
Comment on lines
+304
to
+309
| hitlog_pattern = str(persistent / "complete" / "*" / "garak" / report_dir_name / f"{report_prefix}.hitlog.jsonl") | ||
| agg_hitlog = agg_dir / f"{report_prefix}.hitlog.jsonl" | ||
| with agg_hitlog.open("wb") as out_fd: | ||
| for hitlog_path in glob.glob(hitlog_pattern): | ||
| with open(hitlog_path, "rb") as in_fd: | ||
| shutil.copyfileobj(in_fd, out_fd) |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not publish an empty hitlog artifact.
Opening agg_hitlog unconditionally creates an empty file when no probe produced hitlogs; _collect_report_artifacts() then registers it as a result.
Proposed fix
hitlog_pattern = str(persistent / "complete" / "*" / "garak" / report_dir_name / f"{report_prefix}.hitlog.jsonl")
agg_hitlog = agg_dir / f"{report_prefix}.hitlog.jsonl"
- with agg_hitlog.open("wb") as out_fd:
- for hitlog_path in glob.glob(hitlog_pattern):
+ hitlogs = glob.glob(hitlog_pattern)
+ if hitlogs:
+ with agg_hitlog.open("wb") as out_fd:
+ for hitlog_path in hitlogs:
+ with open(hitlog_path, "rb") as in_fd:
+ shutil.copyfileobj(in_fd, out_fd)
+ elif agg_hitlog.exists():
+ agg_hitlog.unlink()
- with open(hitlog_path, "rb") as in_fd:
- shutil.copyfileobj(in_fd, out_fd)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| hitlog_pattern = str(persistent / "complete" / "*" / "garak" / report_dir_name / f"{report_prefix}.hitlog.jsonl") | |
| agg_hitlog = agg_dir / f"{report_prefix}.hitlog.jsonl" | |
| with agg_hitlog.open("wb") as out_fd: | |
| for hitlog_path in glob.glob(hitlog_pattern): | |
| with open(hitlog_path, "rb") as in_fd: | |
| shutil.copyfileobj(in_fd, out_fd) | |
| hitlog_pattern = str(persistent / "complete" / "*" / "garak" / report_dir_name / f"{report_prefix}.hitlog.jsonl") | |
| agg_hitlog = agg_dir / f"{report_prefix}.hitlog.jsonl" | |
| hitlogs = glob.glob(hitlog_pattern) | |
| if hitlogs: | |
| with agg_hitlog.open("wb") as out_fd: | |
| for hitlog_path in hitlogs: | |
| with open(hitlog_path, "rb") as in_fd: | |
| shutil.copyfileobj(in_fd, out_fd) | |
| elif agg_hitlog.exists(): | |
| agg_hitlog.unlink() |
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 307-307: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(hitlog_path, "rb")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py` around lines 304 - 309,
Update the hitlog aggregation flow around agg_hitlog so it first collects or
checks matching hitlog paths and only creates and writes agg_hitlog when at
least one hitlog exists. Preserve the existing concatenation behavior for
non-empty matches, and ensure _collect_report_artifacts() cannot register an
empty artifact.
Signed-off-by: Paul A. Parkanzky <parkanzky@users.noreply.github.com>
Signed-off-by: Paul A. Parkanzky <parkanzky@users.noreply.github.com>
Signed-off-by: Paul A. Parkanzky <parkanzky@users.noreply.github.com>
Signed-off-by: Paul A. Parkanzky <parkanzky@users.noreply.github.com>
parkanzky
force-pushed
the
AALGO-224/auditor-pause-resume-pparkanzky
branch
from
July 18, 2026 00:26
b25e9aa to
b168ae0
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py`:
- Around line 277-302: Update the aggregate report flow around agg_jsonl and
agg_html to write both command outputs to temporary files in the same directory,
keeping the existing final paths untouched while either subprocess runs. Only
after aggregate_reports and report_digest both succeed should the temporary
files replace the final JSONL and HTML paths atomically; ensure temporary files
are cleaned up on failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bef996e8-4e94-4b2f-8418-d21f7ed8ba2c
📒 Files selected for processing (10)
packages/nmp_platform/config/local.yamlplugins/nemo-auditor/openapi/openapi.yamlplugins/nemo-auditor/src/nemo_auditor/jobs/audit.pyplugins/nemo-auditor/src/nemo_auditor/service.pyplugins/nemo-auditor/src/nemo_auditor/tasks/audit.pyplugins/nemo-auditor/tests/test_audit_job.pyplugins/nemo-auditor/tests/test_service.pyservices/core/jobs/tests/test_config.pythird_party/osv-licenses.jsonthird_party/requirements-main.txt
🚧 Files skipped from review as they are similar to previous changes (7)
- plugins/nemo-auditor/tests/test_service.py
- plugins/nemo-auditor/src/nemo_auditor/service.py
- packages/nmp_platform/config/local.yaml
- plugins/nemo-auditor/src/nemo_auditor/tasks/audit.py
- services/core/jobs/tests/test_config.py
- third_party/requirements-main.txt
- plugins/nemo-auditor/tests/test_audit_job.py
Comment on lines
+277
to
+302
| agg_dir = persistent / "garak" / report_dir_name | ||
| agg_dir.mkdir(parents=True, exist_ok=True) | ||
| agg_jsonl = agg_dir / f"{report_prefix}.report.jsonl" | ||
|
|
||
| result = subprocess.run( | ||
| [garak_python, "-m", "garak.analyze.aggregate_reports", "-o", str(agg_jsonl)] + jsonls, | ||
| capture_output=True, | ||
| text=True, | ||
| check=False, | ||
| ) | ||
| if result.returncode != 0: | ||
| raise GarakFailure( | ||
| f"garak aggregate_reports failed (rc={result.returncode}): {result.stderr[-_LOG_TAIL_BYTES:]}" | ||
| ) | ||
|
|
||
| agg_html = agg_dir / f"{report_prefix}.report.html" | ||
| with agg_html.open("w") as html_fd: | ||
| result = subprocess.run( | ||
| [garak_python, "-m", "garak.analyze.report_digest", "-r", str(agg_jsonl)], | ||
| stdout=html_fd, | ||
| stderr=subprocess.PIPE, | ||
| text=True, | ||
| check=False, | ||
| ) | ||
| if result.returncode != 0: | ||
| raise GarakFailure(f"garak report_digest failed (rc={result.returncode}): {result.stderr[-_LOG_TAIL_BYTES:]}") |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Publish aggregate reports atomically.
Both commands overwrite final paths before success is known. A failed resume can corrupt a previously valid partial aggregate. Write temporary JSONL/HTML files and replace the finals only after both commands succeed.
Proposed fix
agg_jsonl = agg_dir / f"{report_prefix}.report.jsonl"
+ agg_jsonl_tmp = agg_dir / f"{report_prefix}.report.jsonl.tmp"
result = subprocess.run(
- [garak_python, "-m", "garak.analyze.aggregate_reports", "-o", str(agg_jsonl)] + jsonls,
+ [garak_python, "-m", "garak.analyze.aggregate_reports", "-o", str(agg_jsonl_tmp)] + jsonls,
...
if result.returncode != 0:
+ if agg_jsonl_tmp.exists():
+ agg_jsonl_tmp.unlink()
raise GarakFailure(...)
agg_html = agg_dir / f"{report_prefix}.report.html"
- with agg_html.open("w") as html_fd:
+ agg_html_tmp = agg_dir / f"{report_prefix}.report.html.tmp"
+ with agg_html_tmp.open("w") as html_fd:
result = subprocess.run(
- [garak_python, "-m", "garak.analyze.report_digest", "-r", str(agg_jsonl)],
+ [garak_python, "-m", "garak.analyze.report_digest", "-r", str(agg_jsonl_tmp)],
...
if result.returncode != 0:
+ agg_jsonl_tmp.unlink()
+ agg_html_tmp.unlink()
raise GarakFailure(...)
+ os.replace(agg_jsonl_tmp, agg_jsonl)
+ os.replace(agg_html_tmp, agg_html)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| agg_dir = persistent / "garak" / report_dir_name | |
| agg_dir.mkdir(parents=True, exist_ok=True) | |
| agg_jsonl = agg_dir / f"{report_prefix}.report.jsonl" | |
| result = subprocess.run( | |
| [garak_python, "-m", "garak.analyze.aggregate_reports", "-o", str(agg_jsonl)] + jsonls, | |
| capture_output=True, | |
| text=True, | |
| check=False, | |
| ) | |
| if result.returncode != 0: | |
| raise GarakFailure( | |
| f"garak aggregate_reports failed (rc={result.returncode}): {result.stderr[-_LOG_TAIL_BYTES:]}" | |
| ) | |
| agg_html = agg_dir / f"{report_prefix}.report.html" | |
| with agg_html.open("w") as html_fd: | |
| result = subprocess.run( | |
| [garak_python, "-m", "garak.analyze.report_digest", "-r", str(agg_jsonl)], | |
| stdout=html_fd, | |
| stderr=subprocess.PIPE, | |
| text=True, | |
| check=False, | |
| ) | |
| if result.returncode != 0: | |
| raise GarakFailure(f"garak report_digest failed (rc={result.returncode}): {result.stderr[-_LOG_TAIL_BYTES:]}") | |
| agg_dir = persistent / "garak" / report_dir_name | |
| agg_dir.mkdir(parents=True, exist_ok=True) | |
| agg_jsonl = agg_dir / f"{report_prefix}.report.jsonl" | |
| agg_jsonl_tmp = agg_dir / f"{report_prefix}.report.jsonl.tmp" | |
| result = subprocess.run( | |
| [garak_python, "-m", "garak.analyze.aggregate_reports", "-o", str(agg_jsonl_tmp)] + jsonls, | |
| capture_output=True, | |
| text=True, | |
| check=False, | |
| ) | |
| if result.returncode != 0: | |
| if agg_jsonl_tmp.exists(): | |
| agg_jsonl_tmp.unlink() | |
| raise GarakFailure( | |
| f"garak aggregate_reports failed (rc={result.returncode}): {result.stderr[-_LOG_TAIL_BYTES:]}" | |
| ) | |
| agg_html = agg_dir / f"{report_prefix}.report.html" | |
| agg_html_tmp = agg_dir / f"{report_prefix}.report.html.tmp" | |
| with agg_html_tmp.open("w") as html_fd: | |
| result = subprocess.run( | |
| [garak_python, "-m", "garak.analyze.report_digest", "-r", str(agg_jsonl_tmp)], | |
| stdout=html_fd, | |
| stderr=subprocess.PIPE, | |
| text=True, | |
| check=False, | |
| ) | |
| if result.returncode != 0: | |
| if agg_jsonl_tmp.exists(): | |
| agg_jsonl_tmp.unlink() | |
| if agg_html_tmp.exists(): | |
| agg_html_tmp.unlink() | |
| raise GarakFailure(f"garak report_digest failed (rc={result.returncode}): {result.stderr[-_LOG_TAIL_BYTES:]}") | |
| os.replace(agg_jsonl_tmp, agg_jsonl) | |
| os.replace(agg_html_tmp, agg_html) |
🧰 Tools
🪛 ast-grep (0.44.1)
[error] 280-285: Use of unsanitized data to create processes
Context: subprocess.run(
[garak_python, "-m", "garak.analyze.aggregate_reports", "-o", str(agg_jsonl)] + jsonls,
capture_output=True,
text=True,
check=False,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(os-system-unsanitized-data)
[error] 280-285: Command coming from incoming request
Context: subprocess.run(
[garak_python, "-m", "garak.analyze.aggregate_reports", "-o", str(agg_jsonl)] + jsonls,
capture_output=True,
text=True,
check=False,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
[error] 293-299: Command coming from incoming request
Context: subprocess.run(
[garak_python, "-m", "garak.analyze.report_digest", "-r", str(agg_jsonl)],
stdout=html_fd,
stderr=subprocess.PIPE,
text=True,
check=False,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-auditor/src/nemo_auditor/jobs/audit.py` around lines 277 - 302,
Update the aggregate report flow around agg_jsonl and agg_html to write both
command outputs to temporary files in the same directory, keeping the existing
final paths untouched while either subprocess runs. Only after aggregate_reports
and report_digest both succeed should the temporary files replace the final
JSONL and HTML paths atomically; ensure temporary files are cleaned up on
failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local work around to get job submission working correctly - there is a hack in the platform job runner that makes the determination via gpu / cpu combinations which does not match auditor's deployment model. For right now adding local config to override that, but platform will hopefully support it out of the box very soon.
Also, copy back in the pause/resume code from previous release.
Summary by CodeRabbit