[C-Api] fix self-deadlock on single-shot invoke timeout - #691
Conversation
Code review (H1 / H2 of #690)
Verdict summaryThe fix itself is correct, minimal and correctly scoped. What is not yet in place is the merge gate: the added regression tests almost certainly do not execute in either PR-gating CI, and if they did, a regression would hang rather than fail. Details below. 1. Does it solve the stated problem? - YesVerified the deadlock chain and the fix:
The H2 hardening is also correct: 2. Regression risk to other modules - low, and the claims check out
Findings[Major] The new regression test does not run in either PR-gating CI
This repository's build dependencies do not pull it in:
So (This is a pre-existing gap that Please confirm against the actual GBS [Major] A regression manifests as a hang, and the CI test runner has no timeoutThe test note says a regression "hangs until the meson test timeout". That timeout does not apply in CI:
Consequently a regression would deadlock the binary and hang the job until GitHub Actions' 6-hour default job timeout - technically still a red check, but a 6-hour one with no indication of which test caused it, and on GBS builders it may not be bounded at all. Note also that because the deadlocked thread holds the global Suggest making the failure explicit and fast: run the invoke sequence on a worker [Minor]
|
|
Thanks — both Majors were correct, and I verified them before acting.
Fixed in cb22123. Major 1 — Major 2 — a watchdog thread ends the process after 30 s. It cannot be softer than that: the deadlocked thread parks holding the global Minor 1 ( Minor 3 (path coverage was timing-dependent) — deterministic now. The first invoke plus the 500 ms sleep goes through Not changed, with reasons: Minor 2 (loop bound) — Minor 4 (descriptor mutated outside the per-data lock) — taking Minor 5 / Nit 1 / Nit 2 — the header already documents that the data is not available after close. The two custom-filter tests now share 🤖 Generated with Claude Code |
Re-review of
|
| Item | Status |
|---|---|
| Major 1 - tests do not run in either PR-gating CI | Approach resolved, effect not yet achieved. Building the filter in-tree is the right answer and removes the dependency on nnstreamer's unittests RPM. But the new target does not compile (Blocker below), and a missing filter still yields a green test (Major 2 below), so the gate is not in place yet. |
| Major 2 - regression hangs, runner has no timeout | Resolved. The watchdog is correct and the margin is comfortable - see section 3. _exit(1) propagates through run_unittests.sh (run_entry returns $?, the -f branch does exit $?) and through the spec's %build, so a deadlock becomes a ~30 s red job. |
Minor 1 - single_h->filter == NULL turning a leak into an invalid free |
Resolved. fw_allocated = (data->destroy != NULL) is the right fallback: set_destroy_notify() sets destroy only on the allocate_in_invoke branch, so the descriptor is now cleared even with the filter gone, and the worst case reverts to a leak. |
| Minor 2 - clearing loop bound differs from the framework's | Not changed, reason accepted. data->num_tensors is exactly the range _ml_tensors_data_destroy_internal() would g_free(), which is the property the fix needs. |
| Minor 3 - path coverage implicit and racy | Resolved for two of the three sites. With a 200 ms invoke, a 10 ms timeout and the 500 ms sleep, the first release provably goes through __process_output() and the second through the JOIN_REQUESTED branch of invoke_thread(); the ML_ERROR_TRY_AGAIN tolerance is now dead code. The exit: site stays uncovered - agreed it is not forceable from the public API. |
| Minor 4 - descriptor mutated outside the per-data lock | Not changed, reasoning is correct (_ml_tensors_data_destroy_internal() holds G_LOCK_UNLESS_NOLOCK (*_data) across the destroy call, so re-taking it would deadlock). The one-line invariant comment was still not added - see Nit 3. |
Minor 5 / Nit 1 / Nit 2 - doc @note, duplication, ASSERT_* cleanup |
Not changed, accepted. nnstreamer-single.h:139-140 already states the buffer "will not [be] available for use later". _get_test_custom_filter() removes the worst of the duplication. |
Product-code diff (43/-14) re-verified: __release_output_data() covers all three re-entrant sites, g_list_remove() on a non-member is a harmless no-op on the paths where the output was never registered, and ml_single_close()'s g_list_foreach (..., __destroy_notify, ...) still runs after ML_SINGLE_HANDLE_UNLOCK. Behaviour for a non-allocating framework is unchanged. No new issue found in the C source.
2. New findings
[Blocker] The new meson target does not compile - the CI job that runs these tests is red
tests/capi/ml_api_customfilter_slow_allocator.c:21 includes <nnstreamer_plugin_api.h>, which pulls in <gst/gst.h>, but tests/capi/meson.build gives the target only dependencies: [glib_dep, nnstreamer_dep]. nnstreamer.pc declares Requires: nnstreamer-single and nnstreamer-single.pc declares an empty Requires:, with Cflags: -I${includedir}/nnstreamer - neither pulls in gstreamer's include path. Tizen GBS build on Ubuntu (x86_64, --define "unit_test 1") fails:
[29/58] Compiling C object tests/capi/libml_api_customfilter_slow_allocator.so.p/ml_api_customfilter_slow_allocator.c.o
FAILED: ...
cc ... -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nnstreamer ...
/usr/include/nnstreamer/nnstreamer_plugin_api.h:19:10: fatal error: gst/gst.h: No such file or directory
ninja: build stopped: subcommand failed.
error: Bad exit status from /var/tmp/rpm-tmp.QgtaVh (%build)
(https://github.com/nnstreamer/api/actions/runs/33857014167/job/100972550388)
The file needs only gst_tensor_info_get_size() and gst_tensors_info_copy(), and both are declared in nnstreamer_plugin_api_util.h, which includes nothing beyond glib. So the minimal fix is:
#include <nnstreamer_plugin_api_util.h>instead of<nnstreamer_plugin_api.h>, anddependencies: [glib_dep, nnstreamer_single_dep]instead ofnnstreamer_dep.
That is also the more correct dependency: tensor_filter_custom.c is part of nnstreamer_single_sources, gst_tensor_info_get_size() / gst_tensors_info_copy() live in nnstreamer_plugin_api_util_impl.c (likewise single), and c/meson.build:10 deliberately keeps the single-shot side on [glib_dep, gmodule_dep, nnstreamer_single_dep]. As written, the .so would carry a DT_NEEDED on libnnstreamer.so and drag GStreamer into a test binary that is otherwise single-only. If you prefer to keep nnstreamer_plugin_api.h, then add gst_dep, as nnstreamer's own tests/nnstreamer_example/meson.build:36 does for nnstreamer_customfilter_scaler_allocator.
Note that install-test is false in the GBS build, yet the target still fails: meson builds it regardless of install:. So this breaks Tizen unit_test 1 unconditionally, not only test-installing configurations.
[Major] A missing custom filter still passes silently
test_model = _get_test_custom_filter ();
if (test_model == NULL) {
/* cannot find the custom filter built along with this test */
return;
}Both custom-filter tests return green when the filter is not found. That was defensible while the filter came from an external package; it is not any more - the suite builds it itself, so absence means a build or layout bug, and the test should say so. ASSERT_TRUE (test_model != nullptr) (or at minimum GTEST_SKIP() << ..., which at least shows up in the report and the XML) is what makes the gate real. As it stands, renaming or dropping the meson target would silently remove the H1 regression gate again - precisely the failure mode the previous round was about.
[Major] The installed test can never find the filter, so the new debian/ml-api-unittests.install line is inert
_get_test_custom_filter() tries $MLAPI_BUILD_ROOT_PATH/tests/capi/<name> and then tests/capi/<name> relative to the cwd. Both are build-tree layouts:
debian/rulesexportsMLAPI_BUILD_ROOT_PATHandrun_unittests.shdoespushd build-> found. Good.packaging/machine-learning-api.specexports it too -> found. Good.- An installed run is not covered.
packaging/run-unittest.shdoespushd /usr/bin/unittest-ml/testsand runs the binaries from there with noMLAPI_BUILD_ROOT_PATH; the.installline puts the library at/usr/lib/nnstreamer/bin/unittest-ml/tests/libml_api_customfilter_slow_allocator.so, i.e. next to the binary, never under atests/capi/subdirectory. Both tests skip there.
Add a lookup beside the executable (the cwd, or via /proc/self/exe) so the installed / on-device runs exercise it too. Otherwise the .install entry only silences a dh_install --list-missing warning and buys no coverage. Combined with the Major above, this is the difference between "packaged and skipped" and "packaged and gating".
[Minor] The custom filter indexes the input arrays with the output tensor count
for (i = 0; i < prop->output_meta.num_tensors; i++) {
gsize size = gst_tensor_info_get_size (&prop->output_meta.info[i]);
gsize in_size = gst_tensor_info_get_size (&prop->input_meta.info[i]);
output[i].data = g_malloc (size);
memcpy (output[i].data, input[i].data, MIN (size, in_size));
}set_inputDim() copies in -> out, so the counts always coincide today and this is safe. But the loop reads input[i] / prop->input_meta.info[i] out of range the moment someone reuses this filter with a differing output count. Bound the loop by MIN (input_meta.num_tensors, output_meta.num_tensors), or assert equality in set_inputDim().
Otherwise the filter honours the tensor_filter_custom contract correctly, and I checked the two contract gates in custom_loadlib() / custom_open(): getInputDim == NULL, setInputDim != NULL, getOutputDim == NULL satisfies (!getInputDim != !setInputDim) && (!getOutputDim != !setInputDim), and invoke == NULL with allocate_invoke != NULL satisfies the invoke check. g_malloc in allocate_invoke paired with g_free in destroy_notify is a correct pair - better than the upstream scaler example, which malloc()s with no destroy_notify and relies on custom_destroyNotify()'s g_free fallback. Not setting output[i].size matches the upstream example (tensor_filter fills it from output_meta).
[Minor] The gate now rests on a single CI job
Only Tizen GBS ... unit_test 1 compiles and runs the unittests for a PR here. .github/workflows/pdebuild.yml produced no run for this head (gh run list shows only Spell Check, Static checkers, Tizen/GBS and Android Build Test). So after fixing the Blocker, please confirm from the GBS log that both custom-filter tests are reported as run and OK, not merely built. Worth checking carefully: because invoke_10_p / invoke_11_p / invoke_12_p have apparently never executed in this CI, the single-shot ML_NNFW_TYPE_CUSTOM_FILTER path is unproven in this environment, and the first green run is the only evidence that the new tests actually exercise anything.
[Nit] _exit() / <unistd.h>
The test file already special-cases __APPLE__ for the shared-object suffix; the new unconditional #include <unistd.h> and _exit() would not build on a Windows test configuration. Practically fine today, since the tests are POSIX-only.
[Nit] shared_module() would be more accurate than shared_library()
The artifact is only ever g_module_open()ed, never linked against.
[Nit] The invariant comment from the previous Minor 4 is still missing
The explanation in your reply - that __release_output_data() has sole ownership because the timed-out caller never received the handle, and that taking the per-data lock would deadlock against _ml_tensors_data_destroy_internal() - is exactly what the next reader will need. Two lines above __release_output_data().
3. Watchdog: correct, and no false-positive risk
Verified the timing is machine-independent rather than merely generous:
INVOKE_DELAY_USECis 200 ms ofg_usleep(), not CPU work, so a slow or loaded builder does not extend it; the 10 msml_single_set_timeout()therefore fires deterministically andEXPECT_EQ (status, ML_ERROR_TIMED_OUT)cannot flake. This is a real improvement over the previous "a 3 MiB scaler invoke must exceed 10 ms" assumption.- Expected wall time for
invoke_timeout_alloc_in_invoke_pis about 0.71 s (10 ms + 500 ms sleep + 10 ms + ~190 ms ofml_single_close()waiting oninvoking). Against a 30 s budget that is a ~40x margin, all of it ing_usleep/g_cond_wait_until, so it does not shrink on a slow CI machine. - The window covers the whole failure mode: on a regression the invoke thread parks holding
single_h->mutexwithG_LOCK (magic)taken, so the secondml_single_invoke()or theml_single_close()on the main thread blocks insideML_SINGLE_GET_VALID_HANDLE_LOCKED- inside the watchdog's window either way.
The one caveat is that _exit(1) skips the gtest XML writeout, so the CI artifact will not name the test; the g_printerr() message covers that in the console log. Acceptable.
4. Scope, packaging, regression risk
- Scope is proportionate. 43/-14 in
ml-api-inference-single.c, 285 lines of test, a 104-line test-only filter, 8 lines of meson, 1 line of debian. Nothing outside the single-shot internals and the test suite;__destroy_notify()and__release_output_data()remain static. - RPM
%filesis safe.%files -n capi-machine-learning-unittestsuses the directory glob%{_bindir}/unittest-ml, so the new.sois packaged automatically wheninstall-testis on; and in the normalunit_test 1GBS buildinstall_testis-Dinstall-test=falseanyway (onlyrelease_testflips it), so nothing changes there. - Debian
dh_installis safe.unittest_install_dirmaps to/usr/lib/nnstreamer/bin/unittest-ml/testsunder--bindir=lib/nnstreamer/bin, and the added glob matches.--list-missing(not--fail-missing) would only have warned. unit_test 0GBS (armv7l / aarch64) is unaffected --Denable-test=falsemeanssubdir('tests')is never entered.- Android is unaffected - it does not go through meson; all four
Android Build Testjobs are green. - No single-only meson configuration exists that would need the new target excluded.
- No sub-plugin name collision: nnstreamer scans configured directories for the
libnnstreamer_customfilter_prefix;libml_api_customfilter_*cannot be picked up accidentally, and it is not installed into a scanned directory. - Documentation: none required.
nnstreamer-single.h:139-140already documents that a framework-allocated output is freed at close and unavailable afterwards. The optional@noteonml_single_close()remains a Nit.
Overall verdict: Request changes - not approvable
The C fix is correct and I would approve it on its own; the two Majors from the previous round were answered in substance, and the watchdog in particular is now well engineered. What blocks merge is concrete and mechanical:
- Must fix: the Blocker -
Tizen GBS ... unit_test 1fails to compile the new custom filter, so the branch cannot be merged in any case. One include and one dependency. - Must fix: the silent-skip Major - with the filter now built in-tree,
_get_test_custom_filter() == NULLshould fail the test, not pass it. Without this the merge gate is still nominal. - Should fix: the installed-run lookup, so the shipped
debian/ml-api-unittests.installentry is actually usable. - After the fix: please confirm from the GBS log that
invoke_timeout_alloc_in_invoke_pandclose_before_data_destroy_01_pare reported as run and OK - this repo's CI has never executed the custom-filter single-shot path before.
Remaining Minor / Nit items are fine as follow-ups. The PR is still a Draft carrying DO NOT MERGE, which is consistent with the above.
|
The Blocker was already fixed in 2f3c00f, pushed before this review landed — the review ran against cb22123. The two new Majors are fixed in 47bae8c. Installed runs could not find the filter — right, and I had only checked Silent skip — fixed, and the reasoning is the point: a silent skip is exactly how the timeout test lost its teeth to begin with. Now that this suite builds the filter itself, not finding it means the test setup is broken rather than an optional dependency being absent, so both tests Input indexed with the output count — fixed. Only the GBS Nits — 🤖 Generated with Claude Code |
|
CI is green on 47bae8c, and the The three pre-existing custom-filter tests report 0 ms, which is them taking The new ones did run. 200 ms is one invoke through the filter this suite builds, whose To be explicit about what is not empirically shown: I did not push a revert of the C change to watch the test go red. The gate rests on the deadlock being certain from the code (a non-recursive 🤖 Generated with Claude Code |
3rd review of
|
| Round | Item | Status |
|---|---|---|
| 2 | [Blocker] new meson target does not compile (gst/gst.h not found) |
Resolved in 2f3c00f. gst_dep is defined unconditionally at meson.build:29, so nothing else can break by adding it. The GBS x86_64 log confirms [28/58] Compiling ... ml_api_customfilter_slow_allocator.c.o and [42/58] Linking target tests/capi/libml_api_customfilter_slow_allocator.so. This is the review's fallback option rather than the preferred nnstreamer_plugin_api_util.h + nnstreamer_single_dep; see Nit 1 - a cosmetic difference, not a defect. |
| 2 | [Major] a missing custom filter still passes silently | Resolved in 47bae8c. Both tests now ASSERT_TRUE (test_model != NULL). |
| 2 | [Major] the installed test can never find the filter, so the .install line is inert |
Resolved in 47bae8c. The /proc/self/exe candidate covers both installed layouts - Tizen %{_bindir}/unittest-ml/tests (where run-unittest.sh pushdes) and Debian /usr/lib/nnstreamer/bin/unittest-ml/tests - because unittest_install_dir puts the .so in the same directory as the binary. It also covers a developer running the binary from an arbitrary cwd. |
| 2 | [Minor] filter indexes the input arrays with the output tensor count | Resolved, and more strictly than suggested: pt_allocate_invoke() returns -1 on a count mismatch instead of clamping with MIN(). For a test filter, failing loudly is the better of the two. |
| 2 | [Minor] the gate rests on a single CI job | Confirmed and settled empirically - see section 3. |
| 2 | [Nit] _exit() / <unistd.h>; shared_module() vs shared_library() |
Not changed, accepted. |
| 2 | [Nit] invariant comment above __release_output_data() |
Partially addressed. The @note added ("its destroy callback takes the same mutex again") records the deadlock reason, which is the half a future reader needs most. The sole-ownership half is still unwritten. Fine as-is. |
| 1 | Major 1 / Major 2 / Minor 1 / Minor 3 | Confirmed still resolved at HEAD; the round-2 assessment holds unchanged. |
| 1 | Minor 2 / Minor 4 / Minor 5 / Nit 1 / Nit 2 | Not changed, reasons previously accepted; nothing at HEAD changes that. |
I re-verified the C diff at HEAD independently rather than carrying the earlier conclusion forward:
__destroy_notify(): thefw_allocated = (data->destroy != NULL)fallback is only read whensingle_h->filteris NULL; otherwiseallocate_in_invoke()overrides it, which is the pre-existing semantics. Every entry that can reach it withdestroy == NULLcomes fromset_destroy_notify (…, add=TRUE)at:1616on a non-allocating framework, whereallocate_in_invoke()is FALSE, so nothing is cleared and the behaviour is byte-for-byte the old one. No path clears buffers that the data handle owns.__release_output_data():g_list_remove()on a non-member is a no-op;ml_tensors_data_destroy()after__destroy_notify()seesdestroy == NULLandg_free (NULL)per tensor - neither a leak nor a double free.- The three re-entrant sites (
__process_outputtimeout branch,invoke_thread'sstatus != NONE || JOIN_REQUESTEDbranch,invoke_thread'sexit:) are all routed through the helper. The synchronous (timeout == 0)ml_tensors_data_destroy (_out)at:1628still runs under the mutex, but_outthere never haddestroyset - unchanged, and still not a fourth site.
2. Do the new lookup and the ASSERT transition create false failures?
I went through this specifically, because turning a skip into an assertion is exactly the kind of change that trades one CI problem for another. I could not construct a legitimate configuration that now fails spuriously.
- No single-only meson configuration exists to break.
gst_dep(meson.build:29) andnnstreamer_dep(:32) are unconditional top-level dependencies; the project already fails to configure without GStreamer. Addinggst_depto one test target cannot narrow the set of buildable configurations. enable-test=false(GBSunit_test 0, i.e. the armv7l and aarch64 jobs) -meson.build:219gatessubdir('tests'), so neither the filter nor the test binary is built and%checknever runs these. Both jobs are green on47bae8c.install-teston or off - theshared_library()target is unconditional and carries the sameinstall:option asunittest_capi_inference_single. If the binary exists, the filter exists next to it. There is no combination that builds the binary without the filter.- All four runner layouts are covered:
debian/rulesexportsMLAPI_BUILD_ROOT_PATH-> candidate 1.packaging/machine-learning-api.spec:414exports it too -> candidate 1.packaging/run_unittests.shdoespushd build-> candidate 2 (tests/capi/…).packaging/run-unittest.sh(installed / on-device)pushdes/usr/bin/unittest-ml/tests-> candidate 3 (/proc/self/exe).meson test/ninja test->tests/meson.buildsetsMLAPI_BUILD_ROOT_PATHintestenv-> candidate 1.
ML_NNFW_TYPE_CUSTOM_FILTERis always available, so theASSERT_EQonml_single_open()that now becomes reachable cannot fail for lack of a sub-plugin:tensor_filter_custom.cis innnstreamer_single_sourcesunconditionally (gst/nnstreamer/tensor_filter/meson.build:1-6), so whereverlibnnstreamer-singleexists,customexists. Empirically confirmed - this CI had never executed that path before, and it works.- gcov builds skip this binary entirely (
spec:427,%if 0%{?unit_test} && !0%{?gcov}). - Cross builds (armv7l / aarch64 GBS) are the
unit_test 0case above -%checkis not reached for this binary. - The only gap is macOS running from an installed layout - no
/proc, so it would now hard-fail instead of skipping. macOS development runs from the build tree, which candidates 1 and 2 cover, and this project has no macOS CI. Nit at most.
The gst_dep route is also not a runtime hazard, which was worth confirming since it puts a DT_NEEDED on libnnstreamer.so into a .so that is dlopen()ed by an otherwise single-only test binary (b_asneeded=false means the entry is kept even though unused). nnstreamer's gst/nnstreamer/meson.build:107 does nnstreamer_deps += nnstreamer_single_dep, i.e. libnnstreamer.so links against libnnstreamer-single.so rather than statically re-containing it, so there is no duplicated sub-plugin registry and no duplicate-symbol interposition. The cost is load time, not correctness.
3. Is the CI log interpretation sound? Yes - I re-derived it from the raw log
Downloaded job 100976391347 and confirmed the quoted block verbatim, plus one line the excerpt omitted:
[ RUN ] nnstreamer_capi_singleshot.close_before_data_destroy_02_p
** Message: 09:46:01.910: accl = cpu
[ OK ] nnstreamer_capi_singleshot.close_before_data_destroy_02_p (0 ms)
- 0 ms = skip is correct for
invoke_10_p/invoke_11_p/invoke_12_p(andset_input_info_success_02, same cause): they takegoto skip_testbecausennsconf_get_custom_value_string ("filter", "customfilters")finds nolibnnstreamer_customfilter_*. The first review'snnstreamer-unittestsfinding is confirmed at runtime. - 0 ms is not a skip for
close_before_data_destroy_02_p. Theaccl = cpumessage is emitted betweenRUNandOK, which can only happen afterml_single_open()succeeded onadd.tflite. tflite is genuinely enabled in this build (invoke_01takes 132 ms on the same model). So that test really ran; 0 ms is just resolution on a one-element model. - 200 ms is one real invoke through
pt_allocate_invoke()'sg_usleep (200000). Nothing else in that test sleeps. - 801 ms decomposes exactly as claimed, and - more to the point - it proves which branches ran: 10 ms to the first timeout, the invoke thread completes at t≈200 ms and takes
__process_output()'s timeout branch; 500 ms gap; the second invoke times out at t≈520 ms,ml_single_close()setsJOIN_REQUESTEDand blocks oninvoking, the invoke thread completes at t≈710 ms and takes thestatus != NONE || JOIN_REQUESTEDbranch. Two of the three fixed sites are demonstrably executed, in the order the test comment claims. That is a stronger statement than "the test ran".
One clarification on what the green run does and does not prove. cb22123's GBS job was red and 47bae8c's is green, so the job demonstrably gates on this binary; and run_unittests.sh's -f branch (exit $?) inside rpm %build (set -e) propagates a failure. What has never executed is the watchdog's _exit(1) arm itself. I agree with your reasoning that the deadlock is certain from the code and that a revert experiment is not required for correctness - but if you want one datum for the price of one run, temporarily lowering the watchdog budget below the test's own ~0.71 s would exercise the abort path and its exit-code plumbing without touching the C fix. Entirely optional.
4. Final pass over the cumulative change (4764a2f + cb22123 + 2f3c00f + 47bae8c)
- Topic solved: yes. The H1 self-deadlock is closed at all three re-entrant sites, and H2 is hardened so the fix no longer silently depends on nnstreamer's
g_tensor_filter_single_destroy_notify()NULLing the descriptors for it. - Regression risk: low. Product diff is 43/-14, entirely inside
ml-api-inference-single.c;__destroy_notify()and__release_output_data()are static. No public API, ABI, or header change.ml_single_invoke_fast()(the Android JNI path) never entersdestroy_data_listand is untouched; all four Android jobs are green. - Size: 454/-14 total, of which 402 lines are test scaffolding (295 test + 107 filter). Proportionate for a defect whose regression signature is a process-wide hang.
- Test completeness: good, and the pairing is the part I would call out as well designed.
close_before_data_destroy_01_ppins the new "NULL buffer, size 0 after close" behaviour on an allocating framework, andclose_before_data_destroy_02_ppins the opposite on tflite - which is what stops the clearing from later being widened to buffers the data handle owns.invoke_timeout_alloc_in_invoke_pcovers two of the three fixed sites with machine-independent timing plus a watchdog. Theexit:site remains uncovered; I agree it is not forceable through the public API. - Documentation: none required.
nnstreamer-single.h:139-141already states that a framework-allocated output "will not [be] available for use later". An@noteonml_single_close()remains optional.
5. Remaining items, all trivially fixable inside this PR if you want them
None of these is a merge condition.
- [Nit] The
gst_deproute is the heavier of the two options. Two lines get the lighter one:#include <nnstreamer_plugin_api_util.h>instead of<nnstreamer_plugin_api.h>inml_api_customfilter_slow_allocator.c:21, anddependencies: [glib_dep, nnstreamer_single_dep]intests/capi/meson.build:4. I checked that the file needs nothing else:gst_tensor_info_get_size()is declared atnnstreamer_plugin_api_util.h:48,gst_tensors_info_copy()at:227, andtensor_filter_custom.hpulls onlytensor_typedef.h+nnstreamer_plugin_api_filter.h, which itself includes onlytensor_typedef.h. That drops bothlibgstreamer-1.0andlibnnstreamer.sofrom the test.so. The current form is correct and green, so this is aesthetics plus load time. - [Nit] The
ML_ERROR_TRY_AGAINtolerance is now dead code.EXPECT_TRUE (status == ML_ERROR_TIMED_OUT || status == ML_ERROR_TRY_AGAIN)ininvoke_timeout_alloc_in_invoke_p: with a 200 ms invoke and a 500 ms gap the first job is always long finished, soTRY_AGAINis unreachable. Tightening it toEXPECT_EQ (status, ML_ERROR_TIMED_OUT)makes the second release site's coverage asserted rather than merely tolerated - the same argument that motivated replacing the silent skip. One line. - [Nit] macOS + installed layout would now hard-fail rather than skip, since
/proc/self/exeis absent and only build-tree candidates remain. No CI configuration hits it. If you want it airtight,_NSGetExecutablePath()under__APPLE__- but I would not bother. - [Nit] The three older custom-filter tests stay permanently dead in this CI (0 ms, confirmed). Now that the suite ships its own allocating filter,
invoke_11_p/invoke_12_pcould be pointed at it and would start testing something. Agreed this is off-topic for this PR; worth a follow-up issue so the 0 ms lines are not rediscovered a third time.
And one observation that is not a defect in this PR, recorded so it is not mistaken for verified coverage:
- [Minor - informational] The Debian side of this change is untested and cannot be tested here.
.github/workflows/pdebuild.ymlisdisabled_inactivityat the repository level (gh api repos/nnstreamer/api/actions/workflows), so it neither runs on PRs nor fires its nightly cron - which is why no pdebuild run appeared for any commit on this branch. The newdebian/ml-api-unittests.installline and theoverride_dh_auto_testexecution of these tests have therefore never run. By inspection the line is correct:--bindir=lib/nnstreamer/binmakesunittest_install_dir=/usr/lib/nnstreamer/bin/unittest-ml/tests, the added glob matches, anddebian/ruleshardcodes-Dinstall-test=true, sodh_install --list-missinghas nothing to warn about. Pre-existing repo condition, not caused by this PR, and not a reason to hold it. Re-enabling that workflow deserves its own issue.
Overall verdict: Approve - safe to merge
Both round-2 Majors and the round-2 Blocker are resolved in code, not merely in intent, and I verified each against the artefacts rather than the commit messages. The merge gate this PR set out to build now exists and is demonstrably live: the new tests execute in Tizen GBS … unit_test 1, the 801 ms figure proves they reach two of the three fixed sites, a missing filter now fails instead of passing, and a regression turns into a ~30 s red job instead of a multi-hour stall.
The C fix is correct, minimal and correctly scoped, and the round-1 conclusion that no fourth re-entrant site exists still holds at HEAD. Nothing outstanding rises above Nit. From my side the DO NOT MERGE label and Draft status can be lifted.
|
Thanks. On the one gap you flagged — the watchdog's So the abort fires, the exit code survives Not taking the remaining Nits into this PR, with reasons:
Taking this out of draft. 🤖 Generated with Claude Code |
myungjoo-bot
left a comment
There was a problem hiding this comment.
Automated review (transcribed from an AI review agent's report; please verify before acting).
Summary: The PR routes the three in-thread releases of a timed-out, framework-allocated output (__process_output, the status != NONE || JOIN_REQUESTED branch, and the exit: label of invoke_thread) through a new __release_output_data() that calls __destroy_notify() (clearing data->destroy) before ml_tensors_data_destroy(), and makes __destroy_notify() NULL/zero the descriptors it hands back to the framework (H2). The defect was confirmed on upstream/main: ML_SINGLE_GET_VALID_HANDLE_LOCKED takes G_LOCK (magic) then g_mutex_lock (&single_h->mutex) and only then releases magic; _ml_tensors_data_destroy_internal (ml-api-common.c:692-727) invokes data->destroy = ml_single_destroy_notify_cb -> the macro -> self-deadlock on the non-recursive handle mutex while holding magic. Every other ml_tensors_data_destroy under the handle mutex was checked and operates on destroy == NULL data; ml_single_close calls __destroy_notify directly after unlocking; ml_single_invoke_fast and the JNI path never enter the list; ml_single_invoke_dynamic funnels into ml_single_invoke. __destroy_notify takes no locks and only reaches the sub-plugin's destroyNotify with no re-entry into ML-API. The GBS x86_64 unit_test 1 log shows close_before_data_destroy_01_p, close_before_data_destroy_02_p, and invoke_timeout_alloc_in_invoke_p all OK; all 10 checks green at 47bae8c; merge-tree clean; DCO on all four commits. Approving; items below are non-blocking.
- [Low] Commit hygiene —
cb22123(tests/capi/meson.build) does not build on Tizen on its own (missinggst_dep; its GBS run failed) and is fixed only by2f3c00f. It also carries a product-code change (the__destroy_notifyfallback ondata->destroy) inside a[Test]commit. Suggest squashing2f3c00fintocb22123(and ideally47bae8c) so every commit builds, or moving the__destroy_notifyhunk into4764a2f. - [Low] Description precision — "three call sites are affected [by the self-deadlock]" is imprecise: the
JOIN_REQUESTEDbranch (:583) and:609) run afterexit:(ml_single_closezeroedmagic, so on base the callback returnedML_ERROR_INVALID_PARAMETERand_ml_tensors_data_destroy_internalbailed early — a leak of the handle and framework buffers, not a hang. The fix is still correct and desirable there. Suggest wording it as "deadlock at__process_output, leak at the two close-time sites"; the second half ofinvoke_timeout_alloc_in_invoke_p(~:3716-3721) guards the leak path. - [Low] Docs — H2 is consistent with the contract in
c/include/nnstreamer-single.h:139-141(close-before-destroy). The only observable delta isml_tensors_data_get_tensor_data()reportingsize == 0after close (previously original size with a NULL pointer). Acceptable; optionally extend the@notewith "after ml_single_close(), such an output reports a NULL buffer of size 0". - [Low] Test note —
unittest_capi_inference_single.cc:~3709: with a 10 ms timeout, if the invoke thread is not scheduled before the timeout fires, the thread sees NULL input/output (pre-existing #690 L9). With this PR's NULL guard that is a leak, not a failure, so the test cannot flake on it; noted so nobody mistakes the leak for a regression of this PR. - [Low] Custom filter —
tests/capi/ml_api_customfilter_slow_allocator.c:74-75indexes&prop->output_meta.info[i]directly; nnstreamer's examples usegst_tensors_info_get_nth_info()so tensors beyondNNS_TENSOR_SIZE_LIMITresolve toextra. Harmless here (one tensor); switch for parity. Everything else checks out (setInputDim, 200 ms sleep +g_mallocper output inallocate_invoke,destroy_notifyfrees, deps match nnstreamer's example filters, Doxygen on every function). - [Low] "fail loudly" commit — the filter is built unconditionally alongside the test binary, so
ASSERT_TRUE (test_model != NULL)cannot fire in any lane that builds the tests; the/proc/self/exefallback covers the installed/usr/bin/unittest-ml/testslayout. Packaging is consistent:debian/ml-api-unittests.installmatchesunittest_install_dir, the spec's%files -n capi-machine-learning-unittestsuses the%{_bindir}/unittest-mldirectory glob, and%checkexportsMLAPI_BUILD_ROOT_PATH. The pdebuild workflow did not run for this PR (check-rebuild gating), so the Debian line is unverified by CI but correct by inspection. - [Low] Follow-up outside this PR —
c/src/ml-api-common.c:~909: after H2 zeroing,ml_tensors_data_clone()on a closed output doesmemcpy (dst, NULL, 0)(technically UB; previously a use-after-free read, so strictly an improvement). Aif (_in->tensors[i].size)guard would make it clean.
Verified no-issues: the helper preserves behavior for destroy == NULL data and for NULL output; the app's later ml_tensors_data_destroy after close is a guaranteed no-op; ml_tensors_data_set_tensor_data on a STATIC handle now returns ML_ERROR_INVALID_PARAMETER instead of memcpy into NULL; the tests would hang on base at the first ml_single_invoke (watchdog _exit (1) after 30 s, propagated by run_unittests.sh); cleanup is complete on the pass path.
No back-door or suspicious behavior found.
|
Thanks for the approval. Acted on 2, 3 and 5 in d9b344f; 1 needs your call. 2 — description precision. You are right that I flagged all three sites as deadlocks, and that the close-time ones are not. One correction to the suggested wording, though: it is not cleanly "deadlock at 3 — docs. Added to the 5 — custom filter. Switched to 4, 6 — noted, no change. Agreed on both, and thanks for pinning down that the L9 NULL path is a leak rather than a flake under this PR's guard; that is worth having on the record. 7 — 1 — commit hygiene. You are right that it matters here: I did not rewrite it. Collapsing to one product commit plus one test commit needs a force-push over a branch that three review rounds and an approval already point at, and that felt like your decision rather than mine. Two ways to settle it:
CI is running on d9b344f; I will report when it lands. |
|
CI is green on d9b344f — all 10 checks — and the tests still do their work after the Same 200 ms and 800 ms shape as on 47bae8c, so the filter still allocates in invoke and both timeouts still fire. The only thing still open is the commit hygiene question above: rewrite into one product commit plus one test commit and force-push, or squash-merge and leave the branch as it is. |
|
Automated incremental review (transcribed from an AI review agent's report; please verify before acting). Scope: Verdict: no Blocker, no Major, no Minor. Two Nits, both cosmetic. Approval stands and extends to d9b344f. 1.
|
|
Confirmed the constant correction, and it upgrades what item 5 actually was. That also means d9b344f's commit message is wrong where it says The other two Nits (wording "inference engine" vs "neural network framework" in the header, the optional |
When an invoke times out, ml_single_invoke() registers the abandoned output in destroy_data_list and, for a framework that allocates the output in invoke, sets its destroy callback to ml_single_destroy_notify_cb(). The invoke thread later releases that output with ml_tensors_data_destroy() while holding single_h->mutex, so the callback re-enters ML_SINGLE_GET_VALID_HANDLE_LOCKED() and waits for the non-recursive mutex it already holds. That wait happens with the global magic lock taken, so every single-shot API in the process blocks. Three call sites release such an output under the mutex, and they do not all fail the same way. __process_output() deadlocks as described. The status != ML_ERROR_NONE || JOIN_REQUESTED branch deadlocks when an invoke fails after a timeout with no close pending, but merely leaks when it is a close that woke it, because ml_single_close() has zeroed magic by then and the callback returns ML_ERROR_INVALID_PARAMETER before reaching the mutex, which makes _ml_tensors_data_destroy_internal() bail out and drop the handle. The exit: label, reached only on JOIN_REQUESTED, leaks for the same reason. Release such an output through __destroy_notify() first and let ml_tensors_data_destroy() free the handle afterwards. The destroy callback is already cleared by then, so the handle mutex is taken once and the framework buffers go back to the framework on every path. __destroy_notify() now also clears the tensor pointers and sizes it has handed back, so nothing frees or copies into them again. This no longer relies on the tensor-filter vtable clearing them on our behalf, which its contract does not promise. Where the filter is already gone, the destroy callback says the same thing, so the descriptor is cleared rather than left for a plain g_free(). The only observable change is that such an output reports a NULL buffer of size 0 after ml_single_close(), where it previously reported the original size with a NULL pointer, so say that in the header next to the close-before-destroy note. Related to nnstreamer#690 (items H1 and H2) Signed-off-by: MyungJoo Ham <myungjoo.ham@samsung.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The custom filter these tests would naturally use, libnnstreamer_customfilter_scaler_allocator, ships only in nnstreamer's unittests RPM and in no Debian package, and neither packaging/machine-learning-api.spec nor debian/control pulls it in. The existing invoke_10_p, invoke_11_p and invoke_12_p have been skipping silently for that reason, each reporting 0 ms. A regression in the timeout path also shows up as a hang, and packaging/run_unittests.sh starts the gtest binaries with no timeout, so it would stall the job rather than fail it. Build a custom filter for the test suite itself. It allocates the output in invoke, so the single-shot handle takes the allocate_in_invoke path, and each invoke sleeps long enough for a short ml_single_set_timeout() to fire regardless of how fast the machine is. It is found through MLAPI_BUILD_ROOT_PATH, which both debian/rules and the spec %check export, with fallbacks for the build directory that run_unittests.sh leaves as the cwd and for the installed layout that run-unittest.sh uses. Since it is built alongside the test binary, not finding it means the setup is broken, so the tests assert on it rather than skip. Guard the timeout test with a watchdog thread that ends the process after 30 seconds. A self-deadlock parks the invoke thread while it holds the global handle lock, so the run cannot continue anyway; failing right away turns a stalled job into a normal test failure. invoke_timeout_alloc_in_invoke_p times out twice, once with the handle open and once with a close pending, to reach the first two release sites. close_before_data_destroy_01_p checks the close-before-destroy order documented in nnstreamer-single.h on an allocating framework, and close_before_data_destroy_02_p checks that a tensorflow-lite output, which the handle owns, is untouched by the same close. Related to nnstreamer#690 (items H1 and H2) Signed-off-by: MyungJoo Ham <myungjoo.ham@samsung.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d9b344f to
608a2aa
Compare
|
History rewritten as agreed. Five commits became two, force-pushed with a lease on d9b344f:
Each builds on its own now, so The two remaining Nits went in with the rewrite:
Only content change against d9b344f is that header wording; CI is running. The approval on 47bae8c is necessarily stale now — happy to have it re-run against 608a2aa once the checks land. |
Addresses items H1 and H2 of #690.
H1 — self-deadlock after an invoke timeout (the actual defect)
ml_single_invoke()with a timeout registers the abandoned output indestroy_data_listand, for a sub-plugin withallocate_in_invoke == TRUE,sets
data->destroy = ml_single_destroy_notify_cb.invoke_thread()then finishes the invoke, re-takessingle_h->mutex, andreleases that output with
ml_tensors_data_destroy(). That call dispatchesml_single_destroy_notify_cb(), which re-entersML_SINGLE_GET_VALID_HANDLE_LOCKED()and blocks on the non-recursive mutex thesame thread already holds. The macro takes the global
magiclock beforethe handle mutex and only releases it afterwards, so the thread parks while
holding
magicand every single-shot API in the process stops.Three call sites release such an output under the mutex —
__process_output()and the two
ml_tensors_data_destroy (output)calls ininvoke_thread()— butthey do not all fail the same way. Where the handle is still open, the callback
reaches the mutex and hangs. Where a close is already in progress,
ml_single_close()has zeroedmagicthroughML_SINGLE_GET_VALID_HANDLE_LOCKED (single_h, single, 1), so the callbackreturns
ML_ERROR_INVALID_PARAMETERbefore touching the mutex,_ml_tensors_data_destroy_internal()bails out on that error, and the handleplus the framework buffers leak instead.
So:
__process_output()deadlocks. Thestatus != ML_ERROR_NONE || JOIN_REQUESTEDbranch deadlocks when an invoke fails after a timeout with noclose pending, and leaks when it is the close that woke it. The
exit:label,reached only on
JOIN_REQUESTED, leaks. All three want the same fix.The fix routes them through a small helper that calls
__destroy_notify()first (which clears the destroy callback) and only then
ml_tensors_data_destroy(), so the handle mutex is taken exactly once. Datawithout a destroy callback keeps its existing path unchanged, and a NULL
output — reachable when the timeout fires before the invoke thread picks the
job up — is ignored as before.
Reproduction:
ml_single_set_timeout()shorter than one invoke, on aframework that allocates the output in invoke (tensorflow, custom filters with
allocate_invoke).H2 — clearing the buffers handed back to the framework
Worth recording: the double free described in #690 H2 is not reachable
today.
g_tensor_filter_single_destroy_notify()in nnstreamer already setsmem[i].data = NULLfor every output tensor, so the app's laterml_tensors_data_destroy()finds NULL pointers and frees nothing.But nothing in the vtable contract (
tensor_filter_single.h) promises that,and the H1 fix above now depends on it —
__destroy_notify()is what makes thefollowing
ml_tensors_data_destroy()a no-op on the buffers. So__destroy_notify()clears the descriptor itself. The size is zeroed togetherwith the pointer:
ml_tensors_data_set_tensor_data()on a STATIC handlevalidates
data_sizeagainsttensors[i].sizeand would otherwisememcpy()into a NULL pointer.
Tests
Added to
tests/capi/unittest_capi_inference_single.cc:invoke_timeout_alloc_in_invoke_p— times out twice againstlibnnstreamer_customfilter_scaler_allocator, once with the handle open (hits__process_output) and once with a close pending (hits theinvoke_threadjoin path), then closes the handle. A regression deadlocks the process, so
this hangs until the meson test timeout rather than failing an assertion; that
is noted in the test.
close_before_data_destroy_01_p— the close-before-destroy order documentedin
nnstreamer-single.h, on an allocating framework. Asserts the outputreports no buffer after close and that writing into it is rejected instead of
faulting.
close_before_data_destroy_02_p— the same order on tensorflow-lite, whichdoes not allocate in invoke. Asserts the buffer is still valid after close, so
the clearing above cannot be widened to data the handle owns.
All three skip when their model or custom filter is unavailable, following the
existing convention in the file.
Verification
ml-api-inference-single.cand the test file were compiled clean(
-Wall -Wextra), andclang-formatand the CIindentoptions report nochange in the touched regions. A full build could not be run locally — the
machine has no nnstreamer devel package or gtest — so the functional runs are
left to CI.
Scope
No public API, ABI, or documented behaviour changes. Only the single-shot
internals are touched;
__destroy_notifyand the new helper are static to thisfile. The only externally observable difference is that an output from an
allocating framework reports size 0 after
ml_single_close(), where itpreviously reported the original size with a NULL pointer.
ml_single_invoke_fastoutputs (the path the Android JNI binding uses) never enter the destroy list and
are untouched.
🤖 Generated with Claude Code