[BREAKING] testing/cmocka: Compile cmocka as a libary only by default

[linguini1]

Summary

BREAKING CHANGE

This commit introduces a Kconfig switch to include the cmocka binary built from cmocka_main.c. The default behaviour is now changed so that cmocka is built as a library only, which would be the desired behaviour for users creating their own cmocka projects.

To restore the old behaviour (where the cmocka program is compiled), add CONFIG_TESTING_CMOCKA_PROG=y to your legacy configuration.

Impact

Cmocka is now only built as a library, so the cmocka binary is no longer
included by default.

PR (apache/nuttx#17653) to the NuttX kernel preserves the compilation of the cmocka binary for the two existing defconfigs which use the cmocka application.

Testing

Compilation of the cmocka libary without the binary using sim:nsh as a basis:

$ make -j
Create version.h
LN: platform/board to /home/linguini/coding/nuttx-space/apps/platform/dummy
Register: dd
Register: nsh
Register: sh
Register: gcov
Register: hello
Register: dumpstack
Register: gpio
Register: ostest
CP:  /home/linguini/coding/nuttx-space/nuttx/include/nuttx/config.h
CP:  /home/linguini/coding/nuttx-space/nuttx/include/nuttx/fs/hostfs.h
LD:  nuttx
Pac SIM with dynamic libs..
'/usr/lib/libm.so.6' -> 'sim-pac/libs/libm.so.6'
'/usr/lib/libz.so.1' -> 'sim-pac/libs/libz.so.1'
'/usr/lib/libc.so.6' -> 'sim-pac/libs/libc.so.6'
'/usr/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/libs/ld-linux-x86-64.so.2'
'/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
SIM elf with dynamic libs archive in nuttx.tgz

Then, when disabling CONFIG_TESTING_CMOCKA_ASLIB, the compilation is:

$ make -j
Create version.h
LN: platform/board to /home/linguini/coding/nuttx-space/apps/platform/dummy
Register: cmocka
Register: hello
Register: nsh
Register: sh
Register: dd
Register: gcov
Register: gpio
Register: dumpstack
Register: ostest
CP:  /home/linguini/coding/nuttx-space/nuttx/include/nuttx/config.h
LD:  nuttx
Pac SIM with dynamic libs..
'/usr/lib/libm.so.6' -> 'sim-pac/libs/libm.so.6'
'/usr/lib/libz.so.1' -> 'sim-pac/libs/libz.so.1'
'/usr/lib/libc.so.6' -> 'sim-pac/libs/libc.so.6'
'/usr/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/libs/ld-linux-x86-64.so.2'
'/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
SIM elf with dynamic libs archive in nuttx.tgz

and I can run the cmocka application on sim:

nsh> cmocka --help
an elegant unit testing framework for C with support for mock objects
Usage: cmocka [OPTION [ARG]] ...
 -?, --help       show this help statement
 -l, --list       display only the names of testcases and testsuite,
                  don't execute them
 -t, --test A     only run cases where case function name matches A pattern
 -p, --skip B     don't run cases where case function name matches B pattern
 -s, --suite C    only run suites where PROGNAME matches C pattern
 -f, --output-path use xml report instead of standard output
 -d, --shuffle-seed shuffling test sequence,between 0 and 99999,
                   when seed is 0,use time(NULL) as the seed for
                   the random number generator
Example: cmocka --suite mm|sched --test Test* --skip TestNuttxMm0[123]

nsh>

[linguini1]

Build is failing because sim:citest needs the change in apache/nuttx#17653

[linguini1]

Any ideas why the CI test is failing? It seems that the cmocka binary is present, just an assertion is failing.

[acassis]

@linguini1 it is saying that boards/risc-v/qemu-rv/rv-virt/configs/citest/logs/ is not tracked, not sure if it needs to be.

Maybe @simbit18 or @lupyuen has some idea, they know more about the citest

[simbit18]

Please excuse my silence, but I have some important family matters to resolve. Unfortunately, real life is not binary, 0 and 1, black and white, etc.,
but full of nuances, and its complexity requires time and patience.
When I have time, I will take a closer look at this PR.

[cederom]

Thank you @linguini1 very nice hack for reuse of cmocka! Looks like you like it? :-)

One remark - we are changing default behavior therefore we should mark this change as breaking correct? Thank you for providing backward compatible approach where cmocka application can be easily generated "the old way"! Can you please add ! first character mark to the PR title and git commit topics with a quick-fix instructions on how to update on user side after change? :-)

[raiden00pl]

CI fails due to https://github.com/apache/nuttx-apps/actions/runs/20508476636/job/59001537281?pr=3270#step:10:1221

boards/risc-v/qemu-rv/rv-virt/configs/citest/logs/ is result of failed rv-virt/citest/run

more detailed failure report in line 1144:

2025-12-27T13:53:52.3596799Z =================================== FAILURES ===================================
2025-12-27T13:53:52.3597634Z _________________________________ test_cmocka __________________________________
2025-12-27T13:53:52.3598016Z 
2025-12-27T13:53:52.3598307Z p = <utils.common.connectNuttx object at 0x7f124b698550>
2025-12-27T13:53:52.3598701Z 
2025-12-27T13:53:52.3598881Z     @pytest.mark.run(order=1)
2025-12-27T13:53:52.3599747Z     def test_cmocka(p):
2025-12-27T13:53:52.3600122Z         if p.board == "sim":
2025-12-27T13:53:52.3600497Z             os.mkdir("./test")
2025-12-27T13:53:52.3601012Z             ret = p.sendCommand("mount -t hostfs -o fs=./test /data")
2025-12-27T13:53:52.3601532Z         if p.board == "rv-virt":
2025-12-27T13:53:52.3602050Z             ret = p.sendCommand("mount -t vfat /dev/virtblk0 /data")
2025-12-27T13:53:52.3602503Z     
2025-12-27T13:53:52.3602874Z         p.sendCommand(f"echo {cmocka_list_start}")
2025-12-27T13:53:52.3603509Z         p.sendCommand("cmocka --list", "Cmocka Test Completed")
2025-12-27T13:53:52.3604067Z         p.sendCommand(f"echo {cmocka_list_end}")
2025-12-27T13:53:52.3604480Z     
2025-12-27T13:53:52.3604841Z         p.sendCommand(f"echo {cmocka_test_start}")
2025-12-27T13:53:52.3605339Z         ret = p.sendCommand(
2025-12-27T13:53:52.3605856Z             "cmocka --skip drivertest_pm|drivertest_posix_timer|drivertest_block*|"
2025-12-27T13:53:52.3606370Z             "test_playback.*|test_interaction.*|test_stress.*|test_capture.*",
2025-12-27T13:53:52.3607296Z             "Cmocka Test Completed",
2025-12-27T13:53:52.3607771Z             timeout=1200,
2025-12-27T13:53:52.3608109Z         )
2025-12-27T13:53:52.3608481Z         p.sendCommand(f"echo {cmocka_test_end}")
2025-12-27T13:53:52.3608920Z     
2025-12-27T13:53:52.3609174Z         if p.board == "sim":
2025-12-27T13:53:52.3609409Z             os.rmdir("./test")
2025-12-27T13:53:52.3609608Z     
2025-12-27T13:53:52.3609778Z >       assert ret == 0
2025-12-27T13:53:52.3609989Z E       assert -1 == 0
2025-12-27T13:53:52.3610183Z E         +-1
2025-12-27T13:53:52.3610349Z E         -0

EDIT2: CONFIG_TESTING_CMOCKA_PROG=y must be set to pass CI, otherwise there is no cmoka program. Nuttx PR and this PR must be merged at the same time

[linguini1]

Looks like you like it? :-)

Yes! Definitely less configuration options than Unity, but it's worth it for the extra flexibility in performing setup/teardown and running test groups.

One remark - we are changing default behavior therefore we should mark this change as breaking correct? Thank you for providing backward compatible approach where cmocka application can be easily generated "the old way"! Can you please add ! first character mark to the PR title and git commit topics with a quick-fix instructions on how to update on user side after change? :-)

Done!

[cederom]

!testing/cmocka: Compile cmocka as a libary only by default :-P

! should be the first character in the pr and git topic.. we should really close that marking update one day :-P