Nginx Segmentation Fault in libluajit

[matthieumarrast]

In our frontend servers using mfserv v2.2 (2.2.14 and 2.2.12), the nginx worker processes crash and restart continuously with a "core dumped" error.

Log /home/mfserv/log/nginx_error.log:

2026/02/19 08:27:51 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1317456
2026/02/19 08:27:51 [alert] 792677#792677: worker process 1317456 exited on signal 11 (core dumped)
2026/02/19 08:27:51 [notice] 792677#792677: start worker process 1319714
2026/02/19 08:27:51 [notice] 792677#792677: signal 29 (SIGIO) received
2026/02/19 08:27:54 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1317380
2026/02/19 08:27:54 [alert] 792677#792677: worker process 1317380 exited on signal 11 (core dumped)
2026/02/19 08:27:54 [notice] 792677#792677: start worker process 1319738
2026/02/19 08:27:54 [notice] 792677#792677: signal 29 (SIGIO) received
2026/02/19 08:28:21 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1319661
2026/02/19 08:28:21 [alert] 792677#792677: worker process 1319661 exited on signal 11 (core dumped)
2026/02/19 08:28:21 [notice] 792677#792677: start worker process 1319952
2026/02/19 08:28:21 [notice] 792677#792677: signal 29 (SIGIO) received
2026/02/19 08:28:24 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1319703
2026/02/19 08:28:24 [alert] 792677#792677: worker process 1319703 exited on signal 11 (core dumped)
2026/02/19 08:28:24 [notice] 792677#792677: start worker process 1319975
2026/02/19 08:28:24 [notice] 792677#792677: signal 29 (SIGIO) received
2026/02/19 08:30:06 [info] 1319714#1319714: *20756399 client 127.0.0.1 closed keepalive connection
2026/02/19 08:30:51 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1319738
2026/02/19 08:30:51 [alert] 792677#792677: worker process 1319738 exited on signal 11 (core dumped)
2026/02/19 08:30:51 [notice] 792677#792677: start worker process 1321645
2026/02/19 08:30:51 [notice] 792677#792677: signal 29 (SIGIO) received
2026/02/19 08:30:54 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1319952
2026/02/19 08:30:54 [alert] 792677#792677: worker process 1319952 exited on signal 11 (core dumped)
2026/02/19 08:30:54 [notice] 792677#792677: start worker process 1321668
2026/02/19 08:30:54 [notice] 792677#792677: signal 29 (SIGIO) received
2026/02/19 08:31:21 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1319975
2026/02/19 08:31:21 [alert] 792677#792677: worker process 1319975 exited on signal 11 (core dumped)
2026/02/19 08:31:21 [notice] 792677#792677: start worker process 1321887
2026/02/19 08:31:21 [notice] 792677#792677: signal 29 (SIGIO) received
2026/02/19 08:31:24 [notice] 792677#792677: signal 17 (SIGCHLD) received from 1319714
2026/02/19 08:31:24 [alert] 792677#792677: worker process 1319714 exited on signal 11 (core dumped)

Log /var/log/messages showing a segfault with libluajit-5.1.so.2.1.ROLLING:

Feb 19 08:27:51 <server_name> kernel: nginx[1317456]: segfault at 1 ip 00007fa093c106d0 sp 00007ffcf5f792a8 error 4 in libluajit-5.1.so.2.1.ROLLING[7fa093c00000+86000] likely on CPU 0 (core 0, socket 0)
Feb 19 08:27:51 <server_name> kernel: Code: c7 0e 29 f8 44 89 c6 89 c2 31 c6 c1 c2 0b 29 d6 31 f1 c1 ce 07 89 ca 29 f2 31 d0 c1 c2 10 29 d0 c3 66 0f 1f 84 00 00 00 00 00 <0f> b6 0c 0e 44 0f b6 06 89 ca 32 54 06 ff 0f b6 c2 89 ca 31 f8 c1
Feb 19 08:27:51 <server_name> systemd[1]: Started Process Core Dump (PID 1319708/UID 0).
Feb 19 08:27:51 <server_name> systemd-coredump[1319713]: Resource limits disable core dumping for process 1317456 (nginx).
Feb 19 08:27:51 <server_name> systemd-coredump[1319713]: Process 1317456 (nginx) of user 1001 dumped core.
Feb 19 08:27:51 <server_name> systemd[1]: systemd-coredump@12238-1319708-0.service: Deactivated successfully.

We are using Rocky Linux v9.7 and v9.6.

I don't know the cause of this error. I don't have a clue for now. Not able to reproduce it with the metwork docker images.

[thebaptiste]

I suppose mfext 2.2 has been updated with nginx/openresty 1.27.1.2 on these servers ?

You should try to "explore" the coredump :

• set ulimit -c to unlimited
• explore the coredump with gdb or ddd or some debugger

Do you have other examples (with or without problem) on :

• rockylinux 8 ?
• rockylinux 10 ?
• metwork 2.3 ?

[matthieumarrast]

Yes mfext has been updated with nginx/openresty 1.27.1.2 ! for security reason, in most of our servers.
No core file seems to be produced... maybe because in the mfserv environment "ulimit -c " returns 0.
We'll try to set it to unlimited.

So I check many of my servers:

• I have servers in metwork-mfserv 2.3.7 / rocky 9.7 without this error.
• I have servers with metwork-mfserv 2.2.13 / rocky 9.4 without this error.
• I found servers with metwork-mfserv 2.2.13 / rocky 9.7 with this error, but occasionnaly
• I have servers in metwork-mfserv 2.1.5 / rocky 9.4 with this error, also continuously ! :-o
• I have servers in metwork-mfserv 2.1.5 / rocky 8.10 with this error, also continuously ! :-o
• I have servers with metwork-mfserv 2.1.5 / rocky 8.10 without this error....
  ... very strange.

[thebaptiste]

The mfserv release has probably no importance, it's mfext release that matters.

It's important to understand if the problem is linked to the openresty upgrade or not.

For example, do the servers with mfserv 2.2.13 / rocky 9.4 without the error have openresty 1.27.1.2 ?

The mfext releases with openresty 1.27.1.2 are :

• mfext 2.0 >= 2.0.14
• mfext 2.1 >= 2.1.13
• mfext 2.2 >= 2.2.18
• mfext 2.3 >= 2.3.9

[thebaptiste]

It's strange that the problem can't be reproduced in Metwork docker images
Unless I'm mistaken (verify if you make tests), the images with openresty 1.27.1.2 are :

• docker/mfserv-rocky8/9/10 >= 2.3.7 (with mfext >= 2.3.10)
• docker/mfserv-rocky8/9/10 >= 2.2.12 (with mfext >= 2.2.18)
• docker/mfserv-rocky8/9/10 >= 2.1.6 (with mfext >= 2.1.16)

[matthieumarrast]

Yes the servers with mfserv 2.2.13 / rocky 9.4 without the error have openresty 1.27.1.2 (mfext 2.2.20).

[thebaptiste]

So it seems that the problem :

• is not directly (not only at least) linked to openresty 1.27.1.2
• is not directly linked to Metwork 2.2 + rocky 9 (or maybe with rocky 9 > 9.4 ?)

[thebaptiste]

What I would suggest to try :

• try to reproduce in a docker image (same OS, same Metwork releases). Does the coredump occurs at mfserv start or later ? If you succeed to reproduce, I could help...
• compare the installed rpm (the system ones) between a machine with the problem and a Metwork image without it (if possible with the same OS and the same mfext release)
• activate the coredumps and explore one (but careful, with a coredump every 3 seconds, the file system will be soon full)

[thebaptiste]

I do not exclude something to do with openssl

• before Metwork 2.3, nginx (openresty binary) links with openssl 1.1.1k (available on rocky 8, only available as a compatibility package on rocky 9 and not available at all on rocky 10, the release available on rocky 9 and rocky 10 is openssl 3.5.1); mfext 2.2 embeds a copy of the openssl 1.1.1k libraries to allow an install on rocky 9 and rocky 10; libluajit-5.1.so.2.1 doesn't link with these libraries

• since Metwork 2.3 (even before the upgrade of openresty), openssl is built in mfext (openssl 3.4.4 in Metwork 2.3, 3.5.5 in master branch, it was necessary for the build of recent gdal), the build of openresty has been modified to use the corresponding libraries, the openresty binary and libluajit-5.1.so.2.1 are linking with them.

• openssl 1.1.1k is an old package, it wouldn't surprise me if there wasn't a full compatibility with recent package as openresty 1.27.1.2 (it's just a guess, maybe I'm wrong)

That being said, it doesn't provide a solution... except maybe to encourage migration to Metwork 2.3 (if it's confirmed that there is no problem) !

[matthieumarrast]

We activated the core dump file then we get core.nginx.1001.427c23c2a80c45919fc8bdfcb51f1f83.160195.1771600976000000.zst
after extraction, gdb shows:

(gdb) thread apply all bt

Thread 1 (Thread 0x7fcc84a971c0 (LWP 160195)):
#0  hash_sparse_def (seed=<optimized out>, str=0x1 <error: Cannot access memory at address 0x1>, len=1) at lj_str.c:111
#1  0x00007fcc84810ba3 in lj_str_new (L=0x7fcc84768810, str=0x1 <error: Cannot access memory at address 0x1>, lenx=1) at lj_str.c:355
#2  0x00007fcc8481e22e in lua_pushlstring (L=0x7fcc84768810, str=<optimized out>, len=<optimized out>) at lj_api.c:658
#3  0x000000000050c703 in ngx_http_lua_get_peer ()
#4  0x000000000050cf29 in ngx_http_lua_upstream_get_primary_peers ()
#5  0x00007fcc8480abbb in lj_BC_FUNCC () from /opt/metwork-mfext-2.2/opt/openresty/luajit/lib/libluajit-5.1.so.2
#6  0x00000000004e3612 in ngx_http_lua_run_thread ()
#7  0x00000000004e6d92 in ngx_http_lua_content_by_chunk ()
#8  0x00000000004e691e in ngx_http_lua_content_handler ()
#9  0x00000000004625af in ngx_http_core_content_phase ()
#10 0x000000000045d1b5 in ngx_http_core_run_phases ()
#11 0x0000000000467533 in ngx_http_process_request ()
#12 0x000000000046802b in ngx_http_process_request_headers ()
#13 0x00000000004683c6 in ngx_http_process_request_line ()
#14 0x000000000044ea5e in ngx_epoll_process_events ()
#15 0x00000000004458a3 in ngx_process_events_and_timers ()
#16 0x000000000044ce5a in ngx_worker_process_cycle ()
#17 0x000000000044b8b4 in ngx_spawn_process ()
#18 0x000000000044de31 in ngx_master_process_cycle ()
#19 0x0000000000424dc8 in main ()

[matthieumarrast]

I notice upstream.get_primary_peers(u)
is called on our lua script:

• socket_updown.lua - https://github.com/metwork-framework/mfserv/blob/master/config/socket_updown.lua
• upstream_status.lua - https://github.com/metwork-framework/mfserv/blob/master/config/upstream_status.lua

when I look how they are executed in the signal_wrapper (https://github.com/metwork-framework/mfserv/blob/master/adm/signal_wrapper.py), each time I execute the below request:

• curl http://127.0.0.1:18868/__upstream_status
• curl http://127.0.0.1:18868/__socket_down/L2hvbWUvbWZzZXJ2L3Zhci9hcHBfbWZzZXJ2X2Flcm93ZWJfd21zX21haW5fMy5zb2NrZXQ=
  a nginx worker process is killed with code 11 SIGSEGV (segmentation fault)

[matthieumarrast]

On a server where we don't face the error (mfserv v2.1.5, with openresty v1.27.1.2) query "__upstream_status" returns a json:

[mfserv@<server>:~] $ curl http://127.0.0.1:18868/__upstream_status
{"48281ab21ea37f8a4ae13fdd5364112a":[{"id":0,"weight":1,"current_weight":0,"effective_weight":1,"fails":0,
"conns":0,"fail_timeout":10,"checked":1771861204,"max_fails":0,"name":"unix:\/home\/mfserv\/var\/
app_<plugin1>_main_1.socket"},{"id":1,"weight":0,"current_weight":0,"effective_weight":0,"fails":53,
"conns":1.4048432828109e+14,"fail_timeout":0,"checked":53,"accessed":1.4048432828122e+14,
"down":true,"max_fails":1.4048432829254e+14,"name":""}],"748d07169cc9355dc13ffa06b38a072e":
[{"id":0,"weight":1,"current_weight":0,"effective_weight":1,"fails":0,"conns":0,"fail_timeout":10,
"checked":1769003403,"max_fails":0,"name":"unix:\/home\/mfserv\/var\/app_<plugin2>_main_1.socket"},
{"id":1,"weight":0,"current_weight":0,"effective_weight":0,"fails":51,"conns":1.4048432828045e+14,
"fail_timeout":0,"checked":51,"accessed":1.4048432828058e+14,"max_fails":1.4048432829242e+14,"name":""}]}

On the server with the crash, we have:

$ curl http://127.0.0.1:18868/__upstream_status
curl: (52) Empty reply from server

[matthieumarrast]

way to reproduce:

docker run -it metwork/mfserv-rocky9:2.3.9 bash
su - mfserv
bootstrap_plugin.py create foo
cd foo && make develop
mfserv.start
curl http://127.0.0.1:18868/__upstream_status
# curl: (52) Empty reply from server
vim ~/log/nginx_error.log
# 2026/02/23 16:54:15 [alert] 1066#1066: worker process 1155 exited on signal 11 (core dumped)

[thebaptiste]

Ok, I can reproduce in any mfserv docker image (Metwork 2.1, 2.2, 2.3...) with nginx 1.27.1.2

I saw this post that seems quite similar to our problem : https://groups.google.com/g/openresty-en/c/ALTakDRCNqw
So maybe nginx is not yet ready to get upstreams when these lua scripts are first called and something that was allowed by 1.15.8.4 is not allowed with 1.27.1.2 ?

I tried to add in upstream_status.lua, as in init_worker_by_lua.lua, a test before the line local us = get_upstreams() :

    local check
    check = function(premature)
        if not premature then
            local us = get_upstreams()
            (...)
       end
    end

I don't know exactly what I'm doing (and maybe it will bring other problems), but it seems to remove the coredump for this lua file
The same fix would maybe have the same impact on socket_updown.lua