integrating ntopNG with n2disk for hosts wit packet ingestion @40Gbps throughput

[gsdeol2025]

How to get ntopNG to integrate with n2disk on a host with 40G NICs (mellanox connectX-6 adapter) for packet capturing. Host has 72 hyperthreaded cores and 1TB of RAM, 35TB SSD

n2disk is running on the host

PF_RING_ZC is also installed to allow loss less packet ingestion @ 40Gbps throughput

ntopNG and cento are running on the same host as well

I have been told that both cento and n2disk can run in parallel on this host and receive a copy of the packets without dramas.

I would like to know how to extract packets from a conversation (TCP flow) of interest using ntopNG UI instead of relying on a tool like npcapextract from perform the packet extraction from the pcap dump.

Thanks

[cardigliano]

When running at high speed, the n2disk service should be configured manually. Please check:
https://www.ntop.org/guides/n2disk/tuning/examples.html

After configuring the service, please read the "External Traffic Recording Providers" section for n2disk integration in ntopng (to run extractions):
https://www.ntop.org/guides/ntopng/using_with_other_tools/n2disk.html

I suggest to read also this blog post to run n2disk with cento on the same adapter:
https://www.ntop.org/introducing-pf_ring-zc-support-for-mellanox-adapters/

[gsdeol2025]

Thanks @cardigliano. we have managed to get external recorder added to ntopNG. It does seem to start but fails in the below manner, tried deleting all the pcap related folders and doing a fresh restart.

it's warning about invalid packet timestamps constantly and then fails at the end with the following error

01/Oct/2025 00:20:13 [n2disk.c:2386] WARNING: Timestamp of last packet not available
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: Index compression error, digest length is 0
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:3555] ERROR: [mlx:mlx5_0] Unable to create timeline links for /mnt/data1/storage/n2disk/12/pcap/1759277969.164478/1759277969.188983.pcap [error=17/File exists]

have tried it multiple times now and getting the same error/behaviour.
As you are aware we have a mellanox ConnectX-6 Dx adapter receiving these packets and OS is ubuntu 22.04.

Is it a resolution issue or something else we need to tune?

Thanks,
Gurbir

Logs for the n2disk run below

Oct 01 00:19:27 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:27 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277921 (capture started at 1759277949)
Oct 01 00:19:29 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:29 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277923 (capture started at 1759277949)
Oct 01 00:19:31 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:31 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277925 (capture started at 1759277949)
Oct 01 00:19:33 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:33 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277927 (capture started at 1759277949)
Oct 01 00:19:35 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:35 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277929 (capture started at 1759277949)
Oct 01 00:19:37 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:37 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277931 (capture started at 1759277949)
Oct 01 00:19:39 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:39 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277933 (capture started at 1759277949)
Oct 01 00:19:41 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:41 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277935 (capture started at 1759277949)
Oct 01 00:19:43 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:43 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277937 (capture started at 1759277949)
Oct 01 00:19:45 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:45 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277939 (capture started at 1759277949)
Oct 01 00:19:48 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:48 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277941 (capture started at 1759277949)
Oct 01 00:19:50 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:50 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277943 (capture started at 1759277949)
Oct 01 00:19:52 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:19:52 [n2disk.c:3481] WARNING: [mlx:mlx5_0] [writer][#0] Invalid packet timestamp 1759277946 (capture started at 1759277949)
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:2386] WARNING: Timestamp of last packet not available
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: Index compression error, digest length is 0
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:3555] ERROR: [mlx:mlx5_0] Unable to create timeline links for /mnt/data1/storage/n2disk/12/pcap/1759277969.164478/1759277969.188983.pcap [error=17/File exists]
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:1336] Caught termination signal -1...
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:1155] [mlx:mlx5_0] [PF_RING] Total stats: 16626479 pkts rcvd/16626479 pkts filtered/0 pkts dropped [0.0%]
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:1169] [n2disk][mlx:mlx5_0] Processed 16626479 pkts
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:1174] [n2disk][mlx:mlx5_0] Capture Duration: 0:00:01:03:294
Oct 01 00:20:13 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:13 [n2disk.c:1176] [n2disk][mlx:mlx5_0] Average Capture Throughput: 1.91 Gbit / 0.26 Mpps
Oct 01 00:20:14 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:14 [n2disk.c:5308] [mlx:mlx5_0] Packet capture thread terminated
Oct 01 00:20:14 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:14 [n2disk.c:7543] [mlx:mlx5_0] Reader thread terminated
Oct 01 00:20:14 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:14 [n2disk.c:7557] [mlx:mlx5_0] Writer thread #0 terminated
Oct 01 00:20:14 au2-nprobe01 n2disk[89066]: 01/Oct/2025 00:20:14 [n2disk.c:7578] [mlx:mlx5_0] Compression thread #0 terminated
Oct 01 00:20:14 au2-nprobe01 systemd[1]: n2disk@mlx5_0.service: Main process exited, code=exited, status=1/FAILURE

[cardigliano]

Please make sure the adapter clock is synchronized with the system clock, as n2disk expects that they are aligned at least the the second level, you can synchronize it with a command like this:
sudo phc2sys -s CLOCK_REALTIME -c -O 0 -m
As of the other failures, please provide the configuration file for n2disk, and double check permissions on disk.
Thank you.

[gsdeol2025]

Thanks @cardigliano
there was a time drift of about 30seconds between the system and adapter clocks which I have fixed as per your suggestion with the below command and options

"sudo phc2sys -s CLOCK_REALTIME -c /dev/ptp1 -O 0 -m"

dante@au2-nprobe01:~$ sudo phc_ctl /dev/ptp1 get
phc_ctl[1308746.744]: clock time is 1759309701.022023669 or Wed Oct 1 09:08:21 2025

dante@au2-nprobe01:$ date
Wed Oct 1 09:08:22 AM UTC 2025
dante@au2-nprobe01:$

Copied below is the n2disk configuration file used to run the tool
dante@au2-nprobe01:$ sudo cat /etc/n2disk/n2disk-mlx5_0.conf
--interface=mlx:mlx5_0
--dump-directory=/mnt/data1/storage/n2disk/12/pcap
--index
--timeline-dir=/mnt/data1/storage/n2disk/12/timeline
--buffer-len= 8192 #128
--max-file-len= 1024 #64
--max-file-duration=60
--disk-limit=3379200
--snaplen=128
--writer-cpu-affinity=8 #0
--reader-cpu-affinity=9 #1
--compressor-cpu-affinity=2
--index-on-compressor-threads
-u=ntopng
--capture-direction=0
--index-tunnel-content=1
--daemon
--nanoseconds
-P=/var/run/n2disk-mlx5_0.pid
dante@au2-nprobe01:$

Thanks,
Gurbir

[gsdeol2025]

One more thing we have noticed, when ntopNG is configured for traffic recording to use the external n2disk@mlx_5.service provider daemon for packet dumps, the throughput and traffic charts start to show some wild stats. currently traffic is being ingested on a 40Gbps link but the actual aggregate throughput never exceeds 20Gbps.

When traffic recording enabled with external recording Provider configuration, the throughput charts start showing peaks of 240Gbps or a very high number which makes no sense at all.

And start to see pop up messages about too many flows being seen which again isn't true.

is there a bug here or somehow the traffic recording is interfering with ntopNG stats generation on the flow records seen via zmq on tcp://127.0.0.1:5556 fed by cento?

thanks,
Gurbir

[cardigliano]

This does not seem to be a known issue. Could you also provide:

• ntopng configuration file (ntopng.conf)
• cat /proc/net/pf_ring/dev/IFNAME/info (where IFNAME is the mlx5_0 linux interface name)

[gsdeol2025]

Hi @cardigliano

ntopng config

#--ndpi-protocols=/etc/ntopng/custom_protocols.txt
-i=tcp://127.0.0.1:5556c
#-i=mlx:mlx5_0
--http-port=3000
--data-dir=/mnt/data1/storage
--pcap-dir=/mnt/data1/storage/n2disk/12/pcap
-F "clickhouse;127.0.0.1@9004;ntopng;default;Admin@123"

-X=524288
dante@au2-nprobe01:~$

ens2f0np0 is the mellanox ConnectX-6 DX adapter receiving the traffic.

dante@au2-nprobe01:~$ cat /proc/net/pf_ring/dev/ens2f0np0/info
Name: ens2f0np0
Index: 10
Address: 08:C0:EB:7F:8B:10
Polling Mode: NAPI
Promisc: Disabled
Type: Ethernet
Family: Standard NIC

Bound Sockets: 0

TX Queues: 8
RX Queues: 8
dante@au2-nprobe01:~$

dante@au2-nprobe01:$ cat /proc/net/pf_ring/10037-none.11
Bound Device(s) : mlx5_0
Active : 0
Breed : Standard
Appl. Name : cento-mlx:mlx5_0-40
Socket Mode : RX only
Capture Direction : RX+TX
Sampling Rate : 1
Filtering Sampling Rate: 0
IP Defragment : No
BPF Filtering : Disabled
Sw Filt Hash Rules : 0
Sw Filt WC Rules : 0
Sw Filt Hash Match : 0
Sw Filt Hash Miss : 0
Sw Filt Hash Filtered : 0
Hw Filt Rules : 0
Poll Pkt Watermark : 128
Num Poll Calls : 0
Poll Watermark Timeout : 0
dante@au2-nprobe01:$

[gsdeol2025]

wanted to clarify something, since we have 8 RSS queues for receiving the flows, are the flows (two-way TCP flow) being load balanced to a single Rx queue or does each side of the flow (5 tuple used for hash calculation) get processed by a separate queue possibly?

thanks

[cardigliano]

Please add --disable-rss to the n2disk configuration file.
Also provide the cento configuration file (I forgot to ask this before sorry).
As of flow directions, RSS should be symmetric on your adapter, thus they get balanced to the same queue.

[gsdeol2025]

Thanks @cardigliano. have fixed the n2disk run.

Also, thanks for confirming that flow balancing on RSS is symmetric, which ensures both sides of the L4 flow (A->B and B->A) land on the same queue.

cento is running from command line with the following options:
sudo cento -i mlx:mlx5_0@[0-7] --processing-cores 0,1,2,3,4,5,6,7 --zmq tcp://127.0.0.1:5556

I am still seeing the spikes in traffic charts which are not expected. snippets attached.

thanks,
Gurbir

[cardigliano]

@gsdeol2025 if you are using synthetic traffic (e.g. a traffic generator always sending the same flows) this may be expected as it seems traffic spikes are occurring at periodic intervals, which may be related to the periodic flow export. Thus this may be expected. I suggest to try with real traffic in that case.

[gsdeol2025]

Hi @cardigliano, it's real network traffic that's being collected by networks taps and aggregated by an intermediate Packet broker switch and fed upto this host running pfring/cento/n2disk/ntopNG on a single 40G port. In reality traffic doesn't exceed 20Gbps from what I have seen.

The setup works fine if we don't configure an external traffic recorder ( an n2disk@ml5.service instance) in ntopNG. Also, if we introduce local clickhouse DB for record storage from ntopNG the same problem appears with some wild numbers in traffic charts.

Thanks.

[cardigliano]

@gsdeol2025 could you try zooming out to see whether the average matches the expected throughput? As I said it looks to me that you have spikes in correspondence with periodic flow updates sent by cento. With real traffic this is a bit strange as flows should be more distributed, and should improve after a while, however I suggest to leave it running for some time and check the average to figure out if this is the case.

[gsdeol2025]

Hi @cardigliano here's the issue related to pcap from historical flows.

The option to download the flow is available but when i click on Apply it says "invalid value" under the filter. the filter is populated automatically but somehow it's not accepted and errors out.

However if i remove the filter, I am able to download the packets for that duration. Not sure if someone else has come across the same issue before.

Cheers,
Gurbir