Optimize profiler #1743

AlyaGomaa · 2025-11-28T14:35:22Z

use processes instead of threads for true parallelism of profiler workers
increase workers as throughput increases by checking the input.py and profiler.py flows/sec imbalance

…of in each worker

…ughput

…econd of input and profiler

- Add comprehensive dataset documentation with two workflows: - Summarization workflow (event summaries and behavior analysis) - Risk analysis workflow (root cause and risk assessment) - Add detailed workflow implementation guides for both pipelines - Add LLM evaluation framework and results documentation - Add DAG parser technical reference documentation - Organize all documentation under new "Datasets & LLM Training" section in Immune.md - Fix broken cross-references between documentation files - Preserve legacy documentation for historical reference

- Create 'Security & Network Configuration' section for ARP and traffic routing docs - These were incorrectly nested under 'Datasets & LLM Training' section

… before Datasets - Security & Network Configuration section now appears before Datasets & LLM Training - This provides better logical flow in the documentation navigation

Immune dataset documentation

…file in redis since it's no longer used in slips

…ants

…, they don't do diff things

…shmap to be able to access their dports later by the portscan modules

…le to import it anywhere (in the db and in the modules that may need it)

…profile and tw

…ortscan detectors

…is hmaps

…rator of profiles and tws

…ections to use it by the db and the modules

…ble to retreive the needed info in O(1)

…o(1)

…redis channel (new_zeek_fields_line) the recognized indices so other profiler workers know about it

…' line processors from the db instead of the channel because msgs published in that channel won't be accessible by the new worker

…attack starttime when setting an evidence

…r_module to slips_utils to be used by all modules

AlyaGomaa and others added 19 commits November 26, 2025 13:22

Make the 3 profiler threads 3 Processes for true parallelism

0fb9b3c

Add a separate class for the profiler workers

0d3c717

profiler: handle starting the workers

0601da9

profiler: get handler class only once in the main profiler , instead …

e7744e5

…of in each worker

profiler: share the localnet_cache accross all workers

25c74e4

pass bloom filters to profiler workers

0496b82

db: add functions to get and set modules' flows processed per second

b50ed62

store the flows processed per second for the pinput and profiler

b9f4d16

start with 3 profiler workers and increase them up to 10 on high thro…

3bc55df

…ughput

Fix printing the used local network once by each worker

0ff8b8a

Fix problem finding last flows count when calculating the flows per s…

79e1fe8

…econd of input and profiler

Fix documentation structure: move security topics to separate section

30f4827

- Create 'Security & Network Configuration' section for ARP and traffic routing docs - These were incorrectly nested under 'Datasets & LLM Training' section

Reorder documentation sections: move Security & Network Configuration…

1f6a0f4

… before Datasets - Security & Network Configuration section now appears before Datasets & LLM Training - This provides better logical flow in the documentation navigation

profiler: fix problem using input_fps as str

dcb8d41

fix storing profiler workers PID in the db without a name

8c27332

update profiler unit tests

e1aa9bf

add profiler_worker unit tests

d734952

Add profiler worker to CI uni-tests

32a7080

github-project-automation bot added this to Slips Nov 28, 2025

github-project-automation bot moved this to Todo in Slips Nov 28, 2025

AlyaGomaa and others added 9 commits November 28, 2025 16:36

Fix off by one when printing the number of max workers

8651e32

add an entry for failover mechanisms docs in immune.md

2acc595

update changelog

0a425ae

Fix stopping the daemon

180962a

Fix problem killing the daemon's children

4e0728b

bump slips version to 1.1.16

3874540

update slips.gif

e6db165

Merge branch 'develop' into immune-dataset

51486da

Merge pull request #1749 from stratosphereips/immune-dataset

6b38c9e

Immune dataset documentation

AlyaGomaa added 30 commits December 18, 2025 21:12

delete the tracking of SrcIPs and DstIPs times contacted of every pro…

da0a86f

…file in redis since it's no longer used in slips

add get_modified_profiles_since to the list of channels, not db const…

a037cf4

…ants

merge the calls to add_ips() and add_port() into one func for clarity…

5bd5189

…, they don't do diff things

optimize add_ips() by using a redis pipe

42bc43d

keep track of ips contacted by a specific profile/tw in a separate ha…

0a61fe8

…shmap to be able to access their dports later by the portscan modules

add a file with the icmp ports that slips uses for detection to be ab…

0c2aff1

…le to import it anywhere (in the db and in the modules that may need it)

db: add a generator to return dstips_with_not_established_flows of a …

28989a3

…profile and tw

db: add a func to get_info_about_not_established_flows used by both p…

415de90

…ortscan detectors

vertical_ps: get needed info for detections in O(1) using the new red…

a35b146

…is hmaps

change th eformat of msgs in tw_modified channel to avoid _ or : sepa…

799f950

…rator of profiles and tws

profiler: check for stop msgs as early as possible

68e7245

add a common func in slips_utils to validate ips used in portscan det…

8868874

…ections to use it by the db and the modules

db: change the keys needed for horizontal portscand etections to be a…

b449e64

…ble to retreive the needed info in O(1)

db: update the horizontal portscan keys to access the needed info in …

1322cd9

…o(1)

horizontal_portscan: use the new db keys

35e33e0

network_discovery: del unused logic

f4a54ab

Make all profiler workers an IModule for easier use of redis channels

a0d645a

When an input processor parses a zeek #field line, it publishes in a …

0f4f2d4

…redis channel (new_zeek_fields_line) the recognized indices so other profiler workers know about it

When a profiler worker is added due to high throuput, make it get its…

ee0e0ba

…' line processors from the db instead of the channel because msgs published in that channel won't be accessible by the new worker

fix problem stopping the aid manager

6bec796

rename flow_attributes_db.py to scan_detections_db.py

17f31c1

test10/conn.log: add horizontal portscan flows for testing

52344c5

store horizontal portscan detections in a zset to be able to get the …

6be0e45

…attack starttime when setting an evidence

Fix latency logging json alerts to alerts.json

b5520c4

Add an extra zset for storing icmp scan attacks starttime

29a17b8

db: add a key to track the starttime of icmp scans

e26ef98

get the needed info for icmp scan detections in O(1)

30403ea

move convert_str_to_state and is_info_needed_by_the_icmp_scan_detecto…

ab88253

…r_module to slips_utils to be used by all modules

test10/ add an icmp scan to 1 host for testing

2ccd7fe

update unit test files

5877cd6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Optimize profiler #1743

Optimize profiler #1743

Uh oh!

AlyaGomaa commented Nov 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Optimize profiler #1743

Are you sure you want to change the base?

Optimize profiler #1743

Uh oh!

Conversation

AlyaGomaa commented Nov 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants