Cover more postfix logs by afeefghannam89 · Pull Request #49 · NETWAYS/postfix-logstash-pipeline

afeefghannam89 · 2026-04-23T16:35:08Z

filter-20-msgid: Include local component in queue ID extraction — local delivery logs carry the same prefix format
filter-50-bounce: Add separate pattern for postmaster notification copies (postmaster copies:) alongside the existing NDR pattern
filter-50-cleanup: Replace per-stage milter patterns with a single generic pattern covering all stages (CONNECT, HELO, MAIL, RCPT, DATA, END-OF-MESSAGE); add patterns for reject: header and reject: body; fix field mapping for sender/recipient and protocol field name
filter-50-dnsblog: Add pattern for DNSBL DNS lookup failures (warning: dnsblog_query:)
filter-50-lmtp: Split TCP and Unix socket delivery into separate patterns; add numeric status_code field (sent=1, deferred=2, bounced=3) used by the OpenSearch upsert script
filter-50-local: Add status_code field (same mapping as lmtp/smtp)
filter-50-smtp: Add status_code field; mark optional pattern groups for delivery vs. plain error messages
filter-50-qmgr: Add patterns for skipped, status=expired, queue clog warning, corrupted queue file, and empty-sender bounce detection
filter-50-smtpd: Add patterns for NOQUEUE rejections, reverse DNS mismatches, and SASL authentication failures
filter-50-smtps: Fix overly broad pre-check (/from/ → /connect from/); add patterns for lost connection, SSL accept errors, anonymous TLS, and TLS library errors; add connectdetail kv parsing
filter-50-submission: Convert independent if blocks to else if chain; add patterns for lost connection and TLS library errors
filter-70-client / filter-70-server: Fix IPv6 detection — extend regex from /^\d/ to /^\d|:/ to correctly handle all IPv6 forms including ::1
filter-90-geoip: Use ECS-compliant GeoIP targets ([client][geo], [server][geo]) instead of shared geoip field
All filters: Gate [ecs][version] behind grokked tag — unparsed events no longer receive the ECS version field; bump to ECS 9.3.0

GitHub workflow:

Support starting pipeline manuell. Stop the old pipeline run if a new pash occur.
Replace deprecated apt-key add with gpg --dearmor and [signed-by=] apt source entry, required on Ubuntu 22.04+
Switch Elastic apt repository from 7.x to 8.x

widhalmt

I could not test the rules with actual log messages. So I had to rely on my memory and it's possible that I missed something. I tried my best to review the rules but it's a huge PR.

I love how you use [@metadata] and parse it in the end of the ruleset. And how you only add the ECS field when a message is marked grokked

widhalmt · 2026-05-21T14:49:36Z

+    if [message] =~ /^reject: header/ {
+      grok {
+        # header rejected by header_checks map — captures sender, recipient, protocol and reject reason
+        match => ["message", "reject: header %{DATA:[postfix][header]} from %{HOSTNAME:[client][address]}\[%{IP:[client][ip]}\]; from=<(%{DATA:[source][user][email]})?> to=<%{DATA:[destination][user][email]}> proto=%{WORD:[postfix][proto]} helo=%{DATA:[postfix][helo]}: %{GREEDYDATA:[postfix][detail]}"]
+        tag_on_failure => ["_grokparsefailure", "postfix_cleanup_reject_header_failed"]
+        id => "postfix_cleanup_reject_header"
+        add_field => {
+          "[postfix][eventtype]" => "cleanup_reject_header"
        }
+        add_tag => "grokked"
      }
    }

-    if [message] =~ /^reject:/ {
-      if [message] =~ /^reject: header/ {
-        grok {
-          match => ["message","reject: header %{DATA:[postfix][header]} from %{HOSTNAME:[client][address]}\[%{IP:[client][ip]}\]; from=<(%{DATA:[destination][user][email]})?> to=<%{DATA:[destination][user][email]}> proto=%{WORD:[postfx][proto]} helo=%{DATA:[postfix][helo]}: %{GREEDYDATA:[postfix][detail]}"]
-          tag_on_failure => ["_grokparsefailure","postfix_cleanup_reject_header"]
-          id => "postfix_cleanup_reject_header"
-          add_field => {
-            "[postfix][eventtype]" => "cleanup_reject_header"
-          }
-          add_tag => "grokked"
+    if [message] =~ /^reject: body/ {
+      # body content rejected by body_checks map or a milter after the DATA phase
+      grok {
+        match => ["message", "reject: body %{DATA:[postfix][body_text]} from %{HOSTNAME:[client][address]}\[%{IP:[client][ip]}\]; from=<(%{DATA:[source][user][email]})?> to=<%{DATA:[destination][user][email]}> proto=%{WORD:[postfix][proto]} helo=%{DATA:[postfix][helo]}: %{GREEDYDATA:[postfix][detail]}"]
+        tag_on_failure => ["_grokparsefailure", "postfix_cleanup_reject_body_failed"]
+        id => "postfix_cleanup_reject_body"
+        add_field => {
+          "[postfix][eventtype]" => "cleanup_reject_body"
        }


Suggested change

if [message] =~ /^reject: header/ {

grok {

# header rejected by header_checks map — captures sender, recipient, protocol and reject reason

match => ["message", "reject: header %{DATA:[postfix][header]} from %{HOSTNAME:[client][address]}\[%{IP:[client][ip]}\]; from=<(%{DATA:[source][user][email]})?> to=<%{DATA:[destination][user][email]}> proto=%{WORD:[postfix][proto]} helo=%{DATA:[postfix][helo]}: %{GREEDYDATA:[postfix][detail]}"]

tag_on_failure => ["_grokparsefailure", "postfix_cleanup_reject_header_failed"]

id => "postfix_cleanup_reject_header"

add_field => {

"[postfix][eventtype]" => "cleanup_reject_header"

}

add_tag => "grokked"

}

}

if [message] =~ /^reject:/ {

if [message] =~ /^reject: header/ {

grok {

match => ["message","reject: header %{DATA:[postfix][header]} from %{HOSTNAME:[client][address]}\[%{IP:[client][ip]}\]; from=<(%{DATA:[destination][user][email]})?> to=<%{DATA:[destination][user][email]}> proto=%{WORD:[postfx][proto]} helo=%{DATA:[postfix][helo]}: %{GREEDYDATA:[postfix][detail]}"]

tag_on_failure => ["_grokparsefailure","postfix_cleanup_reject_header"]

id => "postfix_cleanup_reject_header"

add_field => {

"[postfix][eventtype]" => "cleanup_reject_header"

}

add_tag => "grokked"

if [message] =~ /^reject: body/ {

# body content rejected by body_checks map or a milter after the DATA phase

grok {

match => ["message", "reject: body %{DATA:[postfix][body_text]} from %{HOSTNAME:[client][address]}\[%{IP:[client][ip]}\]; from=<(%{DATA:[source][user][email]})?> to=<%{DATA:[destination][user][email]}> proto=%{WORD:[postfix][proto]} helo=%{DATA:[postfix][helo]}: %{GREEDYDATA:[postfix][detail]}"]

tag_on_failure => ["_grokparsefailure", "postfix_cleanup_reject_body_failed"]

id => "postfix_cleanup_reject_body"

add_field => {

"[postfix][eventtype]" => "cleanup_reject_body"

}

if [message] =~ /^reject: / {

grok {

# header rejected by header_checks map — captures sender, recipient, protocol and reject reason

match => ["message", "reject: (header %{DATA:[postfix][header]}|body %{DATA:[postfix][body_text]}) from %{HOSTNAME:[client][address]}\[%{IP:[client][ip]}\]; from=<(%{DATA:[source][user][email]})?> to=<%{DATA:[destination][user][email]}> proto=%{WORD:[postfix][proto]} helo=%{DATA:[postfix][helo]}: %{GREEDYDATA:[postfix][detail]}"]

tag_on_failure => ["_grokparsefailure", "postfix_cleanup_reject_header_failed"]

id => "postfix_cleanup_reject_header"

add_field => {

"[postfix][eventtype]" => "cleanup_reject_header"

}

add_tag => "grokked"

}

}

Wouldn't this work as well? Maybe it's harder to understand and your attempt is still better.

widhalmt · 2026-05-21T14:54:12Z

+      # numeric status_code used by the OpenSearch upsert script to track final delivery outcome
+      # sent=1 wins over deferred=2 wins over bounced=3 (higher code = worse outcome)
+      if [postfix][status] == "sent" {
+        mutate { add_field => { "[postfix][status_code]" => "1" } }
+      } else if [postfix][status] == "deferred" {
+        mutate { add_field => { "[postfix][status_code]" => "2" } }
+      } else if [postfix][status] == "bounced" {
+        mutate { add_field => { "[postfix][status_code]" => "3" } }
+      }
+      if [postfix][status_code] {
+        mutate { convert => { "[postfix][status_code]" => "integer" } }
+      }


Is there some sort of "official return code" or status code for Postfix actions? While I understand, why we need this for a specific project, we need to keep the project as generic as possible.

widhalmt · 2026-05-21T14:57:33Z

+      # numeric status_code used by the OpenSearch upsert script to track final delivery outcome
+      # sent=1 wins over deferred=2 wins over bounced=3 (higher code = worse outcome)
+      if [postfix][status] == "sent" {
+        mutate { add_field => { "[postfix][status_code]" => "1" } }
+      } else if [postfix][status] == "deferred" {
+        mutate { add_field => { "[postfix][status_code]" => "2" } }
+      } else if [postfix][status] == "bounced" {
+        mutate { add_field => { "[postfix][status_code]" => "3" } }
+      }
+      if [postfix][status_code] {
+        mutate { convert => { "[postfix][status_code]" => "integer" } }


See comment above

widhalmt · 2026-05-21T14:58:57Z

+      # numeric status_code used by the OpenSearch upsert script to track final delivery outcome
+      # sent=1 wins over deferred=2 wins over bounced=3 (higher code = worse outcome)
+      if [postfix][status] == "sent" {
+        mutate { add_field => { "[postfix][status_code]" => "1" } }
+      } else if [postfix][status] == "deferred" {
+        mutate { add_field => { "[postfix][status_code]" => "2" } }
+      } else if [postfix][status] == "bounced" {
+        mutate { add_field => { "[postfix][status_code]" => "3" } }
+      }
+      if [postfix][status_code] {
+        mutate { convert => { "[postfix][status_code]" => "integer" } }
+      }


See comment above

afeefghannam89 added 6 commits April 23, 2026 17:43

Cover more log lines or cases

99147b0

Cover more cases with bounce cleanup and dnsblog

9fdac9d

Update more cases with lmtp

04c2bde

Update more filters

3c4cf58

Update workflow and some use cases with postfix component

1ced7fc

Fix branch name

8b00b6c

afeefghannam89 requested a review from widhalmt April 24, 2026 07:35

afeefghannam89 added 7 commits April 29, 2026 12:21

Add status_code to use in dashboard

de5bff9

Fix the convert from str to int

1505f45

Change the name temorary

611c6ec

Restore the old name

9e038a5

Filter the message id from local component

91149c9

Add comment to the use cases

cc84dfd

modify some commentar

c1c0958

widhalmt approved these changes May 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cover more postfix logs#49

Cover more postfix logs#49
afeefghannam89 wants to merge 13 commits into
masterfrom
update

afeefghannam89 commented Apr 23, 2026 •

edited

Loading

Uh oh!

widhalmt left a comment

Uh oh!

widhalmt May 21, 2026

Uh oh!

widhalmt May 21, 2026

Uh oh!

widhalmt May 21, 2026

Uh oh!

widhalmt May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

afeefghannam89 commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

widhalmt left a comment

Choose a reason for hiding this comment

Uh oh!

widhalmt May 21, 2026

Choose a reason for hiding this comment

Uh oh!

widhalmt May 21, 2026

Choose a reason for hiding this comment

Uh oh!

widhalmt May 21, 2026

Choose a reason for hiding this comment

Uh oh!

widhalmt May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

afeefghannam89 commented Apr 23, 2026 •

edited

Loading