NNTP Client

[awehttam]

NNTP Upstream Client Transport

Draft. This proposal was generated with AI assistance and may not have been reviewed for accuracy.

Overview

This proposal adds NNTP client transport support to BinktermPHP: the system connects outbound to one or more upstream news servers, pulls articles from subscribed newsgroups, and pushes locally-authored messages back upstream.

This is intentionally different from docs/proposals/NNTPServer.md. That document is about BinktermPHP exposing an NNTP service to end-user newsreaders. This document is about BinktermPHP behaving as a network peer / subscriber to a remote news server.

The two proposals still overlap in important translation behavior. Wherever BinktermPHP constructs or parses NNTP article data, the implementation should prefer shared helpers so the upstream client transport and any future NNTP server do not drift on header semantics.

The design should follow the shape of the existing QWK client work:

• one or more configured remote peers
• per-area transport subscriptions
• scheduled polling
• inbound import into echomail
• outbound queueing from local posts
• relay-policy-aware fanout to other transports

NNTP should be treated as another message transport alongside FTN and QWK, not as a special-purpose side subsystem.

Goals

• Connect to remote NNTP servers as a client
• Pull new articles from configured newsgroups into local echoareas
• Push locally-authored echomail posts to upstream NNTP groups
• Reuse the existing echomail storage model and MessageHandler import/post paths
• Integrate with existing scheduler, relay policy, admin daemon, and admin UI patterns
• Preserve cross-transport dedupe and avoid echoing messages straight back to the same NNTP peer

Non-Goals

• Providing a user-facing NNTP server for newsreaders
• Full Usenet transit peering in the first iteration (IHAVE, TAKETHIS, streaming feeds, wildmat feeds)
• Binary newsgroup support
• Moderation control messages, cancel messages, or supersedes in the first iteration
• Becoming a general-purpose INN replacement

Why NNTP Client Transport Fits BinktermPHP

The current architecture already has most of the needed concepts:

• src/Qwk/QwkPoller.php provides a mailbox-style poller pattern
• src/Qwk/QwkInbound.php shows how external messages are imported into echomail
• src/MessageHandler.php already supports external imports with source_msgid, transport metadata, relay policy, and source-exclusion behavior
• src/Echomail/RelayPolicyManager.php already models per-area origin/target transport rules
• src/Binkp/Connection/Scheduler.php already runs scheduled QWK mailbox polls alongside FTN polling

NNTP is therefore not a conceptual stretch. The main difference from QWK is that QWK exchanges packet files, while NNTP is an online session protocol with:

• article identifiers (Message-ID)
• per-group cursors or watermarks
• optional pull by date or article number
• outbound posting via POST

Recommended Model

Use the same two-layer distinction already introduced by QWK:

• Logical network: a row in networks describing the human-facing network identity and posting rules
• NNTP peer: a remote server connection profile with authentication, polling state, and per-group mappings

Recommended additions:

• add NetworkManager::NETWORK_TYPE_NNTP = 3
• add RelayPolicyManager::TRANSPORT_NNTP = 'nntp'
• store remote-server connection details in a dedicated peer table, not in networks

That separation matters because one NNTP server can carry many groups across one logical network, and in future multiple peers could represent the same logical network.

Protocol Strategy

First iteration: reader-mode client

The first implementation should act like a robust authenticated newsreader/poster, not like a transit peer.

Use this subset:

• CAPABILITIES
• AUTHINFO USER / AUTHINFO PASS
• MODE READER when required by the server
• LIST or LIST ACTIVE
• GROUP
• XOVER or OVER
• ARTICLE or BODY
• POST
• DATE optionally, for server clock sanity/debugging

Avoid first-iteration dependence on:

• NEWNEWS as the sole sync primitive
• IHAVE
• CHECK / TAKETHIS
• reader-specific extensions that are not broadly supported

Why reader mode first:

• much easier to test against common public/private servers
• closer to the current QWK poller model
• sufficient for practical pull/push synchronization
• avoids the complexity of article transfer offers and transit semantics

Pull model

For each configured group mapping:

1. Select group with GROUP <name>
2. Read article range metadata
3. Request overview for articles after the last imported article number using OVER / XOVER
4. For each unseen overview row:
   • prefer dedupe by Message-ID
   • fetch full article with ARTICLE <number> or BODY <number> plus HEAD <number>
   • parse headers and body
   • import into local echomail
5. Advance the local watermark only after successful import

Primary cursor should be per-peer, per-group article number. Also store the last seen Message-ID and last seen article date for diagnostics and recovery.

Push model

When a local post belongs to an area with NNTP subscriptions:

1. Queue the message to one or more NNTP peers
2. During poll/post cycle, build an RFC-compliant article
3. Connect to the peer
4. Authenticate
5. Send POST
6. Stream headers + body
7. Mark queued item sent only on successful 240 response

This mirrors the existing qwk_outbound_messages approach.

Newsgroup Mapping

Each local echoareas row needs one or more NNTP subscriptions, each pointing to:

• an NNTP peer
• a remote newsgroup name

This is the NNTP analogue of echo_area_qwk_subscriptions.

Recommended behavior:

• one local area can map to multiple NNTP peers/groups
• a peer/group pair maps to exactly one local area
• auto-create placeholder local areas for unknown inbound groups should be optional and disabled by default

Unlike FTN, the remote group name is already a stable canonical identifier, so no translation is required unless the sysop wants a different local area tag.

Header and Message Mapping

This section should stay aligned with docs/proposals/NNTPServer.md. Even though this proposal is about upstream client transport rather than end-user service, the same article/header translation rules should be reused wherever possible.

Inbound NNTP article -> local echomail

Suggested mapping:

NNTP header	Local field
Message-ID	source_msgid and optionally message_id-adjacent metadata field
From	from_name plus synthetic/stored from_address
Subject	subject
article body	message_text
Date	preserve in kludge/metadata; date_received remains local receipt time
References / In-Reply-To	map to local reply_to_id when parent already imported
Newsgroups	resolved through peer/group mapping

Important point: do not store the upstream Message-ID as the local FTN message_id value used for generated outbound FTN packets. Keep using source_msgid for foreign-origin identity, as QWK already does.

The parsing logic here should be implemented in a reusable article parser that can also support the future NNTP server work where relevant, especially for:

• Message-ID
• References / In-Reply-To
• From
• Date
• charset / Content-Type

Local echomail -> outbound NNTP article

Suggested header synthesis:

• From: derived from BinktermPHP user identity and network posting policy
• Newsgroups: mapped remote group name
• Subject: from local subject
• Date: current UTC formatted per RFC 5322
• Message-ID: generated by BinktermPHP for the NNTP article
• References: derived from the parent if the parent has an NNTP-side identity for that peer/group
• User-Agent: BinktermPHP version string
• Content-Type: text/plain; charset=UTF-8

The first version should post plain text only. If a local area allows Markdown/media, the NNTP transport should flatten the outbound body to readable plaintext, similar to how other plain-message transports need normalization.

The builder used here should intentionally share the same rules described in docs/proposals/NNTPServer.md:

• Message-ID construction should use the same hostname derivation and formatting strategy as the NNTP server proposal rather than inventing a second format
• From: synthesis should use the same FTN-address-aware email-style representation when the source identity is an FTN-originated message
• References: construction should follow the same single-parent vs reconstructed-chain behavior
• Content-Type and charset declarations should stay consistent across client and server paths
• any synthetic X-FTN-* headers, if emitted for outbound gating/debugging, should use the same naming and semantics

In practice this argues for shared helpers such as:

• src/Nntp/NntpArticleBuilder.php
• src/Nntp/NntpArticleParser.php
• a shared Message-ID helper rather than separate client/server implementations

Dedupe and Loop Prevention

NNTP needs the same loop prevention principle used by QWK gating: if a message came from peer X, do not immediately send the same message back to peer X.

Recommended metadata additions to echomail:

• nntp_peer_id
• nntp_group_name
• nntp_article_number
• nntp_message_id

Recommended semantics:

• for imported NNTP messages, nntp_message_id stores the exact upstream Message-ID
• source_msgid also stores that same upstream ID as the cross-transport dedupe key
• outbound queueing skips the originating nntp_peer_id
• import path checks duplicate by (nntp_peer_id, nntp_group_name, nntp_article_number) and also by source_msgid

This gives:

• transport-local duplicate detection
• cross-transport identity for gating/relay
• a stable reply-chain anchor

Database Changes

nntp_peers

One row per remote news server.

Column	Type	Notes
id	SERIAL PK
name	VARCHAR(100)	Display label
host	VARCHAR(255)
port	INTEGER	default 119 or 563
use_tls	BOOLEAN	implicit TLS
require_starttls	BOOLEAN	for port 119 upgrade
username	VARCHAR(255) NULL
password	TEXT NULL	encrypted via SysK
reader_mode	BOOLEAN	send MODE READER after auth when needed
poll_schedule	VARCHAR(100)	cron expression
enabled	BOOLEAN
auto_create_unknown_groups	BOOLEAN	default false
last_polled_at	TIMESTAMPTZ
last_error	TEXT NULL
created_at	TIMESTAMPTZ

password should reuse the same at-rest encryption approach already used by src/SysK.php and src/Qwk/QwkMailboxManager.php.

echo_area_nntp_subscriptions

Maps local areas to remote groups.

Column	Type	Notes
id	SERIAL PK
echoarea_id	INTEGER FK	-> echoareas.id
peer_id	INTEGER FK	-> nntp_peers.id
newsgroup_name	VARCHAR(255)	canonical remote group
auto_created	BOOLEAN
created_at	TIMESTAMPTZ

Unique constraints:

• (peer_id, newsgroup_name) unique
• optional (echoarea_id, peer_id, newsgroup_name) unique if the first is not enough

nntp_outbound_messages

Queue of local messages waiting to be posted upstream.

Column	Type	Notes
id	SERIAL PK
echomail_id	INTEGER FK	-> echomail.id
peer_id	INTEGER FK	-> nntp_peers.id
queued_at	TIMESTAMPTZ
sent_at	TIMESTAMPTZ NULL
remote_message_id	VARCHAR(255) NULL	message-id we posted, if accepted
last_error	TEXT NULL

nntp_group_state

Per-peer, per-group pull cursor.

Column	Type	Notes
id	SERIAL PK
peer_id	INTEGER FK	-> nntp_peers.id
newsgroup_name	VARCHAR(255)
last_article_number	BIGINT	highest successfully imported article number
last_message_id	VARCHAR(255) NULL	diagnostic / recovery
last_article_date	TIMESTAMPTZ NULL	diagnostic / optional fallback
updated_at	TIMESTAMPTZ

Unique constraint:

• (peer_id, newsgroup_name) unique

Additions to echomail

Column	Type	Notes
nntp_peer_id	INTEGER FK NULL	source peer for imported NNTP messages
nntp_group_name	VARCHAR(255) NULL	source group
nntp_article_number	BIGINT NULL	source article number
nntp_message_id	VARCHAR(255) NULL	source or posted NNTP Message-ID

These should be nullable for non-NNTP messages.

New Classes

Recommended initial file layout:

src/Nntp/
  Transport/
    ClientInterface.php            connect / auth / select group / fetch / post
    SocketClient.php               raw NNTP over stream_socket_client
  NntpPeerManager.php              CRUD + credential encryption for nntp_peers
  NntpSubscriptionManager.php      CRUD for echo_area_nntp_subscriptions
  NntpArticleParser.php            parse HEAD/ARTICLE responses into normalized message objects
  NntpArticleBuilder.php           local echomail -> outbound NNTP article text
  NntpMessageId.php                shared Message-ID construction/parsing helper
  NntpInbound.php                  import upstream articles -> echomail
  NntpOutbound.php                 queued echomail -> POST transactions
  NntpPoller.php                   orchestrate pull + post for one peer

NntpArticleBuilder.php, NntpArticleParser.php, and NntpMessageId.php should be written so they are reusable by both this upstream-client transport and the future implementation discussed in docs/proposals/NNTPServer.md.

If desired, the transport interface could be generalized later so QWK and NNTP both sit under a broader inter-BBS transport framework, but that is not required for the first implementation.

Scheduler and Admin Daemon Integration

NNTP should mirror the QWK integration pattern:

• src/Binkp/Connection/Scheduler.php
  • add checkScheduledNntpPolls()
  • add processScheduledNntpPolls()
  • add getNntpScheduleStatus()
• src/Admin/AdminDaemonServer.php
  • add nntp_poll
  • add nntp_poll_sync
• src/Admin/AdminDaemonClient.php
  • add nntpPoll() and nntpPollSync()
• new CLI script:
  • scripts/nntp_poll.php

Suggested CLI shape:

Usage: php nntp_poll.php [options] [peer-id]
  --all
  --pull-only
  --push-only
  --log-level=LEVEL
  --log-file=FILE
  --no-console
  --quiet
  --json
  --help

Admin UI

Recommended admin additions:

Admin -> NNTP Peers

• list peers with enabled state, host, TLS mode, last poll, last error
• add / edit / delete peer
• test connection action
• poll now action

Echo Area edit -> Transport Subscriptions

The current QWK-specific configuration should be moved toward a transport-neutral presentation. Short term, NNTP can be added beside QWK; longer term this should become one transport subscriptions panel.

For NNTP each area needs:

• zero or more peer/group mappings
• relay mode and relay rules already supported by RelayPolicyManager

Import and Posting Rules

Authentication

• support anonymous pull where the upstream permits it
• support authenticated pull/post
• separate read vs post permissions remain the upstream server's responsibility

TLS

• support implicit TLS (nntps, usually 563)
• support STARTTLS upgrade on standard NNTP port
• certificate verification should be enabled by default

Character encoding

Inbound articles may contain legacy charsets. Normalize to UTF-8 before storage, similar to other imported message paths. Preserve the original declared charset in transport metadata or synthetic kludges if useful for diagnostics.

This should stay consistent with the UTF-8-serving assumptions in docs/proposals/NNTPServer.md. If the codebase later adds NNTP-specific charset normalization helpers, both the client transport and server proposal should use them.

Reply threading

When importing:

• try to resolve References last-hop or In-Reply-To to a known local parent using nntp_message_id or source_msgid

When exporting:

• only emit References if the parent has a known NNTP-side identifier relevant to that peer/group
• do not try to synthesize a fake upstream ancestor chain beyond what is known

This should match the threading guidance in docs/proposals/NNTPServer.md: a single-parent mapping is acceptable in the first iteration, and full ancestor-chain reconstruction is optional enhancement work.

Cross-posts

First iteration should reject or ignore multi-group outbound posting and map one queue item to one configured remote group. If an upstream article arrives with multiple groups in Newsgroups:, import it only through the configured mapped group that matched the subscription being polled.

Reader-Mode Pull vs Transit-Mode Peering

Longer term, a true transit-oriented implementation could add:

• IHAVE
• CHECK / TAKETHIS
• offered-article ingestion by Message-ID
• faster near-real-time peer feeds

That should be a later phase. The first phase should stay with scheduled reader-mode sync because it aligns with:

• the current scheduler architecture
• the admin daemon model
• the QWK poller pattern
• easier debugging

Recommended Implementation Order

1. Add NETWORK_TYPE_NNTP and TRANSPORT_NNTP
2. Add database migrations for peers, subscriptions, queue, group state, and echomail NNTP metadata
3. Implement NntpPeerManager and subscription CRUD
4. Implement low-level NNTP socket client with auth and simple command handling
5. Implement NntpArticleParser
6. Implement NntpInbound
7. Implement NntpArticleBuilder and NntpOutbound
8. Implement NntpPoller
9. Add scripts/nntp_poll.php
10. Add admin daemon + scheduler hooks
11. Add admin UI and API endpoints
12. Refactor the area configuration UI toward transport-neutral subscriptions if needed

Key Risks

• NNTP server behavior varies; OVER, XOVER, MODE READER, and auth flows are not perfectly uniform
• article numbers can expire or gap, so pull logic must tolerate holes
• imported Message-ID values may be malformed or absent on some servers
• plaintext transports need content normalization when local posts use Markdown or rich features
• some upstreams may rewrite Message-ID or Date on accepted posts
• charset handling is more variable than FTN and current QWK flows

Recommendation

Implement NNTP upstream support as a scheduled reader-mode transport that reuses the QWK client architecture patterns and the newer relay-policy model.

Concretely:

• treat NNTP as transport type nntp
• add a dedicated nntp_peers peer table
• add echo_area_nntp_subscriptions
• add nntp_outbound_messages and nntp_group_state
• import into echomail using source_msgid plus NNTP-specific metadata
• queue outbound posts from MessageHandler the same way QWK queueing already works

That gives BinktermPHP a practical path to participate in remote newsgroup networks without requiring the much larger scope of a full user-facing NNTP server or full transit peer implementation.

[coderabbitai]

🔗 Related PRs

#144 - 1.8.2 [merged]
#160 - 1.8.5 [merged]
#182 - 1.8.8 [merged]
#187 - 1.8.9 [merged]
#215 - 1.9.4 [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!