[Go Functions] Close the configuration parity gaps with the Java and Python runtimes (master issue)

[david-streamlio]

Search before reporting

• I searched in the issues and found nothing similar.

This is a master issue. The individual gaps are tracked as the sub-issues listed below; this one exists to hold the comparison that produced them and to say what they have in common.

#26412 is the counterpart for the Python runtime, from the same audit.

Motivation

The Go function runtime honours a much smaller part of FunctionDetails than the Java and Python runtimes do. Most of the difference is silent: the configuration is accepted by pulsar-admin, stored by the worker, reported back faithfully by functions get, carried into the instance in the protobuf, and then never read.

I audited the three runtimes field by field against Function.proto, reading the consuming code rather than grepping for field names — the generated Function_pb2.py and pulsar-function-go/pb contain every field name, so a naive search reports full support for all three.

Configuration	Java	Python	Go
receiverQueueSize	yes	yes	yes
subscriptionType / subscriptionName / subscriptionPosition	yes	yes	yes
timeoutMs	yes	yes	yes
compressionType / batchBuilder	yes	yes	yes
userConfig / secretsMap / logTopic / autoAck	yes	yes	yes
retainOrdering	yes	yes	yes (#26414)
retainKeyOrdering	yes	yes	yes (#26414, #26421)
retryDetails (deadLetterTopic, maxMessageRetries)	yes	no (#26397)	no
forwardSourceMessageProperty	yes	yes	no
batchingSpec	yes	yes (#26392)	yes (#26393)
cryptoSpec (consumer)	yes	yes	no
schemaProperties (consumer)	yes	yes	no
poolMessages / consumerProperties / messagePayloadProcessorSpec	yes	no	no
negativeAckRedeliveryDelayMs	yes	yes (#26413)	yes (#26415)
EFFECTIVELY_ONCE	yes	yes	panics (explicit)

Producer maxPendingMessages and maxPendingMessagesAcrossPartitions are absent from both runtimes on master but are already addressed by the open batching PRs (#26392, #26393), so they are not tracked here. useThreadLocalProducers has no equivalent in either client.

The pattern worth naming. EFFECTIVELY_ONCE is the one the Go runtime gets right:

// pulsar-function-go/pf/instanceConf.go:137
if instanceConf.funcDetails.ProcessingGuarantees == pb.ProcessingGuarantees_EFFECTIVELY_ONCE {
    panic("Go instance current not support EFFECTIVELY_ONCE processing guarantees.")
}

Unsupported, and it says so. Every other row in that table fails the other way: accepted and ignored. An operator has no signal short of observing the behaviour in production and inferring backwards. Where a gap is not going to be closed soon, refusing it explicitly is strictly better than dropping it silently, and that would be a reasonable interim resolution for several of these.

Solution

Progress. Two of the five below are fixed on master, along with the separately-filed batching gaps; the table above reflects the current state rather than the state at filing.

Close the gaps, tracked individually:

• retainOrdering / retainKeyOrdering ignored when selecting the subscription type — [Go Functions] retainOrdering and retainKeyOrdering are ignored, so key ordering is silently not preserved #26405 (fixed by [fix][fn] Honour retainOrdering and retainKeyOrdering in the Go function runtime #26414 and [fix][fn] Allow retainKeyOrdering on Go functions #26421)
• retryDetails populated but never applied, so no dead letter policy is created — [Go Functions] deadLetterTopic and maxMessageRetries are carried into the instance but never applied #26406
• most of ConsumerSpec unapplied: cryptoSpec, schemaProperties, consumerProperties, poolMessages, messagePayloadProcessorSpec — [Go Functions] Most of ConsumerSpec is ignored: cryptoSpec, schemaProperties, consumerProperties, poolMessages #26407
• forwardSourceMessageProperty not implemented — [Go Functions] forwardSourceMessageProperty is not implemented, so source message properties are dropped #26408
• negativeAckRedeliveryDelayMs not applied to the consumer — [Go Functions] negativeAckRedeliveryDelayMs is ignored, so the client default always applies #26409 (fixed by [fix][fn] Honour negativeAckRedeliveryDelayMs in the Go function runtime #26415)

Related, filed separately because they are a different runtime and a different fix:

• Python: poolMessages / consumerProperties / messagePayloadProcessorSpec unapplied — [Python Functions] consumerProperties, poolMessages and messagePayloadProcessorSpec are ignored #26410
• Python: negativeAckRedeliveryDelayMs unapplied — [Python Functions] negativeAckRedeliveryDelayMs is ignored, so the client default always applies #26411 (fixed by [fix][fn] Honour negativeAckRedeliveryDelayMs in the Python function runtime #26413)

Filed separately and not duplicated here: #26390 / #26391 (batching — both closed by #26392 / #26393), #26397 (Python dead letter, PR #26400 open), #26403 (Go custom metric collectors, PR #26458 open).

Each sub-issue is independently fixable and none blocks another. Ordering by user impact, retainKeyOrdering first: it is the only one in the list whose absence silently breaks a guarantee the user explicitly asked for.

Alternatives

Fixing these as one change was considered and rejected: they touch different parts of instance.go, some need a design decision (whether an unimplementable field should refuse rather than ignore), and a single large PR would be hard to review and hard to revert selectively.

Anything else?

The audit was done against origin/master. Java is treated as the reference implementation throughout — I spot-checked the Java side for the rows that mattered (PulsarSource for retryDetails, BatchingUtils for batchingSpec, JavaInstanceRunnable for the guards) rather than auditing it exhaustively.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[david-streamlio]

Correction to the matrix above: the cleanupSubscription row has been removed. It claimed the Go runtime honours the field and Python does not; neither half was right.

cleanupSubscription is not an instance concern in any runtime. The worker implements it in FunctionActioner.java:403, iterating the input specs on function delete and deleting each subscription — so it works identically for Java, Python and Go. The Go runtime does carry the field into its own config at instanceConf.go:103 and never read it, which is what misled me, but nothing there needs to read it.

No other row is affected; the remaining Go gaps in the list are unchanged and verified.

[david-streamlio]

Three of the five sub-issues now have PRs, all green:

• [Go Functions] retainOrdering and retainKeyOrdering are ignored, so key ordering is silently not preserved #26405 (retainOrdering / retainKeyOrdering) → [fix][fn] Honour retainOrdering and retainKeyOrdering in the Go function runtime #26414
• [Go Functions] negativeAckRedeliveryDelayMs is ignored, so the client default always applies #26409 (negativeAckRedeliveryDelayMs) → [fix][fn] Honour negativeAckRedeliveryDelayMs in the Go function runtime #26415

Still unclaimed: #26406 (dead letter policy), #26407 (ConsumerSpec: cryptoSpec, schemaProperties, consumerProperties, poolMessages), #26408 (forwardSourceMessageProperty).

@freeznet — flagging since you have been working in the function runtimes. #26414 is the one worth looking at first if you only have time for one: it is the only item here with a correctness consequence rather than a missing capability, since --retain-key-ordering currently produces a Shared subscription and silently loses per-key ordering.

Note #26414 and #26415 both touch setupConsumer, so whichever merges second needs a trivial rebase. They are otherwise independent.