diff --git a/content/design-decisions/_index.md b/content/design-decisions/_index.md index 2e1468b2..d16b442d 100644 --- a/content/design-decisions/_index.md +++ b/content/design-decisions/_index.md @@ -1,3 +1,5 @@ + + +++ title = "Protobuf Team Design Decisions" weight = 88 @@ -14,4 +16,8 @@ These positions were taken after careful consideration and won't be overturned on a whim, but are open to being revisited as we gain new information, and as things develop in the broader ecosystem context. -* [No Nullable Setters/Getters](/design-decisions/nullable-getters-setters) +* [No Nullable + Setters/Getters](/design-decisions/nullable-getters-setters) +* [Opaque APIs](/design-decisions/opaque-apis) +* [Implicit + Presence](/design-decisions/implicit-presence) diff --git a/content/design-decisions/implicit-presence.md b/content/design-decisions/implicit-presence.md new file mode 100644 index 00000000..2dad6dcf --- /dev/null +++ b/content/design-decisions/implicit-presence.md @@ -0,0 +1,266 @@ + + ++++ +title = "Implicit Presence" +weight = 91 +description = "Covers why the Protobuf team regrets implicit field presence" +type = "docs" ++++ + +Implicit presence — the proto3 behavior in which a singular scalar field has no +hazzer and the value `0`, `false` or `""` is indistinguishable from "never set" +— is a design decision the Protobuf team regrets. + +We do not recommend using implicit presence for new fields. + +* In proto3, prefer to tag your fields `optional`. This is a weak guidance + though; if you are still using the Go Open API you may have good reason to + choose to prefer implicit presence fields. +* Explicit presence is the silent default in [Protobuf + Editions](/editions/overview) going forward. + +For the operational details of how presence behaves in each dialect, see the +[Field Presence](/programming-guides/field_presence) +application note. This page is about *why*. + +## What Implicit Presence Is {#what} + +[Field presence](/programming-guides/field_presence) is +whether a message tracks that a field was set, separately from what it was set +to. + +* Under **explicit presence**, the message stores a presence bit alongside the + value and generates a `has_foo()` hazzer. Setting `foo = 0` makes the field + present and writes it to the wire; clearing it makes it absent. +* Under **implicit presence**, the message stores only the value, and no + hazzer is generated. Setting `foo = 0` is indistinguishable from clearing + `foo` or never touching it, and `0` is never written to the wire. + +The choice for explicit or implicit presence applies only exists for singular +numeric, `bool`, `string`, `bytes` and `enum` fields. Message fields and `oneof` +members are always explicit presence, and repeated fields and maps are always +implicit presence. + +### proto3 {#proto3} + +In proto3, implicit presence is the default for those field types, and the +`optional` keyword opts back in to explicit presence: + +```proto +syntax = "proto3"; + +message UserAccount { + // Implicit presence (the proto3 default). + // No has_login_count() is generated, login_count = 0 is indistinguishable + // from unset, and 0 is never serialized. + int32 login_count = 1; + + // Explicit presence (recommended). + // has_display_name() is generated, and display_name = "" is serialized. + optional string display_name = 2; +} +``` + +### Editions {#editions} + +Editions removed the `optional` keyword and replaced it with the +[`field_presence` +feature](/editions/features#field_presence), which +defaults to `EXPLICIT`. Implicit presence is now the behavior you must +specifically set: + +```proto +edition = "2023"; + +message UserAccount { + // Explicit presence, no keyword required. + string display_name = 1; + + // Implicit presence requires an explicit opt-in. + int32 login_count = 2 [features.field_presence = IMPLICIT]; +} +``` + +> **NOTE:** Implicit presence does not exist in proto2. Every singular proto2 +> field has explicit presence. + +## Why it was Introduced {#why-introduced} + +Implicit presence was not introduced for its semantics. It was introduced to +enable [open struct APIs](/design-decisions/opaque-apis). + +In a language without zero-cost value optionality, a public struct field cannot +carry a presence bit; the only way to represent "unset" is to box the field into +a pointer. The proto3 premise was that if the data model simply deleted the +distinction between "unset" and "set to the default value," generated code could +be plain structs of unboxed public fields, with no accessors and no `hasbits` — +and that this would be a good enough deal to be worth the semantic compromise. + +The original proto3 design doc put it this way: + +> Removal of field presence logic for primitive value fields, removal of +> required fields, and removal of default values. This makes Proto3 +> significantly easier to implement with open struct representations, as in +> languages like Android Java, Objective-C, or Go. + +The design rationale rested on two claims: that open structs made protobuf +meaningfully easier to implement, and that presence tracking was pure overhead +for most fields, both of which turned out to not be true. + +## The Open Struct Premise Did Not Hold {#open-structs} + +Google realized that open structs were overly constraining on implementations, +as explained in [Opaque +APIs](/design-decisions/opaque-apis). + +So the API shape that implicit presence was designed to unlock is no longer +considered a good design, while the semantics it traded away are still present. + +## It Is Not Faster {#performance} + +The most persistent belief about implicit presence is that not maintaining +hasbits would mean less work. This has turned out not to hold in practice. + +Explicit presence packs presence into dense 32-bit words at the front of the +message. That density is the point — it lets whole-message operations make +decisions about many fields with a single load: + +* **Batch skipping.** In `Clear()`, `ByteSizeLong()` and `MergeFrom()`, the + C++ generator gates a group of fields behind a single mask test + (`BatchCheckHasBit(cached_has_bits, 0x000000ffU)`), and can gate whole + regions of a wide message behind a test over several has-words. Absent + fields are skipped without their memory ever being touched, which matters + because real messages are sparse and cold fields are exactly the ones you do + not want to pull into cache. +* **Branchless sizing.** For a group of fixed-width fields, `ByteSizeLong()` + computes the contribution of the entire group as `popcount(mask & + cached_has_bits) * size`, with no branches and without loading a single + field value. The generator restricts this optimization to fields that + `has_presence()`, because it is only valid when the bit is authoritative. + +Implicit presence forfeits all of this. Presence is tied to the value, so +deciding whether a field participates requires loading the field and comparing +it against zero — once per field, every time, across the whole struct. + +More recently, we have begun having hasbits even for implicit presence fields +anyway: these are hints a set bit means "probably present," so accessors must +still compare the value, but the batch skip they enable has proven to be a +beneficial optimization. + +Once you implement this optimization, implicit presence fields actually are +strictly *more* work per field than explicit presence: an explicit-presence +field tests one bit and acts but an implicit-presence field tests the hint bit +*and then* loads and compares the value. + +## The Semantics Are 'Bad' {#semantics} + +Implicit presence does not remove the concept of presence. It makes zero and the +empty string into magic values and ties presence to them. The distinction +resurfaces anywhere the system needs to talk about what was *set* rather than +what the value *is*. + +### Merging {#merging} + +`MergeFrom` copies fields that are present in the source over the destination. +Under implicit presence, a field set to zero is not present, so it does not +merge — which means a merge can produce a value that existed in neither input. + +The canonical demonstration is a well-known type. `google.protobuf.Timestamp` +was introduced with proto3, so both of its fields have implicit presence: + +```proto +syntax = "proto3"; + +package google.protobuf; + +message Timestamp { + int64 seconds = 1; + int32 nanos = 2; +} +``` + +Take three messages containing a `Timestamp`: + +* `A = {timestamp: {seconds: 100, nanos: 50}}` — 100.000000050s +* `B = {timestamp: {seconds: 200, nanos: 1}}` — 200.000000001s +* `C = {timestamp: {seconds: 200, nanos: 0}}` — 200.000000000s + +Submessages merge recursively, field by field: + +* `A.MergeFrom(B)` gives `{seconds: 200, nanos: 1}`. Both of `B`'s fields are + non-zero, so both are present, so both overwrite. +* `A.MergeFrom(C)` gives **`{seconds: 200, nanos: 50}`**. `C`'s `nanos: 0` is + indistinguishable from unset, so it does not overwrite, and `A`'s old + `nanos: 50` survives. + +Merging the instant 200.000000000s onto 100.000000050s produces 200.000000050s: +a timestamp that appears in neither operand, off by 50 nanoseconds, with no +error anywhere. Had `Timestamp` been defined with explicit presence, `C` would +carry a set bit for `nanos` and the merge would be correct. + +Nothing about this is specific to timestamps. Any message whose meaning depends +on a combination of fields — a coordinate, a range, a version triple, a ratio — +has the same hazard, and the hazard is worst precisely where zero is a perfectly +ordinary value. + +### MergeFrom(msg) != MergeFrom(bytes) + +Protobuf intends to have `x.MergeFrom(Parse(bytes))` semantically be the same as +`x.MergeFrom(bytes)` as an important first class semantic. + +One of the rare edge cases where this is violated is implicit presence: if the +bytes did contain a zero value, the `Parse(bytes)` will effectively discard it +and so the subsequent merge operation will not propagate it. By contrast, when +merging directly from bytes the zero value can be 'known' and replace a value +that was already set. + +### Patch Semantics {#patch} + +The same rule breaks partial updates in general. If a field has implicit +presence, a patch cannot express "set this to zero," because a zero in the patch +is indistinguishable from an omission. That gap is what +`google.protobuf.FieldMask` exists to fill: an out-of-band, hand-maintained list +of which fields the caller meant, however `FieldMask` was found to have other +serious design deficiencies. + +### JSON {#json} + +Native JSON implementations essentially have explicit presence: after a parse +you can check if a given key was present on the wire encoding or not. + +In ProtoJSON, with implicit presence there is no hazzer: a zero-valued field +simply disappears from the output, and there is no easy way to 'set' the field +in a way that will signal it should be emitted. The only available remedy was to +add serializer options for including default values, which is exactly the kind +of format-dialect knob we otherwise [work hard to +avoid](/programming-guides/json#json-options). Explicit +presence makes this a non-question: set the field and it is emitted, clear it +and it is not. + +## The Escape Hatches Were Worse {#workarounds} + +Because real systems do need presence, proto3 users were routed to +`google.protobuf.Int32Value` and friends, to `FieldMask`, and to single-field +`oneof` blocks. Each of these is worse than a hasbit on every axis that matters: + +* **Wrapper types are not wire-compatible with the primitive field they + replace**, so the decision has to be made correctly before the first release + and cannot be reversed. +* **Wrapper types are more expensive**, turning one bit in a bitvector into a + nested submessage with its own pointer, its own allocation, and its own + unknown-field set. +* **`FieldMask` moves presence out of the message** and into a parallel + structure the application has to keep in sync by hand. +* **Single-field `oneof` blocks** do carry presence, but they misrepresent the + schema, they generate a different API in every language, and that API is + consistently worse than a hazzer. + +These were not theoretical costs. The extra allocation per wrapper was a +measurable source of latency in real client libraries, and [the GitHub issue +asking for presence to be readded to +proto3](https://github.com/protocolbuffers/protobuf/issues/1606) became the +most-commented issue in the Protobuf repository. + +The `optional` keyword was restored to proto3 as a result: accepted behind +`--experimental_allow_proto3_optional` in v3.12.0, and Editions then made +explicit presence the default outright. diff --git a/content/design-decisions/opaque-apis.md b/content/design-decisions/opaque-apis.md new file mode 100644 index 00000000..46018179 --- /dev/null +++ b/content/design-decisions/opaque-apis.md @@ -0,0 +1,214 @@ + + ++++ +title = "We Prefer Opaque APIs" +weight = 90 +description = "Covers why Protobuf generated code hides a message's memory representation behind accessors" +type = "docs" ++++ + +Generated Protobuf code can present a message in one of two broad styles: + +* **Open struct APIs** make a message's fields public members of a plain + struct. User code reads them, writes them, and takes their address directly: + `msg.foo = 42`, `*msg.foo`, `&msg.foo`. +* **Opaque APIs** keep the in-memory representation private and expose the + message only through generated methods: getters, setters, hazzers, clearers + and, in some languages, builders. + +The Protobuf team favors opaque APIs in every language. Open structs look +simpler, and in some languages they look more idiomatic, but they promote a +message's physical memory layout into public API, which causes long term +problems. + +## Opaqueness Is Leverage {#leverage} + +Protobuf is unusual in two ways, and both point in the same direction. + +First, Protobuf is depended on by nearly everything, so *every* observable +behavior of generated code is depended on by someone, somewhere. This is +[Hyrum's Law](https://www.hyrumslaw.com/) at full strength: in practice, +anything we expose we can never change. + +Second, Protobuf sits underneath serialization, RPC, and storage for entire +fleets of services, so even small improvements to parsing or to the in-memory +representation have an unusually large aggregate impact. + +Put together, too many details exposed in the generated API tends to be the +single largest constraint on our ability to make Protobuf faster and smaller. We +have found that more opaqueness gives a high amount of leverage by comparison: +it lets us be more deliberate about exactly which behaviors are guaranteed, and +it leaves room to change the implementation underneath. + +## Examples of What Opaqueness Buys {#what-it-buys} + +Each of the following is impossible, or badly compromised, when field storage is +public API. + +### Cached State {#cachedstate} + +Due to the binary wire format using size-prefixes, the typical serialize +implementation must do two passes: one to compute encoded lengths and then one +that uses those lengths to write. + +With Opaque APIs, you can maintain a cached encoded length internally, and +invalidate it when modifications occur. This unlocks significantly speeding up +serializations. + +With Open Structs you cannot: without setters there is no way to 'know' if state +has been modified which may have modified the encoded length, which means you +must redo the sizing work on every serialization. + +### Lazy Decoding {#lazy} + +With Opaque APIs, decoding can be deferred past parse time. Lazy submessage +fields, lazy extensions, lazy UTF-8 validation, and similar techniques all work +by keeping the raw bytes and doing the work on first access, inside the getter. +A service that reads a few top-level routing fields and forwards the rest never +pays for the subtree at all. + +With Open Structs you cannot: a plain field read is not a function call, so +there is no interception point, and the parser has no choice but to eagerly +resolve everything as it materializes the message tree. + +### Unboxed Values and Dense Hasbits {#hasbits} + +Tracking [explicit +presence](/programming-guides/field_presence) requires +storing one bit of information per field. An opaque API stores it as exactly +that: one bit in a dense `hasbits` bitvector next to the inline field value. + +An open struct in a language without zero-cost value optionality cannot do this, +because the presence bit is not reachable from the field expression itself. The +typical representation ends up being a pointer per field — `*int32`, `*bool`, +`java.lang.Integer` — which costs a machine word in the parent struct, a +separate allocation, an extra indirection on every read, and garbage collector +pressure, all to carry one bit. Even in languages like Rust with Option, the +Open API precludes maintaining a dense hasbit bitmask over all of the fields, +and instead spreads out the option bit throughout the struct (including +typically with a lot of wasting padding per field). + +Dense hasbits are not just cheaper to store; they make whole-message operations +cheaper, because `Clear()`, `ByteSizeLong()` and `MergeFrom()` can test a mask +of many fields at once and skip them without ever loading the field data. For +more on this, see [Implicit +Presence](/design-decisions/implicit-presence#performance). + +### Freedom to Change the Memory Layout {#layout} + +When callers cannot depend on field offsets and cannot hold a pointer into a +message, the runtime can rearrange the message freely: + +* **Layout by access pattern.** Frequently accessed fields can be placed + together in the first cache line, and rarely set fields can be moved out + into an overflow allocation that only exists when one of them is set. These + are workload-specific decisions, which means they are best driven by + profiles rather than fixed at code-generation time. +* **Arena and custom allocation.** Message internals can be allocated out of + contiguous arenas or slabs, which is only safe if no caller is holding a raw + pointer to a field whose lifetime we are about to change. +* **Compact representations for strings and bytes.** Inline small buffers, + reference-counted or shared slices, and views into the parse buffer are all + available when the accessor mediates every read. + +None of these are one-time wins. The point is that they remain *available*: a +future representation change ships as a new runtime, not as a migration of every +caller. + +## Case Study: GoProto {#go} + +Go is the canonical example of a language that started with an open struct API +and moved to an opaque one. + +Historically, GoProto generated public structs in which explicit-presence +scalars were pointers (`*int32`, `*string`) and proto3 implicit-presence scalars +were bare values. Open structs were considered a requirement for a high-quality +Go implementation at the time proto3 was designed. + +In practice the opposite proved true: the Go maintainers found that open structs +had boxed them in. Fields could not be reordered, lazy decoding could not be +implemented, and presence could not be unboxed. + +Callers could also depend on the generated struct itself, to the point that +adding a single unused zero-sized field to generated code broke thousands of +tests , +which meant that even changes that were pure wins were prohibitively expensive +to roll out. + +The Go protobuf developers, who argued strongly that they could not create a +high-quality, efficient implementation without open struct, reversed this +decision and introducing accessor methods. They realized that using open structs +boxed them in, restricting what optimizations they are able to implement. With +open structs, they can't reorder fields or implement lazy decoding. If they had +avoided open structs, they would have more flexibility now with how to optimize. + +The result was the Go Opaque API, in which struct fields are unexported and +access goes through `Get*`, `Set*`, `Has*`, `Clear*` and builders. It is the +default for [Edition 2024](/editions/overview) and newer, +and the published results line up with the reasoning above: presence for +elementary fields moved from a pointer per field to a bit per field, allocation +counts in the reference benchmarks dropped by roughly 46% and 58%, and the lazy +decoding benchmark saves over 50% of the work and over 87% of allocations. For +details, see [Go Protobuf: The new Opaque +API](https://go.dev/blog/protobuf-opaque), [Go Opaque API +FAQ](/reference/go/opaque-faq) and [Go Opaque API +Migration](/reference/go/opaque-migration). + +Note that moving to the Opaque API is a migration, not a deprecation: the Open +Struct API continues to be supported. + +Go was not the only such experiment. `javanano`, an open struct Java generator +for Android, was also considered important when proto3 was designed — its +generated messages were mutable objects with public fields and no builders. It +was likewise abandoned in favor of the encapsulated `javalite` runtime. Two +independent attempts at open structs, in two very different languages, both +ended the same way. + +## How Opaque Is Opaque Enough? {#how-far} + +More opaqueness is always more implementation freedom, but it is not free: past +some point it costs fluency in the target language's ordinary idioms. There is a +real trade-off here, and we make it deliberately, per language, based on what +capability is actually unlocked. + +Protobuf Rust is the current frontier of that trade-off. Idiomatic Rust would +have submessage accessors return `&Foo` and `&mut Foo`, and string accessors +return `&str`. Instead, Rust Protobuf returns view and mut *proxy* types, +`FooView<'_>` and `FooMut<'_>`, because native references would impose two +language-level invariants we cannot satisfy: + +* **A `&Foo` requires a real Rust `Foo` to exist in memory at that address.** + A core goal of Protobuf Rust is adding Rust to binaries that already use C++ + Protobuf at zero cost, sharing messages across the FFI boundary as plain + pointers rather than serializing across it. Under that design a parsed + message's children live in the C++ heap and have no corresponding Rust + object to borrow; handing out `&Foo` would mean eagerly materializing a + mirror Rust struct for every submessage. It would also fix the memory + representation, which is exactly what we want to keep free: Protobuf Rust + supports several kernels with different layouts behind one API. +* **A `&mut Foo` can be passed to `std::mem::swap`.** A bytewise swap of two + submessages belonging to different parents cannot fix up arena pointers, and + more generally it removes our ability to maintain any invariant between a + parent message and its children. Note that this example may be fixed in + upcoming changes to Rust, but it is still the case today that there is no + way to offer `&mut` which is ineligible for `swap`. + +Proxy types cost Rust callers some familiarity, and we know it. We took that +cost because it is what makes zero-copy C++/Rust interoperability possible at +all; no smaller amount of opaqueness would have done it. See [Rust Proto Design +Decisions](/reference/rust/rust-design-decisions#view-mut-proxy-types) +for the further discussion. + +## What Opaqueness Is Not {#what-it-is-not} + +Opaque APIs are about hiding *representation*, not about hiding data or adding +ceremony. In particular: + +* Accessors are not an invitation to add behavior. A getter returns the field + value, or the field's default if it is unset, see [No Nullable + Setters/Getters](/design-decisions/nullable-getters-setters). +* Opaqueness is not immutability. Whether messages are mutable is a separate, + per-language decision. +* Opaqueness is not a reason to skip reflection. Every runtime offers Protobuf + reflection; what an opaque API removes is the temptation to reach for + *language* reflection over fields instead. diff --git a/content/programming-guides/field_presence.md b/content/programming-guides/field_presence.md index 22d99891..23a02d21 100644 --- a/content/programming-guides/field_presence.md +++ b/content/programming-guides/field_presence.md @@ -14,9 +14,12 @@ where the generated message API stores field values (only), and *explicit presence*, where the API also stores whether or not a field has been set. {{% alert title="Note" color="note" %}} We -recommend always adding the `optional` label for proto3 basic types. This -provides a smoother path to editions, which uses explicit presence by -default.{{% /alert %}} +recommend adding the `optional` label for proto3 basic types by default. This +provides a smoother path to editions, which uses explicit presence by default. +See +[Implicit Presence](/design-decisions/implicit-presence) +for the reasoning, and for the cases where it doesn't +apply.{{% /alert %}} ### Presence Disciplines {#disciplines} @@ -174,7 +177,10 @@ generated as in proto2 APIs. This default behavior of not tracking presence without the `optional` label is different from the proto2 behavior. We recommend using the `optional` label with -proto3 unless you have a specific reason not to. +proto3 unless you have a specific reason not to. See +[Implicit Presence](/design-decisions/implicit-presence) +for why the Protobuf team regrets implicit presence, and for the cases where +choosing it is still reasonable. Under the *implicit presence* discipline, the default value is synonymous with "not present" for purposes of serialization. To notionally "clear" a field (so