Extend prost-build to generate message builders #901

mzabaluev · 2023-08-28T00:19:18Z

First crack at #399

Based on #1011

Advantages over generic macro-generated builders like derive_builder:

Type system knowledge from protobuf is applied to make the API more ergonomic:
- The setter of a field with message type Foo does not need to be specially configured to not use Option<Foo> as the argument type, we assume the setter is only used when the field is meant to be set so it's impl Into<Foo>.
- Similarly, setters of oneof fields take the oneof's enum type argument rather than Option, but no generics with the Into bound here as that could obfuscate call sites more than necessary (maybe it's a wrong assumption and the argument conventions should be made uniform).
- Setters of repeated fields are generic with an impl IntoIterator argument.
- Setters of enum fields use the domain-checked enum typed argument rather than the i32 type that the corresponding struct field is generated with.
The builder API is explicitly emitted in generated code which can be revision-controlled, rather than magicked in by a separate macro.
- Faux-derive macros that do not actually derive a trait implementation, but inject additional named API into the scope, can be considered bad style.

mzabaluev · 2023-08-29T12:24:35Z

Notes on the present implementation:

Builders for matching messages are configured with a PathMap.
Each matching message type gets an associated function to start off the builder, with the name given by the map value (converted to snake_case). This will conflict with a field getter method generated by the Message derive macro in the rare case when a message has an optional scalar or an enum field with the same name.
The builder type for a message is generated in the message-named module alongside the nested types of the message defined in the protobuf. The type's name is given by the configuration map value.
The final method of the builder is named build, but I intend to make it a configurable default.
The field setter methods of the builder use Into and IntoIterator conversions on the generic argument.
- For primitive types, this allows passing any values as long as the conversion is lossless, as per the convention on From.
- For boxed fields, impl Into<Box<Foo>> provides the convenience of being able to pass Foo or Box<Foo>.
- For repeated fields, the IntoIterator bound allows population from any iterable object. For example, a repeated string field can be initialized from an array of string literals.
- For message fields, the conversion can be useful if the application provides a Rust-idiomatic "domain type" with a From conversion into the generated message struct.
- Enum and oneof field setters do not use Into generics, for clarity at the call sites and better code completion experience.

caspermeijn

I think this PR is difficult to review. Could you do these things to make it easier to review?

Can you explain why this APi is better then a generic builder? For example: https://crates.io/crates/derive_builder
Add documentation about how to use this builder.
Reorganize the commits so that they change a single thing with a description of what and why you do that. If that commit generally improves the codebase, consider opening a separate PR for just that commit. This allows me to review each commit individually.
I am not sure about this one: Is it beneficial to split the build code generator into a separate source file?

mzabaluev · 2024-03-09T14:17:25Z

Thanks for the feedback @caspermeijn! I have rebased the branch and squashed the changes into a few topical commits.

Can you explain why this APi is better then a generic builder? For example: https://crates.io/crates/derive_builder

I have added an explanation in the description comment.

Add documentation about how to use this builder.

I mean to do it, but first I'd like to settle on a more flexible way to configure the builders.
What do you think of a way to selectively generate the builders for matching paths, and be able to configure the name to satisfy author's stylistic preferences and avoid name conflicts?

prost_buid::Config::new()
    // .mypackage.Foo gets foo::Builder and the associated Foo::builder() function
    .builders(".mypackage", "Builder")
    // ...

I am not sure about this one: Is it beneficial to split the build code generator into a separate source file?

I was hesitant to make a separate module for this, as hardly any functionality used in code_generator.rs had been broken out at the time, and there is some code reuse and similarities with other generator functions. But I can reorganize if you think it's better.

Also, I feel that the builders should be a core feature, because as discussed in #399, this together with #[non_exhaustive] brings prost-build into compliance with Protobuf conventions on evolving message definitions, where adding a field should not result in breaking code that uses the generated bindings.

mzabaluev · 2024-03-26T10:03:40Z

@caspermeijn

Reorganize the commits so that they change a single thing with a description of what and why you do that. If that commit generally improves the codebase, consider opening a separate PR for just that commit.

I could try to split off a PR with the refactoring changes in code_generator.rs that I needed to reuse data in the builder code generation, if that makes reviewing easier for you.

mzabaluev · 2024-03-27T18:53:28Z

I could try to split off a PR with the refactoring changes in code_generator.rs that I needed to reuse data in the builder code generation, if that makes reviewing easier for you.

Done in #1011 and rebased on top.

For each message, optionally generate a struct to provide the forward-compatible builder API, with inherent methods for field setters and the .build() method to produce the fully initialized message struct. If generation of builders is enabled with a name `Builder` for a message named `Foo`, an impl block with the associated fn `Foo::builder()` is appended to the top-level file scope, and the definition of the builder struct `Builder` and its methods is appended to the module `foo`. Reuse the helpers that generate field definitions for a message, but use a mode to output code for the builder fields, which must be private and devoid of docs and prost annotations. The added Config::builder method populates a PathMap with values given in the second argument to configure the builder name.

Adorn a struct with #[non_exhaustive] to showcase and test how the builder API (and Default) can be used to forward-compatibly construct values of generated message types, while the struct initializer syntax is forbidden. Some of the doc tests showcase working around name conflicts between the associated function generated for the builder API and a getter method generated by the Message derive for a field named `builder`. Also verify the no-conflict case where the field does not have a getter generated for it, so the `builder` associated fn is fine. The test crates in tests-2015 and tests-no-std are uniformly named "tests" to simplify boilerplate in the doc tests.

This allows plugging a message builder into other builders' setters for fields of the message type without the need to call .build().

The setter for field named `test` in proto must be named `r#test`.

jregistr · 2024-07-03T20:08:22Z

What else does this PR need to be merged?

caspermeijn · 2024-07-12T06:41:24Z

prost-build/src/code_generator.rs

+        self.buf.push_str(" = ");
+        if repeated {
+            self.buf
+                .push_str("value.into_iter().map(Into::into).collect();\n");


A repeated type is a Vec, but this setter doesn't allow the direct setting of a Vec. It will reallocate. I think the setter should just use Vec as its input type.

I consider the setter a convenience method. The field is public, so there is always the option of assigning to it directly rather than going through FromIterator.

When specialization is stabilized in future Rust, the setter can be specialized for Vec without breaking the API.

An impl Into<Vec<T>> argument looks like a reasonable compromise: apart from Vec<T> with zero copying, it accepts &[T], &[T; N], and [T; N], so it should cover the most common cases without extra conversions. To set from an iterator, the caller would have to call .collect::<Vec<_>>()

caspermeijn · 2024-07-12T06:43:27Z

prost-build/src/code_generator.rs

+            self.buf
+                .push_str("value.into_iter().map(Into::into).collect();\n");
+        } else if optional {
+            self.buf.push_str("Some(value.into());\n")


The setter for optional fields should accept a Option. For "normal" fields I think it is good to forgo the Option.

I applied the same rationale as for repeated fieds: the setter is complementary and allows shorthand code, while there is the option of assigning the field directly.

Thinking of the most common use case here, where the application has a valid value to set. It should be rare to need to explicitly set a field to None, that is not addressed by Default impls on the struct and the Option field type.

On the other hand, maybe the case of an explicitly optional field (as opposed to a "normal" field which must have a default for the uninitialized case) is special enough that the developers meant it to be used this way. I will change to it to Option.

caspermeijn · 2024-07-12T06:48:55Z

prost-build/src/config.rs

+    /// **`path`** - a path matching any number of types. It works the same way as in
+    /// [`btree_map`](#method.btree_map), just with the field name omitted.
+    ///
+    /// **`builder_name`** - A name for the builder type. The struct with this name


The path can match multiple messages. I think the builder_name should be a suffix so that it doesn't generate multiple builders with the same name.

The builder is generated inside the submodule named after each message, so a collision could only occur with the message's nested types and enums. This should be rare enough that it can be solved by a specific matcher for messages where it occurs.

Stylistically, I prefer my_message::Builder to my_message::MyMessageBuilder or self::MyMessageBuilder.

caspermeijn · 2024-07-12T06:49:35Z

tests-2015/Cargo.toml

@@ -11,6 +11,7 @@ edition = "2015"
 build = "../tests/src/build.rs"

 [lib]
+name = "tests"


Why is this needed? It is not related to builders.

This is needed so that the doctests I have added to the tests crate also work in these shim crates.

caspermeijn · 2024-07-12T06:51:53Z

tests/src/builders.rs

@@ -0,0 +1,139 @@
+//!#[doc(hidden)]
+
+/// # Doc tests


Why do you use hidden doc tests?

These are compile_fail test cases and other tests of generator behavior and corner cases that are of no interest to a normal library user.

Since the tests crate is publish = false, this should not matter much though.

caspermeijn · 2024-07-12T06:57:34Z

tests/src/lib.rs

@@ -24,6 +24,7 @@ cfg_if! {
    }
 }

+pub mod builders;


I would like to see an additional test: generate a builder for TestAllTypesProto3 and write that to file. This has two purposes:

It adds a example of all posible setters to the PR

It forces changes to builder API to be visible in future PRs.

The repo has examples for a test writing to file and asserting nothing changes: test_generate_no_empty_outputs

caspermeijn · 2024-07-12T06:58:19Z

What else does this PR need to be merged?

@jregistr Could you provide an additional review of the PR and the generated code?

mzabaluev · 2024-11-10T11:53:22Z

tests-2015/Cargo.toml

@@ -11,6 +11,7 @@ edition = "2015"
 build = "../tests/src/build.rs"

 [lib]
+name = "tests"
 doctest = false


There is doctest = false in both crates (also in tests-2018 which is not included in the workspace for some reason), but doctests are compiled by cargo test even without the --doc option.

mzabaluev force-pushed the builders branch 3 times, most recently from a38d703 to 1fb7926 Compare September 1, 2023 13:56

mzabaluev mentioned this pull request Sep 14, 2023

Make it easy to write forward-compatible code: struct builders? #399

Open

caspermeijn reviewed Mar 8, 2024

View reviewed changes

mzabaluev force-pushed the builders branch from b5f549b to dcfed8c Compare March 9, 2024 11:57

mzabaluev mentioned this pull request Mar 27, 2024

prost-build: refactor code generator (ground work for builders) #1011

Closed

mzabaluev force-pushed the builders branch from c052587 to eab10f7 Compare March 27, 2024 18:49

This was referenced Mar 29, 2024

Sundry refactoring in code generator (ground work for builders) #1016

Closed

style(prost-build): Consolidate field data into struct #1017

Merged

prost-build: CodeGenerator::boxed method #1019

Merged

mzabaluev added 5 commits May 9, 2024 15:29

prost-build: generate From<Builder> impls

182d22a

This allows plugging a message builder into other builders' setters for fields of the message type without the need to call .build().

Test builder for field name that is a keyword

6a16716

The setter for field named `test` in proto must be named `r#test`.

Document builders in README

91036e3

mzabaluev force-pushed the builders branch from eab10f7 to 91036e3 Compare May 9, 2024 13:19

Fix clippy lint in builders test

b525600

caspermeijn requested changes Jul 12, 2024

View reviewed changes

mzabaluev commented Nov 10, 2024

View reviewed changes

This was referenced Nov 17, 2024

Add support for unknown fields #2

Open

Auto-generate convenient constructors #813

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend prost-build to generate message builders #901

Extend prost-build to generate message builders #901

mzabaluev commented Aug 28, 2023 •

edited

Loading

mzabaluev commented Aug 29, 2023 •

edited

Loading

caspermeijn left a comment

mzabaluev commented Mar 9, 2024

mzabaluev commented Mar 26, 2024

mzabaluev commented Mar 27, 2024

jregistr commented Jul 3, 2024

caspermeijn Jul 12, 2024

mzabaluev Nov 10, 2024 •

edited

Loading

mzabaluev Nov 10, 2024

caspermeijn Jul 12, 2024

mzabaluev Nov 10, 2024 •

edited

Loading

mzabaluev Nov 10, 2024

mzabaluev Nov 10, 2024 •

edited

Loading

caspermeijn Jul 12, 2024

mzabaluev Nov 10, 2024

caspermeijn Jul 12, 2024

mzabaluev Nov 10, 2024

caspermeijn Jul 12, 2024

mzabaluev Nov 10, 2024

caspermeijn Jul 12, 2024

caspermeijn commented Jul 12, 2024

mzabaluev Nov 10, 2024 •

edited

Loading

Extend prost-build to generate message builders #901

Are you sure you want to change the base?

Extend prost-build to generate message builders #901

Conversation

mzabaluev commented Aug 28, 2023 • edited Loading

mzabaluev commented Aug 29, 2023 • edited Loading

caspermeijn left a comment

Choose a reason for hiding this comment

mzabaluev commented Mar 9, 2024

mzabaluev commented Mar 26, 2024

mzabaluev commented Mar 27, 2024

jregistr commented Jul 3, 2024

Choose a reason for hiding this comment

mzabaluev Nov 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzabaluev Nov 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzabaluev Nov 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caspermeijn commented Jul 12, 2024

mzabaluev Nov 10, 2024 • edited Loading

Choose a reason for hiding this comment

mzabaluev commented Aug 28, 2023 •

edited

Loading

mzabaluev commented Aug 29, 2023 •

edited

Loading

mzabaluev Nov 10, 2024 •

edited

Loading

mzabaluev Nov 10, 2024 •

edited

Loading

mzabaluev Nov 10, 2024 •

edited

Loading

mzabaluev Nov 10, 2024 •

edited

Loading