Projects: allow feature flag configuration in our config file

[agjohnson]

Maybe subtitled: "Add a non-strict dictionary to our config file for faster feature development"

We have a couple issues around feature flags and configuration of features that might make sense to view as related:

• We start features we're testing off as feature flags, but then the feature never moves to product.
• Users have to ask to be included in a feature flag. These features should either be moved to product, or should at least be opt in while in beta testing.
• And tangentially related, when adding a new feature we want to test, it's really common for us to avoid the complicated step of adding configuration specification. This makes sense, as the feature is maybe not permanent. However, we're also making a tradeoff on feature polish when we avoid nice user configuration.

I've moved discussion of finalizing outstanding feature flags at #9779. I'll stick to just discussing forward options to avoid using feature flags for testing and user opt-in for new features.

---

The bare minimum we need here is some way to allow users to at least enable beta features, so we can stop creating feature flags for optional behavior. This would be a dedicated configuration option in our file at very least.

An idea that I'm mildly excited about -- make this a non-strict dictionary that we can use for surfacing opt-in for feature flags and feature flag configuration:

readthedocs:
  features:
    magic_exit_codes:
      skip_and_fail_on: 183
      skip_and_pass_on: 184
    dont_shallow_clone: true
    sphinx_parallel: true

  # Or something like..
  beta: {}

The idea being that this is a sandbox namespace of sorts. When we are authoring a feature like magic_exit_codes, we would just allocate part of this namespace without writing the configuration specification and validating it. I actually don't know how realistic this is with our current validation code.

This would avoid surfacing user-facing feature flags, and could give a pattern to allow users to opt into beta features. We might not have to worry as much about configuration validation or schema, but give the user all the control.

If the feature is a production feature, it should have configuration on sphinx: {} or similar though.

Does this seem possible? Does it make feature development easier?

[humitos]

I like this idea in general. However, since we have already deprecated and made some features built-in into the project I'm not sure how much value it will add at this point. I don't think I have in mind currently any "feature flag" that could be a good candidate for this config schema right now. Having a specific feature we want here, will give us a lot of context to build exactly what we need, instead of something "generic" without knowing if it's going to cover our use case.

That said, I think we could discuss a little more:

• what are the kind of features we are thinking to integrate into this schema?
• are we talking here only about "build features"? I understand that all the addons configuration is still planned to be done from the dashboard UI (even while it's in beta/experimental status)
• once we deploy them as beta.feature: "value", how are we going to remove it from there once it's migrated into the product itself?

Being more permissive on the YAML validation here could be good, but it would be good to the "new build notification system" already in place for that, so we can communicate issues to users in a clear way. Like "You wrote beta.feature.* when it should be beta.features.*" or similar: readthedocs/ext-theme#164

[agjohnson]

That all makes sense, we might not need this implementation right now at all.

Great point on the build vs hosting beta feature/feature flag features, my suggestion above was definitely more for just the build process features. We have since talked about using a dashboard UI for the hosting features, so maybe that pulls this sort of configuration over towards a UI instead.

This actually seems like better UX for all feature opt in too. We'll want to eventually expire beta features, so hardcoding these in a configuration file is awkward, and we can even selectively show beta features to users (that is, we give out explicit access to opt-in to beta features).

It is a little awkward to opt into a build feature in a UI though -- I'm thinking of feature flags like using mamba, or sphinx build option beta features. Those would feel very natural in the config file.

Just some ideas, seems we could talk about this more to weigh options more, as I think there are lots of merits in all these directions.

[humitos]

This actually seems like better UX for all feature opt in too. We'll want to eventually expire beta features, so hardcoding these in a configuration file is awkward, and we can even selectively show beta features to users (that is, we give out explicit access to opt-in to beta features).

Yeah, using the UI for this experimental features may make sense because of this reason in particular. Hardcoding beta/testing/experimental features may confuse people that read them in other project's config files. Also, adding a field in the database is a lot easier than adding a config in the YAML file 😄

It is a little awkward to opt into a build feature in a UI though -- I'm thinking of feature flags like using mamba, or sphinx build option beta features. Those would feel very natural in the config file.

This is also true and I agree with you. Hopefully, we won't have these kind of feature flags anymore 🤞🏼 .

I think we have two clear directions here, but we will need more specific details about what are these experimental feature flags we want to expose to users to know what is the best place for them. So, my suggestion here is to wait on this issue and see how the projects that may require these experimental flags move on. That will give us more context and it will allow us to make a more informed and accurate decision.

[agjohnson]

Hopefully, we won't have these kind of feature flags anymore

Yeah, I was thinking the same. Build jobs and commands probably address this in a better way for users already, so maybe we don't have to worry about this sort of feature opt in anymore anyways.

So, my suggestion here is to wait on this issue and see how the projects that may require these experimental flags move on.

Sounds good 👍

I will say that if we think we can avoid major build feature flags that would be user configurable, I would lean towards a purely UI driven feature opt-in/beta opt-in UI. Maybe you'll have some more opinions here after doing more deprecations however.

[humitos]

I would lean towards a purely UI driven feature opt-in/beta opt-in UI

We already started following this path with new beta addons.

I'm closing this issue since we are moving away from exposing feature flags to users and we are happy with the current approach of exposing beta things in the UI.