Expose available blocks via an API

[johnbillion]

Apologies if this is a duplicate, but GitHub search returns hundreds of results for anything relating to blocks and APIs.

Given that most blocks are defined client-side in React, I don't believe there's currently a means of exposing the available blocks via an API outside of the context of the Gutenberg editor.

I would like to see, for example, a REST API endpoint which exposes all the available blocks and their attributes for a given post ID or post type. This would open up the possibility of programatic management of blocks within a post via an external editor.

[adamsilverstein]

@johnbillion Possibly related: this PR to add block data to the posts rest api endpoint - #2649

[johnbillion]

I've been thinking about this some more. Client-side-only registering of block types almost completely eliminates the ability to interact with blocks beyond the block editor.

Available blocks aren't exposed to anything beyond Gutenberg in a web browser, which means it's impossible to add support for blocks to:

• The WordPress mobile apps.
• The importer and exporter.
• WP-CLI.
• The REST API and XML-RPC.
• The template hierarchy.
• Server side block validation.
• Plugins for AMP, Apple News, Instant Articles, anything else that may wish to reformat content.
• Literally anything that isn't the Gutenberg editor in a web browser.

I think the optional call to register_block_type() in PHP should be made required in order to register a block and its attributes.

[mtias]

This is an area we have bounced several ideas back and forth. I totally agree with the use cases @johnbillion mentions, specially knowing which blocks are available server side. There are a few nuances that need to be taken into account:

• The client-side only registration is mostly meant for static blocks (save HTML directly).
• Static blocks are not meant to be manipulated server side after they are saved. This would cause deviations in what you see in the editor and what you get in the site.
• The use of static blocks in contexts like templates has been as placeholders which only become data when saved.
• How and where a block is defined or registered has different pros and cons we need to weight.
• Loading blocks without server involvement in purely client side apps.

All that said, one thing we discussed exploring some time ago with @nb and @aduth was to have a json entry point for a block, which is both loaded server and client side if needed, and defines a block in a context-agnostic way. It also has some drawbacks — loading it is arguably more convoluted, access to translation functions becomes harder, etc, but it might make sense for the "know which blocks are registered". The other consideration is that blocks can be registered or unregistered at run time based on multiple considerations (CPTs, capabilities, etc).

Still, this is one of the final API items that need attention and clarity.

Here are some relevant issues:

#886 — PHP wrapper to add new blocks.
#2751 — Consolidate attributes definition, server-side awareness of block types.
#3962 — Initial explorations to migrate to server-registered blocks (merged).

[aduth]

I think this could be considered a duplicate of #2751 .

I've had quite a few discussion to this point, and one thing that seems clear — to me anyways — is that for better and worse, we settled on HTML as the ultimate source of truth, not blocks (see "Shape of the Data" in Editor Technical Overview). Given this, blocks in the Gutenberg client are an implementation detail, the output of which being markup that the server happily saves to post_content.

I'd not oppose granting more access to block meta-information from the server, particularly registered name, description, etc. But I'm cautious of setting expectations that there are client-only implementations which favor user and developer experience but compromise the ability to perform all actions from the server, notably generating the output markup of a static block.

It is technically possible that save could be implemented from the server, but this also comes with a number of drawbacks: (a) losing code reuse between edit and save, (b) fragmentation of implementation for largely overlapping logic, (c) increased dependency on a server component (meaning Gutenberg as a client is not very portable to other or non-PHP systems), (d) server round-trips for currently trivial tasks like switching to HTML mode, toggling block HTML editing.

Even when it comes to server understanding of the blocks which exist in a post, we've granted some ability to inspect blocks via the post grammar, but early API discussions in #104 and a desire to avoid excessive data duplication led to the introduction of attribute markup sourcing which, while not technically impossible to support in PHP (e.g. via DOMDocument), I expect would not be quite as robust as a browser implementation.

As I see it, we're in this strange conflict where we'd originally established that the server component should only care about the markup of post_content, but as time has progressed we want make available more of what blocks constitute the shape of content; some "desired", some obviously required as we look forward to blocks as the basis for site building customization and templating.

Forgive me as this is clearly a brain dump, and I don't have clear suggestions aside from those laid out in #2751 and related issues. I'm merely acknowledging and giving insight to some of the compromises which have knowingly been made in having the bulk of the block registration occur in JavaScript.

[mtias]

The key thing here is that we accurately expose the needs and tradeoffs. The notion of a "block" is a bit misleading in this case because it applies to several entities that work differently and have different mechanisms, even if it is all presented coherently to the user. For example, we have two distinct types of blocks: those that need the server to compute the output and those that don't. The ones that don't — which we have so far called static blocks — are inherently portable and ephemeral and there is no expectation that the server can build them (the server not being an editor).

As I see it, we're in this strange conflict where we'd originally established that the server component should only care about the markup of post_content.

I would add some more nuance to this. The server does care about the structure of post_content (knowing which and how many blocks are present, for example) but it doesn't care about the rendering mechanism for a static block which is just HTML to the server. It is only augmented as a block during editing operations. On the other hand, the server obviously cares about dynamic blocks since it has to build them for output. Dynamic blocks are entirely built from their JSON-encoded attributes and not from markup.

One of the problems this split generates is that we end up with blocks that are solely registered on the client and the server doesn't know they even exist. This is something we can address, though, as a specific case of utility. The reason being that exposing which blocks are available on the server is useful, regardless of their nature (static or dynamic).

[bradleykirby]

How feasible is it to rework static blocks to originate from structured data on the server? It sounds like dynamic blocks already work this way but I think it's important for static blocks to do so as well.

Treating markup as the source of truth for static blocks is going to be a real obstacle to efforts to read and manipulate block data outside of Gutenberg (Rest APIs, CLIs, alternative editors like 3rd party page builders.)

I understand this would be a major change but otherwise I fear Gutenberg becomes a bit of worst-of-both-worlds where some pre-Gutenberg expectations are still broken but all the awesome promise of easily portable and manipulatable block content isn't possible either.

[mtias]

@bkirby989:

How feasible is it to rework static blocks to originate from structured data on the server?

Technically? Not too hard. But such a change is way beyond the scope of Gutenberg as it supposes fundamentally changing how WordPress stores post content, which would need its own release cycle, planning, and explorations as it multiplies the amount of problems and things to sort out, not to mention compatibility with front-end tools (themes and plugins). Internally, Gutenberg operates as if content was purely structured data, so, if anything, it paves the way for such an approach to happen in the future for WordPress.

A feasible thing, if you are interested, would be to explore along these lines as a plugin #4636 (comment)

Treating markup as the source of truth for static blocks is going to be a real obstacle to efforts to read and manipulate block data outside of Gutenberg

Static blocks can be seen as convenient ways to write semantic and consistent HTML, and the expectation is that only editors would want to manipulate them at the attribute level — for which we have a grammar specification that is portable, and sourcing functions for JS contexts.

You might have already read it, but I suggest looking at https://wordpress.org/gutenberg/handbook/language/ for more context.

[johnbillion]

I'd like to keep this issue on the topic of exposing available blocks via an API. Once a block instance is in place in a post (stored in post content), for the sake of this issue it should be assumed that is the source of truth with regard to the block instance.

However, what is needed is the ability for clients other than Gutenberg in a web browser to be able to retrieve a list of available block types for a given post or post type. Such an API shouldn't need to care whether a block's output is static or dynamic. It needs to know which block types are available, which properties they have, and how to save their data (the last point could be optional if their was a CRUD API which abstracted away the storage of each block instance).

As I mentioned above, I think the optional call to register_block_type() in PHP should be made required in order to register a block and its attributes. There's no other way to expose block types to anything beyond Gutenberg in a web browser.

[jasonbahl]

Web browsers aren't the only thing interacting with WordPress content. . .we also have iOS and Android apps that should be able to have knowldedge of what blocks exist so they can create an interface for interacting with Gutenblocks as well.

[johnbillion]

Some work has started on this in #4841.

[johnbillion]

This issue is fundamental to the future of WordPress. I don't say that lightly - Gutenberg is paving the way for a new WordPress, yet its available block types are completely siloed and are not exposed to anything except the client-side Gutenberg editor in a web browser.

WordPress is more than a text editor in a web browser.

We're in beta 2 of WordPress 5.0 and this low-level issue still stands. I'm concerned that this is a late, breaking change, but I stand by my recommendation that block type registration must be accompanied by a server-side call to register_block_type() in order to expose available block types to other parts of both WordPress itself and its ecosystem.

Feedback appreciated.

[mtias]

@johnbillion I agree this is crucial for the future of WordPress. It's been difficult to pinpoint exactly how we should approach it if we consider all the use cases that need to be balanced and their tradeoffs, and I'm leaning on making it a focus for 5.1 instead so we have a bit more perspective and use-cases to make the right calls.

Perhaps a middle ground would be to register the core library of blocks both on the server and the client so block-library can continue to exist as an npm package while exposing all core blocks on the server.

Some questions that have surfaced when attempting to implement this:

• How are blocks going to be distributed going forwards? (.org directory placement, wp-admin screen for available blocks).
• How are blocks going to be discovered? (From the inserter / library.)
• How are dependencies going to be structured? - blocks and plugin relationships.
• How would it impact other CMSs trying to use Gutenberg packages directly?
• How would it impact consumption of block-library from @wordpress/block-library?
• How would it impact developer experience (including both ease of registration and ease of access)?
• How would we make these accessible for native registration on mobile depending on the apps strategy?

My latest thinking on it was to have a .json entry point per block that would be automatically consumed by the PHP block APIs and make that the entry point for WordPress while still allowing direct consumption of the json + JS in a client-only context if needed — which would be super useful for automated testing as well, or in-browser block editor tools. However, this has a non trivial amount of overhead for registering a block (a json + JS, or json + JS + PHP) and might not feel immediately native to WordPress. The fact we are just starting to see the different use cases also makes me pause a bit (couldn't have anticipated interest from Drupal community, for example). The native mobile app project is also in its infancy but it should provide us a better reference within a few months time.

Outside of the higher level infrastructure issues, there are also some specific details around the APIs:

• How much is configuration vs behaviour in a block definition?
• Where do attribute specification belong considering some attributes require save specifications for being fully valid?
• How do we specify icons (like SVGs) server-side?
• Do we treat implementations (edit, save) as contingent?
• If we use a manifest kind of file as a definition point, how do we handle translations?

This is an important enough space that I think needs to be properly modeled (and could use as much help from everyone interested) both in a way that serves current needs and paves the way for what we want to accomplish.

That said, if you have specific suggestions on how it could be approached now, I'd love to hear it.

cc @nb @youknowriad @aduth @mcsf @gziolo

[m]

Definitely concur this is a ripe area for exploration in future releases, and could be a good fit for 5.1 in March or April if work and discussion continues on it now. It would pair well with blocks existing outside of posts and pages.

[jasonbahl]

As far as approaches go, I think it might be worth looking at how Advanced Custom Fields is approaching registering new Gutenberg blocks: https://www.advancedcustomfields.com/blog/acf-5-8-introducing-acf-blocks-for-gutenberg/

They expose an acf_register_block PHP method that allows you to register a block on the server. Then, you register fields the same way you would for other contexts for ACF, this time just setting the Context to the block.

Then you register your render_callback.

These are all Server-Side APIs so blocks are fully exposed to the server, so they can be exposed in WP CLI, the REST API, AND the Gutenberg JS client.

Not sure it's the ideal way to approach this for core, but worth taking some inspiration from.

By having a rich server-side Block registry, Blocks become useful to all clients that interact with WordPress, whether it be CLI, REST, the Gutenberg client, other page builders UI's that approach the design and UX of content building differently than the Gutenberg client app, but still want to use core APIs, etc.

Also worth checking out Gutenberg Fields Middleware (https://github.com/rtCamp/gutenberg-fields-middleware) to see how they approach things, as you can register blocks 💯 on the server without a single line of custom JS. It sends up your server-side registered blocks to the client and allows Gutenberg to use that registry and build out a UI for interacting with the blocks.

With this approach as well, since blocks are registered fully on the server, I can interact with blocks via the Gutenberg JS client app, CLI, REST, WPGraphQL, etc.

A good server-side API will enable all clients to know what blocks are and how to interact with them.

[gziolo]

Let me add a few words to what @mtias said.

We did some explorations (#5802) in the past how we could move attributes to the server and even allow to filter them, so plugins could have a better control over it. It's all doable and can still continue to work with the existing JS API.

There was also another exploration (#5652) to use JSON files to share definitions between JS and PHP which I found very promising myself. Example structur:

{
  "name": "core/categories",
  "category": "widgets",
  "supports": {
	"html": false
  },
  "attributes": {
	"showPostCounts": {
	  "type": "boolean",
	  "default": false
	},
	"displayAsDropdown": {
	  "type": "boolean",
	  "default": false
	},
	"showHierarchy": {
	  "type": "boolean",
	  "default": false
	},
	"align": {
	  "type": "string"
	}
  },
  "render_callback": "render_block_core_categories"
}

I think @mtias has covered all issues, but let me emphasize the most important ones:

• how to ensure that you can set a default value on both client and server which depends on external state (database, i18n, etc)
• by default, JSON doesn't work with i18n or support dynamic values
• using JSON definition and glob for discovery could greatly simplify many things, you might not need PHP files in many cases but it's magic, so could be harder to get at first glance
• icons can take only strings, which doesn't play nice with SVG

[nb]

The spectrum of use-cases is so wide – both of the current implementation and those mentioned in this issue and in #2751 (especially those by @jasonbahl) – that no technical solution will work well for all of them. Keeping everything on the client limits interoperability, using the server as source of truth for more than simple metadata forces us to use a lot more complex (and possibly limiting) ways to declare the block structure and logic, and splitting between both brings both confusion and extra deployment hardship.

I don't know what’s the better trade-off, though I’d argue that before users and developers start playing with Gutenberg more, nobody would be able to tell what are the use-cases that need reinforcing and what sounds cool, but not many people want. As we see more experiments like the ACF one and a lot more after 5.0 is released, we will be able to bring back the best approaches that touch the widest audience in core, while maintaining a smaller and simpler API until then.

[gziolo]

This proposal would also allow some nice performance wins for websites which have many blocks registered. It would pave the way to introduce solutions which would allow async loading for blocks which @ockham requested yesterday in #12232 and I filed over a year ago in #2768. This would give us an amazing opportunity to defer loading some chunks of the client code until it's really necessary. It's a common practice these days and supporting it out of the box would be a big win for all plugin developers.

[johnbillion]

Some movement on this: #21065

[mcsf]

Some movement on this: #21065

With this one now merged, can we consider the present issue closed?

[gziolo]

We still didn’t expose Embed blocks through API and context related properties need to be exposed but overall I think everything is ready.

[spacedmonkey]

There is now a core patch ready for 5.5 - WordPress/wordpress-develop#317

[johnbillion]

We also have https://core.trac.wordpress.org/ticket/50263 now.

[gziolo]

There are still some follow up tasks left though to have all core block fully exposed:

• translatable properties aren't included in block.json and therefore they aren't present in the block types endpoint, see Block API: Add i18n support to block.json metadata file #23636, it still needs its own ticket in Track
• embed blocks need to be consolidated into one block type with multiple block type variations (similar to Social Link block) and it needs to have block.json introduced, this item still needs its own issue and ticket in Track

Update: issue for Embed block is here: #22660.

[gziolo]

Everything is ready to be released in WordPress 5.5 with the exception of integration of Embed blocks with REST API as mentioned in my previous comment. We can close this issue. Let's work on fixes and improvements in follow up tasks.

One thing that might be interesting to explore is how to filter the list of blocks based on post type or post id. @spacedmonkey, is that possible as of today maybe? Well, I don't know if that should be the responsibility of the block types endpoint. It might be something that should be sourced from another endpoint that exposes editor settings defined on the server.

[johnbillion]

Thanks for your work on this everyone