Split descriptions from manifest.json file

[siljamardla]

Current behaviour

During the compile process dbt reads all the YML and makes this one big manifest.json file (and the semantic_manifest.json file, but this is not relevant for the discussion here). For bigger projects this file can become pretty big. We currently have ~3500 dbt models and the manifest file is 75 MB. The compile is slow, the dbt documentation site is extremely slow.

Idea

Split the manifest.json file: keep the rest of the information, but move descriptions to another file.
(Or consider creating even more separate files for different kinds of information)

If we think about the amount of text that goes into the YML file, then for well documented projects we can see descriptions being a major part of the file. And we want projects to be well documented.

Also, the usage of descriptions is different from the usage of the rest of the information in the file. Descriptions are not necessary to check the integrity of the project. They are also not necessary to run the models, to build or test the project.

If we could make the file that is necessary for running the project much smaller, we could speed up:

• the development process (only compile the manifest, without docs)
• the CI/CD testing process (only compile the manifest to verify project integrity and run the necessary models)
• the production process (only compile the manifest, without docs, in every production job that we have)
  The documentation part would be compiled only once, on PR merge. And separately for previews of documentation in the dev process, if necessary.

The documentation site logic could also be updated: start displaying the site and all the nodes based on the smaller manifest file, only read the appropriate part of the descriptions file on demand, if a person opens a specific page). At the moment we need to download a ~75MB file just to open the landing page of the dbt docs.

[siljamardla]

Actually, another very big component can be the SQL that is also in the manifest.
Especially if it is a result of a MetricFlow query (which I know is currently a workaround but might be the future for many use cases).