Project: make grapher and site standalone projects within repo

[larsyencken]

Make grapher a standalone project within the existing repo.

Scope

• Consider moving grapher config type and dependencies up into a separate project at the beginning of the dependency chain so that our clientUtils can use the type definition for a grapher config (or move them to clientUtils)
• Separate grapher and the OWID site into separate standalone projects in the same repo #1157
• Package grapher separately to the OWID site #1667
• Make a style interface for grapher
• Create a default style for grapher that differs from OWID's, but with an override for our site
• Document how to develop, build and consume grapher on its own
• Publish grapher to npm
• (Optional) Come up with a release method for grapher

Why / why now

Currently people can’t try out or use grapher without spinning up our entire site. This project would allow others to use grapher, find problems with it and contribute fixes back.

This year we would like to start to seek external contributions for data and for grapher.

Risks

• Could lose the git history for one project or the other (solved via monorepo)
• Could end up with a development flow that is way tricker/harder/more expensive when we make grapher changes as part of a combined change (solved via monorepo)
• Other people creating grapher charts could dilute our brand, if they look too similar (solved via styling)
• Risk to have to maintain backwards compatibility for other people 😱 which could slow our development
  Set low expectations for the general public to begin with in our README
• If our module boundaries aren’t currently a good match, it could blow out in time because of refactors that are needed

[marcelgerber]

There are two way we can go about this:

1. Go with a clean split, i.e. split the project up into as many modules as we can
2. Go with the cheapest possible split, i.e. only split the grapher off from the rest (since the grapher is certainly the most-requested reusable component)

I made an effort to identify some possible modules and their interdependences:

• grapher (reusable)
• db / api (?)
• admin-backend
• admin-frontend
• baker (static site generator)
• site (other frontend code for site, e.g. styles, cookie banner, footnotes, ...)
• wordpress (?)
• utils (?)
• types

In the below diagram, dashed lines indicate a "consumes API" relationship.

graph TD;
    grapher;
    db/api;
    admin-backend;
    admin-frontend;
    baker;
    site;
    wordpress;
    utils;

    utils --> grapher;
    utils --> admin-frontend;
    utils --> admin-backend;
    grapher --> admin-frontend;
    grapher --> site;
    grapher --> baker;
    admin-backend -.-> admin-frontend;
    db/api -.-> admin-backend;
    wordpress -.-> baker;

Loading

Plan

Try structuring out utils first, to see how well that works and which problems we hit.
Try to do the "nice" solution first (structuring out into ~8 packages), if that doesn't work out then go with the minimal solution (utils, grapher, everything else).

We will also want to rework our current tsc + webpack setup, by instead using a TS plugin for Webpack (or ts-node for node scripts) so TS compilation happens transparently.

Default styling of a reused grapher should include:

• No logo
• Different fonts
• Different default color scheme (?)
• Replace "CC BY" by "Powered by ourworldindata"

[larsyencken]

Solid plan!

So, in the ideal scenario, the owid-grapher repo would end up as a monorepo containing README + 8 folders, each of which is an independent project with its own README, versioning and packaging?

I could see the versioning being annoying, if for everything other than grapher and utils we only care about latest. I had imagined that maybe the grapher admin might become standalone, but in practice I think it will be quite coupled to our MySQL backend, especially with bulk FASTT being part of the admin.

After the move, we could consider whether we want to in time break more things out, or whether we want to do the opposite (e.g. move owid-content into the monorepo rather than keeping it separate).

[larsyencken]

Closing this, we don't have the appetite to address it for now.