Merge PyPSA-Eur-Sec into PyPSA-Eur

[nworbmot]

We plan to merge PyPSA-Eur-Sec into PyPSA-Eur in March 2023. The PyPSA-Eur-Sec repository would then be deprecated. PyPSA-Eur would include all features from PyPSA-Eur-Sec (additional energy sectors as well as myopic and perfect foresight).

Reasons for merger

• Future energy systems will have enough sector-coupling that modelling electricity separately will be hard to justify
• The workflow is currently very messy, with PyPSA-Eur-Sec removing items (like storage and conventional generators) from PyPSA-Eur and then reading them in in a different way
• PyPSA-Eur-Sec has features that PyPSA-Eur can benefit from (myopic and perfect foresight, new scenario management, existing wind and solar via IRENA database)
• There are confusing duplications between config files
• It is difficult managing the different library dependencies between the two repositories
• Snakemake sub-workflows are deprecated (to be replaced by modules) and anyway hard to work with
• Merging avoids duplication of ancillary scripts (solving, summarising, plotting, scenario management)

Reasons against the merger

• PyPSA-Eur-Sec is more complicated, with more complicated data and library dependencies than PyPSA-Eur, which may put off newbies and casual users; we can mitigate this difficulty by turning off non-electricity sectors by default

Planned steps

• introduce basic CI testing to both PyPSA-Eur and PyPSA-Eur-Sec so we can detect problems (could test inputs but also outputs e.g. for load shedding, energy mix from temporally clustered results, prices, etc.)
• pull perfect foresight code into PyPSA-Eur-Sec (@lisazeyen will do this in February 2023)
• merge the PyPSA-Eur and PyPSA-Eur-Sec snakemake workflows and data with minimal additional changes (perhaps as an initial test to see what goes wrong)
• slowly iron out the discrepancies (adding/removing of components before/after clustering, etc.)
• test it rigorously
• document everything

We estimate it will take 2-3 weeks and plan it for March 2023.

Potential difficulties

• PyPSA-Eur adds components at high resolution (renewables, conventional power plants, load), then clusters, whereas PyPSA-Eur-Sec removes some of these and then adds back components after the clustering; this should be harmonised so that only components that need high resolution are added before clustering (probably renewables, conventional power plants and load are needed, since they are used for clustering in some methods), then rest is added afterwards (e.g. storage that doesn't depend on high resolution and is not used for clustering)
• PyPSA-Eur uses PyPSA's generator component for conventional power plants, whereas PyPSA-Eur-Sec uses links (since need to track fuel that can be fossil or green-hydrogen-based, as well as for CO2 tracking); tentative suggestion: only use links in future
• PyPSA-Eur-Sec has IRENA database for existing wind and solar, whereas PyPSA-Eur has its own methods
• The myopic and perfect foresight in PyPSA-Eur-Sec needs to be "back-ported" to work with electricity-only PyPSA-Eur
• The deduction of heating and industry electricity demand from total electricity demand in PyPSA-Eur-Sec needs to be harmonised with PyPSA-Eur (should be fine since the deduction should only happen if these sectors are chosen in the config)
• PyPSA-Eur-Sec handles CO2 with links and stores (because of CCU/S) whereas PyPSA-Eur uses a global constraint on CO2 (suggestion: use PyPSA-Eur-Sec format)
• The new scenario management needs to work across both
• We need a clever design that simplifies everything by default for new users (e.g. defaults to electricity only and overnight)
• We need to provide a migration path for existing users

[martavp]

I think this makes a lot of sense, the current duplication of scripts and elements in the config file is confusing for new users. We at Aarhus University will organize to contribute or, at least, be test users of the merged model.

[FabianHofmann]

To add on the procedure: Given that we don't want to lose all the contributions to pypsa-eur-sec we should git merge the repositories. Therefore I suggest the following steps:

0. Do preparations in both repos (perfect foresight, Transition to linopy implementation pypsa-eur-sec#282, allows clustering with different build years, other important features)
1. Merge the repos locally with

   git clone [email protected]:PyPSA/pypsa-eur.git
   cd pypsa-eur
   git checkout -b merge-pyspa-eur-sec
   git remote add pypsa-eur-sec [email protected]:PyPSA/pypsa-eur-sec.git
   git fetch pypsa-eur-sec --tags
   git merge --allow-unrelated-histories pypsa-eur-sec/master
   git commit

2. Push to pypsa-eur repo with merge conflicts, open a draft PR
3. List all merge conflicts and address them
4. List all functional conflicts and address them