introduce hierarchical agglomeratice clustering (hac)

[martacki]

Changes proposed in this Pull Request

...story continues...

Introduce hierarchical agglomeraticve clustering method into pypsa-eur, now that it is (finally) included in PyPSA PyPSA/PyPSA#289.

(additional) changes:

1. simplify now by default aggregates buses that are no substations.
2. clustering now defaults to the new method "hac", if not specified for kmeans.
3. clustering method in simplify_network can be different from the method used in cluster_network.

Locally the function(s) worked fine.

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
• Changes in configuration options are added in all of config.default.yaml, config.tutorial.yaml, and test/config.test1.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes.

[fneum]

Looking good!

So HAC will be the new default? Could you highlight this in a release note? It's quite a significant change.

Two maps with clustered regions (~200 nodes) with previous k-means default and new HAC defaults would also very nicely complement this PR. Do you have something like that at hand?

[martacki]

thanks @fneum for suggestions and improvements for the code, should all be in.

However, I want to make a change to the network "topology" sort-of and change two country-labels of n.buses. I'll do this in a new PR (somewhat cleaner), but the new PR should be in before this one.
HAC aggregates only connected nodes, but there are two nodes (one in HR, substation "Dakovo" and one in LU, substation "Vianden") that have no connections to other nodes within their own country. This throws a warning for HAC, because we do the clustering within each country seperately. It doesn't break HAC, but if HR/LU get assigned 2 nodes, this particlular one will never be aggregated and this is not the best partition.

I'll add some figs and definitely a note for the release_notes as well!

[martacki]

Ok, the following commit is rather ugly...

We can just remove the commit if we decide we don't want that "feature", but accounting for it results in a better representation of the grid topology of the network after clustering.

Here's a map extract for better reference.

[martacki]

and, as requested, here are some plots for comparison (k-meas vs. hac-time) for different spatial resolutions:

hac-time+kmeans_50.pdf
hac-time+kmeans_100.pdf
hac-time+kmeans_150.pdf

[fneum]

Looks good! But I still had a few comments and suggestions for improvements. Could you also add a release note with two or three sentences how to use it and what to keep in mind?

[fneum]

Happy to merge this.

Although I've been thinking about whether it would be wise to first introduce this new feature and then make it a default in the version after the next. This would just mean changing the default config from hac to kmeans. What do you think @FabianHofmann @martacki?

@martacki could you also add a few lines and instructions to the release notes?

[martacki]

Happy to merge this.

Although I've been thinking about whether it would be wise to first introduce this new feature and then make it a default in the version after the next. This would just mean changing the default config from hac to kmeans. What do you think @FabianHofmann @martacki?

@martacki could you also add a few lines and instructions to the release notes?

Yes I agree we shouldn't change default in the same go as we introduce the new clustering, just to be on the safer side. Changed to old default.