Household load areas and demand profiles

[gplssm]

Load areas of household sector and related peak load/annual demand area identified

• add destatis_zensus_household_per_ha to schema: society
• add automatic downloader for Zensus2011_Personen.csv and h0_profiles.h5 from Next Cloud
• add Zensus2011_Personen.csv to db as it's not used elsewhere, we skip this
• add h0_profiles.h5 to db better we load them from .h5-file
• direct db.engine() to egon-data.configuration.yaml in local db-folder
• replace numbers for OR and OO in mapping_people_in_households
• Search through all FIXME and TODO in code Discuss last FIXME note
• Add function to calculate aggregate HH demand time series for each MV grid district
• Add HH demand per MV grid to eTraGo PF tables
• Remove bug that leads to missing data in table demand.household_electricity_profiles_in_census_cells
• Add dataset documentation
  • Gentle introduction for external people
  • Use module docstring and reference to function docstrings for details (in particular regarding implementation details)
  • Focus on table demand.household_electricity_profiles_in_census_cells as central result, also mentioning MV profiles (and load area peak load)
• Remove tables used for checking data with geometries
• Clean-up code and add missing documentation
• Black & isort
• Add note to CHANGELOG

[nailend]

the following 2 files need to be added manually to eGon-data/src/egon/data/processing/hh_demand/

Zensus2011_Personen.csv
h0_profiles.h5

You can find them here: https://next.rl-institut.de/f/895985

[nailend]

table destatis_zensus_household_per_ha does not exist in db anymore after new update...

[gplssm]

After generating profiles aggregated per MV grid, the comparison between annual HH electricity demand in Schleswig-Holstein by demandregio and calculated on zensus cell level reveals very similar numbers:

DemandRegio in GWh	eGon-data in GWh
4,153.66	4,152.99

Hence, 0.016 % deviation. I'm fine with that. How about you @nailend ?

[nailend]

Hence, 0.016 % deviation. I'm fine with that. How about you @nailend ?

I think this was an old and outdated comment anyways as we scale the egon-data annual-sum at nuts3 level with demand-regio-data. I would consider the deviation as rounding error

[gplssm]

Regarding the insertion of household electricity demand time series at HV-MV level into eTraGo tables, I have a few questions @ClaraBuettner

I had at look on the two tables grid.egon_pf_hv_load and grid.egon_pf_hv_load_timeseries. I guess HH demand time series go into the latter. I understand in grid.egon_pf_hv_load there are buses linked to loads. The load_ids are reference in grid.egon_pf_hv_load_timeseries.
Now, I would assume I have to add HH demand to existing demand in these tables. Questions:

1. There is one single load per bus for all demand sectors (HH, CTS, industry, ..), right?
2. As I don't see data in the time series table (in branch where I'm working), I don't understand the data structure to the last detail. Do you plan to store arrays in the columns p_set and q_set in the sense that one row represents the time series for one year for one scenario, load and version?
3. Related to Q 1: how should the demand time series from different sectors should be summed up? Shall we store the sectoral demand in a temporary/intermediate table, sum the up with pandas and insert into eTraGo tables?

Feel free to reply with less text if possible 😄

[ClaraBuettner]

1. There is one single load per bus for all demand sectors (HH, CTS, industry, ..), right?

2. As I don't see data in the time series table (in branch where I'm working), I don't understand the data structure to the last detail. Do you plan to store arrays in the columns p_set and q_set in the sense that one row represents the time series for one year for one scenario, load and version?

Both is correct. In case of CTS demands, currently only p_sets are stored. (I don't think that there are q_sets in demandRegio). eTraGo also has a function to generate q_sets based on p_sets and a cos(phi). So if we can not create q_sets for any sector we could also do this there.
The timeseries data is created on the dev branch, it looks like this, the p_set column is an array of 8760 values:

3. Related to Q 1: how should the demand time series from different sectors should be summed up? Shall we store the sectoral demand in a temporary/intermediate table, sum the up with pandas and insert into eTraGo tables?

Sounds good. Maybe we can create 3 tables demand.egon_etrago_electricity_households, demand.egon_etrago_electricity_cts, demand.egon_etrago_electricity_industry which store the p_sets (q_sets), scenario and the bus_id?
The load_id could be added in the aggregation script for the final table.

[gplssm]

1. There is one single load per bus for all demand sectors (HH, CTS, industry, ..), right?

2. As I don't see data in the time series table (in branch where I'm working), I don't understand the data structure to the last detail. Do you plan to store arrays in the columns p_set and q_set in the sense that one row represents the time series for one year for one scenario, load and version?

Both is correct. In case of CTS demands, currently only p_sets are stored. (I don't think that there are q_sets in demandRegio). eTraGo also has a function to generate q_sets based on p_sets and a cos(phi). So if we can not create q_sets for any sector we could also do this there.
The timeseries data is created on the dev branch, it looks like this, the p_set column is an array of 8760 values:

3. Related to Q 1: how should the demand time series from different sectors should be summed up? Shall we store the sectoral demand in a temporary/intermediate table, sum the up with pandas and insert into eTraGo tables?

Sounds good. Maybe we can create 3 tables demand.egon_etrago_electricity_households, demand.egon_etrago_electricity_cts, demand.egon_etrago_electricity_industry which store the p_sets (q_sets), scenario and the bus_id?
The load_id could be added in the aggregation script for the final table.

Cool! Thanks for the detailed answer!
I'll create the table demand.egon_etrago_electricity_households using the structure of the existing table for CTS demand. I skip the columns load_id and temp_id for this step, but add the subst_id.