Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HH load area profile generator #260

Merged
merged 111 commits into from
Sep 7, 2021
Merged
Changes from 1 commit
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
fcaf960
added hh-demand-profile-generator
nailend May 20, 2021
16672b3
add note for missing files #256
nailend May 20, 2021
ef1e667
add TODO for db.engine() #256
nailend May 22, 2021
c0f1a2f
fix type in year
nailend May 25, 2021
5710ce9
Put code in if _name__
May 25, 2021
1ac40fe
add description about data sources concerning files to be downloaded
nailend May 25, 2021
da582bd
add docstrings and remove not used functions
nailend May 25, 2021
0e9a839
add not on hh-people-mapping
nailend May 25, 2021
16a57e2
Download HH load profiles from NextCloud
May 25, 2021
243bd57
Download zensus households data from nextcloud
May 25, 2021
34a4815
Correct table name
May 25, 2021
e9e15bf
add calculation of OO-facotr for people-to-household-mapping
nailend May 25, 2021
9ec6a96
add note
nailend May 25, 2021
fd541ec
Reformat and extend docstrings
May 26, 2021
0f76f66
Merge branch 'features/#256-hh-load-area-profile-generator' of github…
May 26, 2021
311308c
Describe files in download function
Jun 3, 2021
25da37a
Remove unused code
Jun 3, 2021
0a70277
Re-work code and add comments
Jun 4, 2021
2255e64
Move households in census cells mapping into function
Jun 4, 2021
21b7895
Store census cell x household electricity profile mapping in db table
Jun 4, 2021
2dd9e1e
Introduce option to calculate aggregated peak load
Jun 8, 2021
5adb362
Create HH demand profiles for each MV grid
Jun 9, 2021
ff1eeaa
Add functions to check created HH data
Jun 9, 2021
9efc0a9
fix false lable of grid_id and add cell_id to output-table
nailend Jun 10, 2021
14ea586
Change cell_id type to int and cast data before insertation explicitly
Jun 10, 2021
5e0f1d3
Inform about the small deviation introduced by ceiling people to int
Jun 10, 2021
f4f2ef4
Remove TODO note
Jun 10, 2021
71e496b
Black and isort
Jun 10, 2021
30256fc
Add draft of module docstring used as dataset description
Jun 10, 2021
f5f3c8d
change household rounding at cell-level to np.rint
nailend Jun 11, 2021
739998e
fix np.rint
nailend Jun 11, 2021
e55169c
fix np.rint
nailend Jun 11, 2021
17d1be8
add info about household-rounding to docstring of get_cell_demand_met…
nailend Jun 11, 2021
deeb9eb
add info about household-rounding to docstring of get_cell_demand_pro…
nailend Jun 11, 2021
4234188
extend docstring of process_nuts1_zensus_data()
nailend Jun 11, 2021
47d753a
add dataset description in module-docstring
nailend Jun 11, 2021
c98b7f0
Merge branch 'features/#256-hh-load-area-profile-generator' of github…
nailend Jun 11, 2021
36d1c89
Write HV/MV substation HH load profile to database
Jun 15, 2021
8df877e
Revise module docstring for documenting the dataset
Jun 15, 2021
a68e965
Add missing docstrings
Jun 15, 2021
354044c
Remove code for data checking
Jun 15, 2021
6614341
Write MV grid aggregated profiles directly to eTraGo compatible table
Jun 15, 2021
ff1bb08
solve merge conflict
nailend Jun 17, 2021
7ca9012
add hh_demand_profiles to pipeline
nailend Jun 17, 2021
837efcd
add more dependencies
nailend Jun 18, 2021
798499d
remove op_kwarg for mv_HH_electricity_load_2050
nailend Jun 18, 2021
5889e7c
remove hh_demand-profiles.py
nailend Jun 21, 2021
3a11a36
rename hh_demand_profiles_tools.py to hh_demand_profiles.py
nailend Jun 21, 2021
680d7cd
black&isort
nailend Jun 21, 2021
9326f11
add changelog
nailend Jun 21, 2021
e02047a
change import of hh_demand
nailend Jun 21, 2021
b2d8b33
Extent CHANGELOG note
Jun 21, 2021
b57afdb
change implementation of dataset versioning to partial
nailend Jun 24, 2021
c8b2402
black&isort
nailend Jun 24, 2021
b7c52f6
move PythonOperator into pipeline
nailend Jun 24, 2021
ca1c98d
Resolve minor merge conflicts in CHANGELOG and pipeline
nailend Jul 2, 2021
08a2fa9
Move hh_demand_profiles to datasets directory
nailend Jul 13, 2021
53679cc
Clean imports
nailend Jul 13, 2021
49811f7
Fix typo
nailend Jul 13, 2021
1c04d0e
Fix Task-string
nailend Jul 21, 2021
e129433
Merge branch 'dev' into branch features/#256-hh-load-area-profile-ge…
nailend Jul 26, 2021
eac9866
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nailend Jul 26, 2021
09b0575
filter VG250 states - use land only (GF=4) #260
nesnoj Jul 28, 2021
f28f410
use drawing with replacement if pool size < sample size, #260
nesnoj Jul 28, 2021
990edd4
Exclude zensus cell data which is unacceptably modified
nailend Jul 28, 2021
3c34d97
Extend Module-docstring for census data correction
nailend Jul 29, 2021
f6ab4a2
Add create_missing_zensus_data function
nailend Jul 29, 2021
2ea5eb9
Create directory in working directory for downloaded files
nailend Aug 2, 2021
1b22726
Add egon_ prefix to db-table-name and change to singular form #349
nailend Aug 2, 2021
6e0f748
Change etrago table from kWh to MWh
nailend Aug 12, 2021
facddae
Update download url for household_electricity_demand_profiles
nailend Aug 12, 2021
cb46601
Change db import of egon_etrago_electricity_households to pandas.to_s…
nailend Aug 13, 2021
ea0289f
Add function to write iee-hh-demand-profiles to db
nailend Aug 15, 2021
8ef8a2b
Separate hh-demand-profile processing to implement write_hh_profiles_…
nailend Aug 15, 2021
85ac5b9
Rename download functions
nailend Aug 15, 2021
8d62604
Separate zensus data processing into function
nailend Aug 15, 2021
f1e9054
Rename download function and add processing to mv_grid_district_HH_el…
nailend Aug 15, 2021
df9fae6
Update module docstring
nailend Aug 15, 2021
be4a91a
Add docstrings
nailend Aug 15, 2021
7c0a688
Black&Isort
nailend Aug 15, 2021
69c680b
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nailend Aug 15, 2021
b25016d
Update module import names and dependencies
nailend Aug 15, 2021
70986d4
Create task of zensus_inside_ger for dependecies of hh_demand
nailend Aug 16, 2021
59e0675
Restructure code and split functions
nailend Aug 16, 2021
92e31d7
Move Dataset definition to beginning of code
nailend Aug 16, 2021
4e256d5
Change file import to egon-data-bundle for household_electricity demand
nailend Aug 16, 2021
6e0d78a
Add unit descriptions to hh-profiles
nailend Aug 17, 2021
6d86717
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nesnoj Aug 18, 2021
52ece7e
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nesnoj Aug 18, 2021
ebd84ef
Merge branch 'features/#256-hh-load-area-profile-generator' of github…
nailend Aug 19, 2021
fb0ddac
Add code comment for get_load_timeseries()
nailend Aug 19, 2021
7a41903
Update file sources to egon-data-bundle and remove download backup
nailend Aug 19, 2021
ec24a5c
Add functions to retrieve scaled profiles from db
nailend Aug 19, 2021
bfade72
Add minor docstring infos
nailend Aug 19, 2021
e09b769
Fix some pylint issues
nailend Aug 19, 2021
74a73ce
rm return val from HH task
nesnoj Aug 25, 2021
7d5d294
bump version no
nesnoj Aug 25, 2021
2e706cd
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nesnoj Aug 25, 2021
aa3e2d2
Add fixed random_seed
nailend Aug 25, 2021
7a1cdf2
Add autorship
nailend Aug 25, 2021
3d33f05
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nesnoj Aug 27, 2021
2cad501
Obtain global random seed (from CLI param)
nesnoj Aug 27, 2021
c2df97b
Remove re-seeding the random generator on every iteration
nesnoj Aug 27, 2021
63867b5
Remove re-seeding the random generator
nesnoj Aug 27, 2021
08d230b
Seed python's and numpy's random generator on init
nesnoj Aug 27, 2021
53bf6c3
add peak_load_only option to get_scaled_profiles_from_db() #275
nailend Aug 30, 2021
03c6423
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nesnoj Sep 1, 2021
6769675
Set column 'type' as index for table iee_household_load_profiles
nailend Sep 1, 2021
e59f137
Merge branch 'features/#256-hh-load-area-profile-generator' of github…
nailend Sep 1, 2021
b4946c3
Merge branch 'dev' into features/#256-hh-load-area-profile-generator
nesnoj Sep 6, 2021
1fbd656
Fix table column name
nesnoj Sep 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions src/egon/data/datasets/hh_demand_profiles.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,9 @@
Base = declarative_base()


# Set random_seed as long as no global_seed exists (see #351).
RANDOM_SEED = 42

# Define mapping of zensus household categories to eurostat categories
# - Adults living in househould type
# - number of kids not included even if in housholdtype name
Expand Down Expand Up @@ -499,12 +502,12 @@ def create_missing_zensus_data(
if difference > 0:
# add to any row
split = split.round()
random_row = split.sample()
random_row = split.sample(random_state=RANDOM_SEED)
nesnoj marked this conversation as resolved.
Show resolved Hide resolved
split[random_row.index] = random_row + difference
elif difference < 0:
# subtract only from rows > 0
split = split.round()
random_row = split[split > 0].sample()
random_row = split[split > 0].sample(random_state=RANDOM_SEED)
split[random_row.index] = random_row + difference
else:
split = split.round()
Expand Down Expand Up @@ -923,6 +926,7 @@ def get_cell_demand_profile_ids(df_cell, pool_size):
# np.random.default_rng().integers(low=0, high=pool_size[hh_type], size=sq) instead of random.sample
# use random.choices() if with replacement
# list of sample ids per hh_type in cell
random.seed(RANDOM_SEED)
cell_profile_ids = [
(hh_type, random.sample(range(pool_size[hh_type]), k=sq))
if pool_size[hh_type] >= sq
Expand Down