Integrate Energi Data Service (Danish)

[jlaur]

In order to make data from Energi Data Service easily accessible in openHAB, I would like to create an add-on implementing this API.

The first milestone will have focus on Elspot Prices and add a channel being updated with the current electricity price (per kilowatt hour). This was immediately achieved when starting the work on this add-on today:

Data Filters

Filters were based on https://github.com/MTrab/energidataservice/blob/master/custom_components/energidataservice/tariffs/energidataservice/chargeowners.py

They have all been optimized and tested, and issues based on findings were created in this repo.

Plan

This is the public TODO (in headline form) in order to reach "MVP" and publish a pull request:

• Expose future prices as advanced channel with JSON array (since I'm not aware of any neat way of doing this currently).
• Implement smarter retry policy on failures (e.g. exponential backoff and jitter).
• Add support for tariffs (dataset DatahubPricelist).
• Provide pre-made filters for known grid companies.
• Add channel configuration for overriding pre-made filters.
• Add grid companies to option list.
• Add channels for Energinet (TSO) tariffs.
• Add action for calculating price for specified consumption in a period. See https://community.openhab.org/t/dishwasher-price-calculation-automation/139207.
• Load/keep retrospective data, e.g. 12 hours back in time. This would make it possible to write rules giving feedback on price when an appliance has ended a program.
• Documentation cleanup.
• Code cleanup (and a few more unit tests).

Status

Draft code is here (work in progress):
https://github.com/jlaur/openhab-addons/tree/14222-energidataservice/bundles/org.openhab.binding.energidataservice

Features

• Optimized service calls: Spot prices are retrieved once per day and tariffs only upon expiry (usually months/years).
• Error handing: Policies for retry strategies are in place to make sure data will be retrieved as soon as possible after failed calls. This includes exponential back-off with jitter.
• Channel configuration for adding VAT to prices based on openHAB regional settings.
• Grid company is selectable in configuration.
• Pre-configured filters are included for all known grid companies.
• Filters can be overridden by channel configuration.
• Spot prices and all tariffs are supported.
• Properties shows number of remaining calls as well as timestamp of last call.
• Actions for getting prices and performing calculations.

Testing

• org.openhab.binding.energidataservice-4.0.0-SNAPSHOT.jar (latest)
• org.openhab.binding.energidataservice-3.4.2-SNAPSHOT.jar (backported 27.01.2023, last updated 08.02.2023 21:53)

Screenshots

Properties and Configuration

Channels

Channel configuration

Total price

Related to openhab/openhab-core#3478

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/bringing-electricity-information-from-eloverblik-dk-and-energidataservice-dk-into-openhab/143470/4

[b-r-y]

Hi Jacob,

I tried this and all works as expected. Few comments/questions:

1. Any reason not to add the VAT to the price before exposing? Did you perhaps want without on purpose?
2. What can one use the future data json string for? Can this be plotted within Openhab somehow?

I am no good at java programming but i can provide proper filters for the DatahubPricelist dataset so i think your binding can very quickly replace my horrid scripting.

Cheers.

[b-r-y]

For the plotting i have done this simple pull of the ElSpotPrices in grafana. But adding the VAT alone seems at the limit of what grafana can handle. So i think the binding exposing an interface with math already done will be much more powerful. On windy days transport can be the dominant part of the cost so looking only at the electricity prices can be misleading.

[jlaur]

1. Any reason not to add the VAT to the price before exposing? Did you perhaps want without on purpose?

I'm trying to keep it simple. Calculating VAT is another (cross-API) concern. In Denmark it's 25%, but the binding can also be used from Sweden and Norway. Additionally, when adding tariffs and fees on top of the spot prices, it would be best to add VAT as last step. I'm not saying VAT can't become relevant, but for now I left it out on purpose to expose the raw data. It can easily be added by a proxy item or other rule logic.

2. What can one use the future data json string for? Can this be plotted within Openhab somehow?

For me that's the most interesting part since it allows me to calculate cheapest timeslot for running appliances in the future, i.e. planning energy consumption. See the dishwasher example.

I am no good at java programming but i can provide proper filters for the DatahubPricelist dataset so i think your binding can very quickly replace my horrid scripting.

I would be very interested in some tips on getting the correct tariffs for my area. I was looking into the DatahubPricelist dataset yesterday, but didn't find a proper way of navigating it. Issues I had:

• The date filtering seems broken? It seems you have to supply both start and end date, and you will then only get results fully within the boundaries of this period? I would expect being able to get results valid at a specified day (e.g. today).
• I didn't find a way to look up ChargeOwner in a sensible way. I'm not even sure it's ChargeOwner I should be looking for. And I see different names like "N1 A/S - 344". In other words: I'm completely lost here trying to find the tariffs that are relevant to me (and other users eventually).

[b-r-y]

You are right that the DatahubPricelist dataset is huge - it includes many companies and each one can supply many different customers with different tariffs. The easiest way to figure out how to fetch the ones for you is to look how eloverblik.dk has done it. If you log in there you can download a price list and that list also has important information for your specific supplier. For example i have N1 as well and in the downloaded list you can see that it has a charge owner (Ejer) which maps to the GLN field in the database. For N1 where i am it is 5790001089030. So applying that filter goes from 170+K to 4K+ records. Then you need the tariff type. In the downloaded list this is the name (navn) which maps to the Note in the dataset. For most private customers this is Nettarif C. Applying this filter as well narrows it down to only 10 records, which are different only in the validity period - and there will only be one currently valid period. This is why the time filters you apply don't seems to work - they don't really search for the charge now but rather for validity time of the charge. So if you set the time to now, it does not find anything - now is not a start time of the validity period. In a binding it should be easy to fetch all periods and match the current one to select the correct charge. You will also notice that if you use these filters you get N1 A/S - 131. The 131 might be something regional. The state charges come under GLN 5790000432752 so you can find the Elafgift, Systemtarif and Transmissions nettarif.

For the N1 transport tarif the query at my place would be:
https://api.energidataservice.dk/dataset/DatahubPricelist?offset=0&filter=%7B%22GLN_Number%22:[%225790001089030%22],%22Note%22:[%22Nettarif%20C%22]%7D&sort=ValidFrom%20DESC&timezone=dk
With the currently valid record being 1.

I can extract the rest of the rules later today. There are some tricks when it comes to the discounts. You might see Rabat på nettarif N1 A/S in your prices file from eloverblik.dk. I currently get 0 but before Christmas, it was full discount so transport was technically free (values were given as negative numbers cancelling out the other records). This can change so should be included in the calculations.

[jlaur]

@b-r-y - thanks for this valuable information! I think the best we can do right now, at least for this binding, will be providing some instructions for getting the EAN of the owner like you described. I happen to have the same as you and see the 10 rows. I think this could be mapped into some channels containing current prices, as well as being integrated into the future prices JSON, so all information would be at hand here.

Regarding the date filters - I have the same understanding as you. But it means they are useless. They should either provide the filter as one lookup date or a period - and then match all validity periods intersecting with that date or period. In the current implementation you can't use the filters, so the payload is 10 times bigger than needed for no reason.

Example:

• Start: 01.01.2018
• End: 31.12.2025

The only way to have this row included in your results would be providing start date as <= 01.01.2018 and end date as >= 31.12.2025. But since you don't know the dates, you can't do that. I will probably contact them about this - unless you think I'm missing something?

[b-r-y]

Agree on the time filters - you don't know them and can't know them before parsing. I am not sure it matters in a binding though. Payload of 10 records doesn't sound that bad to me and finding the matching period is also somewhat easy. But perhaps for other cases the size can be problematic. For the nettarifs one can assume semi-static behavior. I think once you grab the one for the current period you can rely on it for quite some time. At least a full day. Some have end date so this can be months - but not always - I've seen without end date so scheduling the next update according to that wont always work.

For the other fields here is some guidance, which i just realized is somewhat N1 specific.

• nettarif by the netselskab (N1) - you can use ChargeTypeCode=CD --> same effect as Note=Nettarif C AND GLN=5790001089030 for N1
• for the discount you can use ChargeTypeCode=CD R (note the space between D and R)
• In the discount case above the ChargeTypeCode is uniquely identifying this but GLN will bring extra robustness.

This is the tricky part really. These seem to fit only N1. It seems to me some fields are subject to operator input so they probably wont work for others. As i have no special knowledge on the topic and have simply spent too much time reverse engineering it, i think our best hope is to simply allow people to set it up for themselves as a parameter and over time build a list of options. After all there are about 20 netselskaber in Denmark so it should be fine. It does hurt the generality you were going for though...

The government ones are easier:

• Systemtarif --> Use Note=Systemtarif and GLN=5790000432752
• Elafgift --> Use Note=Elafgift and GLN=5790000432752
• Transmissions nettarif --> Use Note=Transmissions nettarif and GLN=5790000432752
  And then the of course the trouble with the validity filtering of course.

Finally you can also extract the subscription costs for the netselskab. These are fixed costs and don't depend on usage. They are typically charged every three months but appear in the database as monthly. For the example of N1. the filters are:

• Net abo C forbrug--> Use Note=Net abo C forbrug and GLN=5790001089030
• Rabat på abonnement N1 A/S --> ChargeTypeCode=CDR (note the lack of space between D and R!!!!)

The Elspot prices are really the easy part but it is also worth pointing out that those are the market prices at which the elleverandør buys. So in reality the price to customers for any given one hour slot will be some 5 - 10 øre higher. At the moment of writing this post the spot price for DK1 region is 1,23 DKK with VAT, while my elleverandør (Norlys) charges me 1,2707 DKK with VAT. You can see this difference if you use the generic Watts.dk app vs an elleverandør like Norlys app. At this moment i have not found a way to pull any one specific elleverandør's hourly prices.

Finally for fixed electricity prices customers... well none of this matters. The price is fixed to whatever the contract from the elleverandør says and you don't care how much goes to who i guess.

NOTE for the non-Danish speakers (i don't know precise terms in English to differentiate the different types of companies doing different things so one can get electricity, therefore i mix the danish terms... sorry):

• ellevarandør - the electricity company that you call for your electricity. They make the bill, provide support etc. Most people only talk to them. They buy and sell electricity from the grid on your behalf. They make money by selling you electricity at just a bit more than they bought it for - a retailer really.
• netselskab - the company that digs the cables into the ground and wires up the houses. They own the cables and make money from subscriptions to keep you connected to the grid and by charging transport fees.
• the Danish state - they operate the big super powered highways of electricity distribution, interconnect with other countries and of course charge for that + some tax.

[jlaur]

@b-r-y - thanks a lot for the verbose explanation. With that I was able to progress a lot yesterday. I have net tariff working now, just need some fine-tuning, unit tests and integrating this into the future prices JSON. I hope to finish this part in the evening so I can push the changes.

I need to take care with the naming (in English). For now I called the CD one just "tariff", but realize now that this is inaccurate, since there are multiple tariffs.

For stuff like discounts possibly having codes specific to the grid company I'm considering if it should be possible to create dynamic channels, so the charge type code could be configurable. This really needs to be considered carefully. On one hand the binding shouldn't be bloated and having dependencies to specific companies. On the other hand it should be as simple as possible to use. So I probably need at little bit more understanding and digging into data from other grid companies before finally deciding how to best model this into openHAB.

The next steps adding fees/tariffs from Energinet should be fairly easy after yesterday's work. I believe the government tariffs/fees you mention are actually Energinet (EAN 5790000432752)? Terminology is key here, don't want to mess this up and confuse both myself and future users. 🙂

[b-r-y]

@jlaur writing java code is not for me but I can review. Discussions on implementation, naming and so on might be best directly on a PR. I can review such things if you point me as to where.

5790000432752 is indeed the state it looks. Not sure if it can change. I was thinking that the channels can have custom fields to be added to the query and if the query fails for some reason you get a 0. Would that not be easier than custom channels? Or may be that's what you meant?

[jlaur]

@b-r-y - I've just pushed my latest changes and updated the JAR. I'm filtering net tariffs by ChargeTypeCode "CD" and "CD R", which works pretty well for N1, so only one net tariff channel is exposed which will automatically take the discount into consideration.

I'm now working my way through the cross-GLN dataset with start=2023-01-01, filtered by charge type=D03 and selecting only columns ValidFrom,ValidTo,GLN_Number,ChargeTypeCode,Note. On first thought this would mean one of two options:

• Dynamic channels which can be added and tailored to any needs. Most possible flexibility, but also hardest for users to set up. Also, since channels would be fully customized there wouldn't be any common model which could be used for calculations etc.
• Static channels with configuration options. For example, currentNetTariff which has a dedicated meaning. The "CD" and "CD R" could then be configurable (like in the HTTP binding).

I'm leaning most towards static channels with some configuration options. But before going in that direction I'll try to make some more sense out of the full dataset across GLN's, so the default configuration will work for most users. So far I found also charge type code "C" which seems to be similar. So maybe the particular codes we are interested in all start with the letter "C". Since we can't filter by wilcard ("C*"), it might be a solution to build a list of all possible codes in current dataset (e.g. "C", "CD", "CD R"). This would work straight out of the box as long as there are no overlaps. The numbers can then be summarized (e.g. net tariff = "C" + "CD" + "CD R" (again, assuming no overlaps, so since we don't have C, we'll just get the sum of the two others which corresponds to our actual tariff). The configuration could then fill in the gaps in case new codes would appear.

I'm having a friend check the GLN of his company which is not N1. I'll use that as first verification of my findings to see if this kind of generic algorithm could work.

[b-r-y]

I did some digging and reading ...

I'm leaning most towards static channels with some configuration options.

I think this would be better mostly because of my current understanding of how this works. Which leads me to

I'll try to make some more sense out of the full dataset across GLN's, so the default configuration will work for most users.

I don't think this will work without a user generated list of values. I went through these documents:

1. https://energinet.dk/media/k2yo4lpc/brs-guide-forretningsprocesser-for-det-danske-elmarked.pdf
2. https://energinet.dk/media/wp5j4hq2/tillg-til-brs-guide.pdf
3. https://energinet.dk/media/qnklh1d0/rsm-guide-edi-transaktioner-for-det-danske-elmarked-ver-5-7-7.pdf

My summary understanding is that these describe how energidataservice.dk works in synch with other EU services (https://www.entsoe.eu/). To play the game you need to be an actor. Templates for the various actors and their actions are described in the interface specifications (third bullet) and can be browsed here: https://energinet.dk/media/hbblxc0u/skemaer-og-wsdl-filer-dh-2-0.zip

My conclusion is that the ChargeTypeCode can only be D01/D02/D03 as defined in 3. Yet this is displayed in the interface we deal with as ChargeType only and the actual ChargeTypeCode we see is in fact the internal PartyChargeTypeID. PartyChargeTypeID is in defined as 10 character whatever the actor (netselskab in this case) wants to put in. So having in mind there are some 20 such entities in DK, it should be relatively painless over time to build this user database. Yet it can all change at any point...

So i am starting to think going directly for eloverblik.dk might be better. As can be seen from 3. the back end interface has people's names and CPR numbers so i can see how this is not public. I think eloverblik.dk does the filtering based on the back end data with more info than we are dealing with. I think Watts.dk has access only to eloverblik.dk + the energidataservice.dk but not the back end and that is why the prices are not elleverandor specific.

Have a look at these few highlighted aspects and see what you think. I will try the new version in the meantime.

[jlaur]

@b-r-y - thanks, I'll have a look at your links tomorrow. See also the updated issue description where I added a Data Filter chapter which contains a link to a JSON file from Home Assistant which is essentially a database of filters for the different grid companies.

I agree that using eloverblik.dk will be simpler because it has the link to the user, so can connect the dots. This may be the next project, right now I'm determined to make some sense out of Energi Data Service. 🙂 It has not only disadvantages: It's actually quite fast and allows quite many calls. And of course, it's simpler to start using because it doesn't require authentication. So having both would be ideal, and users can pick whatever they want from each binding.

Regarding ChargeType you are right - it's even documented here:
https://www.energidataservice.dk/tso-electricity/DatahubPricelist#metadata-info

and mapped here:
https://github.com/jlaur/openhab-addons/blob/14222-energidataservice/bundles/org.openhab.binding.energidataservice/src/main/java/org/openhab/binding/energidataservice/internal/api/ChargeType.java#L24-L26

[jlaur]

@b-r-y - please note, I haven't read your linked documents yet. 🙂 But here's a summary of my current ideas after yesterday's data digging session:

• Users knowing their grid company should be able to configure this without having to log in on eloverblik.dk and downloading price data.
• In this case we need to deduce GLN from name of grid company/area. But it must also be possible to configure GLN of a company not on our static list. Simplest solution would be an option list for the gridCompanyGLN configuration parameter, so that GLN would be the key and name of company would be seen by the user. However, I'm not sure if it's possible to have it as a input field with selection options. If not, we would either need two parameters, which would not be elegant - or list the companies in the README and have only the GLN input field. Also not ideal. I'll figure out what's possible here, but at least from a data perspective it's quite settled that we need to store GLN in configuration.
• Next, we need to provide the correct filters for that GLN. Here I think a static filter list will suffice - we can base it on own findings as well as the Home Assistant JSON.
• Since the grid companies can change filters anytime, and since new ones can appear, there must be a manual override. This will ensure that the community can help with the correct configuration if something is suddenly broken by such changes, or if the built-in filters are wrong. This can be provided as channel configuration: note and chargeTypeCode. If they are left blank, the built-in filters for the GLN will be used. Otherwise the user-configured criteria will be used.

[jlaur]

I have implemented the above and pushed source as well as new JAR. Filters are implemented for all known grid companies. Only thing missing for net tariff configuration is a nice way of selecting appropriate grid company:

In this case we need to deduce GLN from name of grid company/area. But it must also be possible to configure GLN of a company not on our static list. Simplest solution would be an option list for the gridCompanyGLN configuration parameter, so that GLN would be the key and name of company would be seen by the user. However, I'm not sure if it's possible to have it as a input field with selection options. If not, we would either need two parameters, which would not be elegant - or list the companies in the README and have only the GLN input field. Also not ideal. I'll figure out what's possible here, but at least from a data perspective it's quite settled that we need to store GLN in configuration.

This describes the issue in more detail:
https://community.openhab.org/t/configuration-parameter-options/143818

[jlaur]

I have some doubts about the channel configuration "Include VAT" - it doesn't feel right. It means that these factors can influence the items linked to the channels:

• Enabling/disabling the parameter itself.
• Switching country in regional settings.
• VAT rate update in the code, for example adding support for more countries. Could also be actual changes in VAT percentage, which would require code changes and be unsynchronized with the openHAB distribution/upgrade.

Additionally, some channels can be configured with VAT and others without, leading to inconsistency. All of this is made worse if such items are persisted, since it would break history when any change is made.

Furthermore, the channel configuration affects all linked items. So you can't have one item without VAT and another one with VAT.

I would prefer "raw" values excluding VAT, and then let users configure transformations for multiplying by 1.25, for instance. This could be configured on a per-item basis, and could be done in a sum group channel for display purposes while keeping the individual channels without VAT. For the user it would require more configuration, but it would be more flexible.

I'll look into possibilities here. I know there is a simple math transformation in SmartHome/J, but I don't think it exists in openHAB. Perhaps some inline JavaScript transformation would be possible. If I can find a simple standard way of doing this, which could be documented, I will probably remove the "Include VAT" channel configuration again.

@cweitkamp - are you the creator of the math transformation? What would you think about bringing it to openHAB?

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/entsoe-e-binding-for-nordpool-spot-prices/143833/4

[jlaur]

I have now sent a mail to Energinet and suggested an improvement to the date filters (see #14222 (comment)).

[jlaur]

I have now sent a mail to Energinet and suggested an improvement to the date filters (see #14222 (comment)).

They confirmed the issues and let me know that they are restructuring the DatahubPriceList dataset (without any ETA). In the meantime I was able to optimize and individualize all filters, so they are working correctly and fetching the least amount of data possible.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/dishwasher-price-calculation-automation/139207/1

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/dishwasher-price-calculation-automation/139207/4

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/bringing-electricity-information-from-eloverblik-dk-and-energidataservice-dk-into-openhab/143470/11