Persistence Extensions: Riemann Sums

[mherwege]

Closes #4439

At the moment, it is not possible to easily calculate the integral value of the curve represented by persisted values. This is useful for calculating e.g. total energy consumed (in kWh) from instantaneous power meter readings (in W).

The current sum persistence actions calculate a sum without considering the time dimension, and therefore are only useful when we have a guaranteed constant interval between persisted values for this.

The average actions do use a time weighting in their calculation. Multiplying the average with the total duration considered would give an approximation. However, the average calculations assume a constant value over the bucket and take the value at the start of the bucket. Using this average to calculate an integration would also mean an extra calculation (already divided in the action code and multiplied again in the rule).
An alternative approach to get an integral value is by setting up an integrated item and a rule that adds value * duration since previous change/upate, triggered on base item change. This works and is flexible in its calculation (value can be approximated based on previous and new item state). It suffers from things like unexpected shutdowns, where the base item and its aggretation may run out of sync.

This PR proposes to create a group of new actions, riemannSum , that will calculate the RiemannSum as an approximation for the integral value. Analoguous to the other actions, there are variants taking (or not) startDate, endDate and serviceId as input.
There is one extra key parameter, the Riemann type, representing the type of approximation used. Valid values are:

• RiemannType.left: takes the persisted value at the start of the bucket to represent the value for the whole bucket. This is most useful when there is a persistOnChange strategy and the values represent a step function. An example would be dynamic electricity rates, as they will effectively be constant inside the bucket.
• RiemannType.right: takes the persisted value at the end of the bucket.
• RiemannType.trapezoidal: takes the average of the persisted value at the start end the end of the bucket, effectively making a linear interpolation to fit the curve. This type is most useful when the real values change continuously. It can be used for any persistence strategy and any interval.
• RiemannType.midpoint: uses 3 persisted values and uses the middle of the values as an approximation for the value halfway in the interval between the middle of point 1 and 2 and the middle of point 2 and 3. This is the best approximation when the real values change continuously, the persistence intervals are short and the bucket sizes between persisted values are relatively constant.

The existing average, variance and deviation actions have been enhanced to use the results of the Riemann sum calculations. They now have variants with an extra parameter where this type can be selected.
For backward compatibility and consistency between defaults of the different actions, the action versions for riemannSum, average, variance and deviation without a Riemann type parameter all use RiemannType.left as a default.

Tests have been enhanced end confirmed the existing actions with the existing parameters still behave as before.

If this PR is accepted the scripting libraries should be enhanced to cover these new actions and the new parameters in the existing actions.

For testing, a compiled jar can be downloaded from here.

[mherwege]

Tests run fine locally. I am trying to figure out why they do not work in the PR.

[mherwege]

I put in some extra test logging to try to analyze where the difference is between my succeeding local tests and the failing online tests.

[mherwege]

All test are passing now and extra logging has been removed. This is ready for review.

[andrewfg]

Shouldn't these enums be in ALL_CAPS ??

[mherwege]

Yes, they should be. I will change it.

[mherwege]

Shouldn't these enums be in ALL_CAPS ??

Changed

[jimtng]

This class is getting very cumbersome with all the parameter permutations!

[mherwege]

This class is getting very cumbersome with all the parameter permutations!

Totally agree. Do you have any suggestions? As it is all about static methods in one class, it is not easy to split without introducing a lot of breaking changes. The private methods could move to one or more utility classes, but I am not sure this will make maintaining it any easier.

[jimtng]

This comment applies to all the other similar methods.

Whilst implementing this into the jruby helper library, I came across one particular issue.

Could you please make the type parameter @Nullable, and provide the default (i.e. RiemannType.LEFT) inside the internalXxxxXxxx methods, and pass null by the callers? This would:

• DRY and collect all the defaults into one place (in the internalXxxx method) rather than in the callers
• Make it easy for the helper libraries to also call it with null to indicate "default", and therefore letting core decides what the default is.

As it is now, I'd either have to also hard-code this default in the helper library, thus maintaining a duplicate default, or conditionally call one of the overloaded methods that doesn't take the type argument in order to use the "default" type.

[mherwege]

Yes, I will do that. Makes sense.

[mherwege]

Done

[mherwege]

@florian-h05 I was looking into making these new methods available in JS. I don’t know how to properly handle the RiemannType enum arguments. Does a Java enum work properly in GraalVM JS scripting? Or should I be using string arguments in this PR? That may make it harder as well as the serviceId argument is also a string and will make it impossible to distinguish in varargs.

[florian-h05]

I would expose the RiemannType enum as a static member of the ItemPersistence class, I think that should work (even though enums do not natively exist in JS).