Add getLastStateChange, getLastStateUpdate, and getLastState to GenericItem

[jimtng]

These methods are handy in many situations and isn't possible with mapdb persistence, so previously it would require something like influxdb.

It also solves the issue of needing to wait until persistence write operation is finished (if it was at all set up to persist on change), so this gives us a cheap and reliable / dependable way of getting the last change / last update time.

See discussions in
https://community.openhab.org/t/persistence-on-off-transitions-with-influx/156271/
https://community.openhab.org/t/blockly-persistence-last-changed-returns-no-result-quite-often/157842

[jimtng]

ping @mherwege @rkoshak

[rkoshak]

I like the idea of this information being part of the Item. Of course it won't go back beyond OH startup but something is better than nothing. It certainly handles this use car better than persistence I think given all the different ways perspective can be configured.

How does this interact with restoreOnStartup? If an item is restored and not changed does this return the restored time or null? What makes the most sense? I'm not sure I know (it's close to bed time here so maybe I'll sleep on it).

[jimtng]

How does this interact with restoreOnStartup? If an item is restored and not changed does this return the restored time or null?

It is not aware of persistence restoration, so I guess once persistence restores the state, it will return the time the state was restored.

[spacemanspiff2007]

Imho it should be possible to query these values from the db so restore on startup should work as expected.
Will these fields also be part of the RestAPI / Websockets response?

[jimtng]

Imho it should be possible to query these values from the db so restore on startup should work as expected.

@spacemanspiff2007 would you mind elaborating this please?

Will these fields also be part of the RestAPI / Websockets response?

Good point. I will add them there too.

[spacemanspiff2007]

It's possible to query the last persisted state from persistence and subsequently restore this state as the current state on startup. It's also possible to query the persisted state before that.
Subsequently it should be possible (imho) to get all the values from the persistence service so restore on startup should also work for these fields as expected.

[jimtng]

It's possible to query the last persisted state from persistence and subsequently restore this state as the current state on startup.

Yes, isn't this what restoreOnStartup does?

It's also possible to query the persisted state before that.
Subsequently it should be possible (imho) to get all the values from the persistence service so restore on startup should also work for these fields as expected.

I don't quite understand the above statements.

This PR doesn't interfere with persistence's restoreOnStartup. If I understand @rkoshak's question correctly, it is a question of whether getPreviousState should return (java) null as is the case when there was no previous state being set on the item, or whether it should return UnDefType.NULL because that's the item's state before restoration. Currently it's the latter.

[spacemanspiff2007]

I think it's a misunderstanding.
All I am saying is that it's technically possible to restore these new fields from persistence so imho it makes sense to do so during restoreOnStartup.

[jimtng]

@spacemanspiff2007 I've added it to the REST response

{
  "link": "http://192.168.1.10:8484/rest/items/TestSwitch1",
  "state": "ON",
  "previousState": "NULL",
  "lastUpdate": 1723872081231,
  "lastChange": 1723871965049,
  "editable": false,
  "type": "Switch",
  "name": "TestSwitch1",
  "tags": [],
  "groupNames": [
    "TestSwitches"
  ]
}

[mherwege]

All I am saying is that it's technically possible to restore these new fields from persistence so imho it makes sense to do so during restoreOnStartup.

That depends on the persistence service. And it is not the case for mapDB, the most used persistence service for restoreOnStartup.

I would argue that, when the item state is updated as a consequence of restoreOnStartup, it should not update these lastChange and lastUpdate fields because now is probably the wrong time. But I am not sure this is feasible. I doubt the item knows anything about the source of the update/change.

In general, I think having the extra fields is a good idea. I just feel like we are still not there to give an answer to a simple question: when did my item last change?
If I use this new function (lastChange):

1. Item did changed since startup -> OK, correct answer, and better then what I could get from persistence as persistence could miss the everyChange strategy or could have compressed
2. Item did not change since start -> now, which is wrong
3. If I can figure out the answer is wrong and I have a persistence service with everyChange: call lastChange on the service ->
   a. Current state equal to last state in DB -> correct answer from persistence (ignoring for now impact of compression)
   b. Current state different from last state in DB -> no answer (null)
   c. If no answer, you could return now in the calling rule, although in this case the startup time might be a more appropriate approximation

In an ideal world, mapDB would just store the lastChange time of an item with the state and we could restore that together with the state. I am just not sure it is easily feasible.
Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup, would it make sense to remove restoreOnStartup from persistence all together and make it part of the core infrastructure, just to take care of this? I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

We can keep adding layers to this, but there are always issues and exceptions. I like the idea of keeping lastChange and lastUpdate with the item. But I think the real simplification comes when we have a simpler restore mechanism on startup. Because than, you always get the right answer. Other persistence services then have the focus on data for graphing and calculations, which makes sense in my view.

[jimtng]

OK, I think I now understand what @spacemanspiff2007 was alluding to, thanks to @mherwege's explanation.

Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup, would it make sense to remove restoreOnStartup from persistence all together and make it part of the core infrastructure, just to take care of this? I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

Not a bad idea.

[rkoshak]

In an ideal world, mapDB would just store the lastChange time of an item with the state and we could restore that together with the state. I am just not sure it is easily feasible.

Why would this not be feasible? I beleive all the apis already exists in persistence to get the time from persistence. Once you have that it's just a matter of adding the last change and last update time as a parameter on the Item.

Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup,

I'm not sure we can make that case that's it's always only used for restoreOnStartup. It's useable if you only care about the most recent change to the Item in a rule (given it's current default strategies). One could change the strategies and it would work for most recent update.

But if these properties are added to the Item and we go down a path where those are made available and accurate on the the Item this use case is handled.

I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

Be careful here.

Not all uses want all Items restored.

Not all users want all Items restored to their most recent change or update (e.g they use a rule with .persist to control when the state gets saved).

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

[spacemanspiff2007]

That depends on the persistence service. And it is not the case for mapDB, the most used persistence service for restoreOnStartup.

It would be trivial to modify mapDB persistence service to save the additional three values. A quick glance at the docs shows it should be possible. If not it's always possible to name mangle additional values with a separator that's not a valid item name (e.g. MyItem#lastUpdate)
The question is - as you correctly pointed out - if that's something that results in a meaningful behavior.

2. Item did not change since start -> now, which is wrong

No - it should return null because it's undefined.

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

I strongly agree. For many devices that report the state themselves I explicitly do not save the item state so I don't have the wrong state after startup. If the restore makes sense depends strongly on the device, the user and how the rules are written.
I for once was struggling with a rule and strange startup behavior during testing and the root cause was rrd4j restoring states during startup which was automatically configured (rrd4j).

---

Just a hint:
HABApp already provides the last_update and last_change datetimes. I deliberately chose not to make them null to simplify the rules so the users don't have to always check for that. Instead I chose to initialize them with the timestamp when the item gets created.

That would also result in a proper behavior when restoring from persistence.

Time 0 - item creation:
  state: NULL
  previousState: NULL
  lastUpdate: 0,
  lastChange: 0,
...
Time 10 - persistence restore:
  state: "asdf"
  previousState: NULL
  lastUpdate: 10,
  lastChange: 10,

So I am basically suggesting the opposite of what I wrote above 🙈 🤣

[jimtng]

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

Each item can have a special metadata restoreOnStartup=true. When this is on a group item, persist all its direct members.

[mherwege]

Each item can have a special metadata restoreOnStartup=true. When this is on a group item, persist all its direct members.

I think, what @rkoshak also implied is that one may want to use different rules for persisting values, not just on every change, but e.g. persist it once and always use that same value at startup. And that would mean we also need to support that part of the configuration in core. And then we are back to doing it the way it is done now anyway.

Thinking further about it:

• Changing the restoreOnStartup logic to also restore lastChange, lastUpdate and previousState at startup is fairly easy to do. The method to change is

  openhab-core/bundles/org.openhab.core.persistence/src/main/java/org/openhab/core/persistence/internal/PersistenceManagerImpl.java

  Line 521 in 506ccd4

  private void restoreItemStateIfPossible(Item item) {

  . You may need some extra methods (or an extended setState method that has extra parameters) in GenericItem to also set these fields, although it should probably only be used for restoreOnStartup.
• Many persistence services can already provide the extra info required. MapDB cannot and should be extended to store and retrieve the extra information. I don't know how easy that would be to do. We also seem to be using a very old version of MapDB, so I don't know how much impact that has on feasability.

With the above logic in place, it also would make sense to revisit the persistence extension actions to not return null, but return what is actually in the database, and document it as a breaking change, advising users to use the Item methods for this use case.

[mherwege]

• Many persistence services can already provide the extra info required. MapDB cannot and should be extended to store and retrieve the extra information. I don't know how easy that would be to do. We also seem to be using a very old version of MapDB, so I don't know how much impact that has on feasability.

Looking at MapDB code, it is actually not that difficult. The item state and time are already stored as a JSON string. Adding extra fields should not be a problem.

[rkoshak]

I think, what @rkoshak also implied is that one may want to use different rules for persisting values, not just on every change, but e.g. persist it once and always use that same value at startup. And that would mean we also need to support that part of the configuration in core. And then we are back to doing it the way it is done now anyway.

Since a user needs a rule for calling .persist, if there's also way to turn off the restoreOnStartup for that Item in this new core feature, maybe that would be sufficient as they could create a system started rule to restore the Item's state manually from whatever persistence they are using instead of this new core one. Or maybe we can add a persistence extension to restore the state with the most recent value in the database that could be called from a rule in place of the restoreOnStartup strategy that would be removed.

This new core service doesn't need to do all the use cases. We just need to ensure that all the use cases are covered somehow.

With #4324 I think we will reliably be able to count on persistence being there by runlevel 100. So maybe a system started rule with previousState would be sufficient. I don't know though, I haven't fully explored all the edge cases that can crop up due to timing and such.

2. Item did not change since start -> now, which is wrong

No - it should return null because it's undefined.

Maybe we are thinking too hard about this.

What do we intend these new properties to represent? I see two options:

1. reflect the changes/updates made to the Item
2. refelct the changes/updates to the device

If it's just 1, we don't really need to worry about most of these complexities. If an Item doesn't change after OH startup, getLastChange will be the time when the Item was created and set from restoreOnStartup and getPreviousState would be NULL. If you want the actual device information and not just information about the Item during this run session of OH, then you need to use the existing persistence capabilities. But this also means we don't need a new restoreOnStartup service in core or any of that complexity and the implementation becomes pretty simple.

If we want to make 2 happen then it gets much more complicated, as we can see and we probably need something built into core.

But do we really need to solve 2 here? Maybe 1 is sufficient? I think users who prefer not to use persistence at all would love to have 1.

We also need to thing of second tier impacts like the increase in writes for those running on SD cards and the like if this core restoreOnStartup feature becomes the default behavior.

[jimtng]

So what's the next step?

• Changing the restoreOnStartup logic to also restore lastChange, lastUpdate and previousState at startup is fairly easy to do. The method to change is

  openhab-core/bundles/org.openhab.core.persistence/src/main/java/org/openhab/core/persistence/internal/PersistenceManagerImpl.java

  Line 521 in 506ccd4

  private void restoreItemStateIfPossible(Item item) {

  
  . You may need some extra methods (or an extended setState method that has extra parameters) in GenericItem to also set these fields, although it should probably only be used for restoreOnStartup.

If we are going to proceed in this direction, I would suggest:

• First merge this PR without the extra setState / setters for lastChange, lastUpdate etc.
• If and when the integration with the persistence extensions is added, then add the setters as you suggested. This way, it's clearer how the setters are implemented according to how it's actually used.

[mherwege]

LGTM

[mherwege]

If we are going to proceed in this direction, I would suggest:

• First merge this PR without the extra setState / setters for lastChange, lastUpdate etc.
• If and when the integration with the persistence extensions is added, then add the setters as you suggested. This way, it's clearer how the setters are implemented according to how it's actually used.

I am fine with this.

[spacemanspiff2007]

I think this is the best way, too.
Maybe it makes sense to make persistence of lastChange etc. configurable. This could and should be discussed in the according PR.
Do the the new values also get reported in the corresponding item events?

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/lastupdate-and-null-values/158134/6

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-monitor-if-item-is-frequently-updated/158260/6

[jimtng]

how do we resolve the access to these methods vs the persistence methods with the same name within rulesDSL?

[mherwege]

how do we resolve the access to these methods vs the persistence methods with the same name within rulesDSL?

Maybe I miss something here, but what gets called would be determined by the imports I would think. As both items and persistence are imported by default, it indeed leads to a conflict which will force changing the rules to have a fully qualified name.

The options then become:

1. Give these new methods another name: this is fully backward compatible.
2. Change the name of the existing persistence actions to getLastPersistedState, getLastPersistedUpdate and getPreviousPersistedState.

In both cases, rules would work without being changed, but 2 would have a slight change in behaviour. I would still prefer 2 because it is more natural not to call persistence if not needed. If one absolutely wants the value from persistence the rule would need to be changed, but it would be very clear what is expected. At the same time, the persistence actions could be modified again to not return null, but the last effective persisted value.
2 also requires adapting the scripting languages to call these changed methods.

[rkoshak]

t doesn't need to do anything special for backwards compatibility towards older openHAB versions, because the new library will never be used inside an older openhab version. @florian-h05 can you confirm this?

That's no necessarily true. openhab-js can be installed independently so one can have a newer version of openhab-js with an older version of the add-on.

Obviously there are limitations to this as breaking changes happen which limit how far back openhab-js can go. But I beleive the goal is to maintain backwards compatability as much as possible.

In this case it shouldn't be too hard. I've not looked at the code but I would expect something like the following could work:

const lastChange = (PersistenceExtension.lastPersistedChange === undefined) ? PersistenceExtension.lastChange : PersistenceExtension.lastChange;
lastChange(...

[florian-h05]

@jimtng Your outlined impact on JS Scripting is correct, except for what @rkoshak said:

Users can install new openhab-js versions on older openHAB versions, so openhab-js needs to provide backward compatibility, but it should be do-able like in Rich's code snippet.

A more general though though: IMO it feels wrong to rename a PersistenceExtensions method because there will be a new method on the Item with the same name. It sounds a bit awkward to have PersistenceExtensions::getLastPersistedChange as the persistence word is already part of the class name.

I however really love the new functionality provided by this PR - the architecture has to be decided by the core maintainers.

[jimtng]

A more general though though: IMO it feels wrong to rename a PersistenceExtensions method because there will be a new method on the Item with the same name. It sounds a bit awkward to have PersistenceExtensions::getLastPersistedChange as the persistence word is already part of the class name.

I agree with this. The change of name is purely for the benefit of rulesdsl. Is there another way to solve it?

[florian-h05]

Probably have a wrapper class of the persistence extensions for rules DSL so rules DSL can have non-colliding names.

[mherwege]

Probably have a wrapper class of the persistence extensions for rules DSL so rules DSL can have non-colliding names.

Do you know a way to do that without breaking all persistence extensions? At the moment, they are just called with itemName.methodName syntax, whereby itemName is the first argument and this is standard Xtend. And of course that forces use of unique names with item methods or there would be a name conflict (both persistence extensions and methods on items are imported by default). It would have been cleaner to be explicit from the start in the call about using persistence, but it is not. And I don’t like the idea to introduce something different just for these methods.

Everything is solved if we use another name for the new methods, different from what is proposed. But I don’t know anything that is clearer. And in DSL context, it would make a lot of sense to rename the old methods on persistence to make the difference in usage very clear and obvious. Doing so even allows old DSL rules to run without change as they will get the same info from the item rather than persistence.

[jimtng]

Here's an idea: how about removing them from persistence.

[florian-h05]

And in DSL context, it would make a lot of sense to rename the old methods on persistence to make the difference in usage very clear and obvious. Doing so even allows old DSL rules to run without change as they will get the same info from the item rather than persistence.

My proposal with the wrapper class was targeting at that. I should probably have stated clearer what I meant to say.

I my proposal is to leave the PersistenceExtensions as they are, i.e. not rename methods, and introduce a new class just for rules DSL that just maps its method calls to the PersistenceExtensions. This way we decouple the method naming in rules DSL from the PersistenceExtensions.

[mherwege]

I my proposal is to leave the PersistenceExtensions as they are, i.e. not rename methods, and introduce a new class just for rules DSL that just maps its method calls to the PersistenceExtensions. This way we decouple the method naming in rules DSL from the PersistenceExtensions.

Sorry, it is probably me, but I still don’t get it. Instead of using the PersistenceExtensions class it would be possible to use a wrapper class that calls the methods in PersistenceExtensions. It would need to duplicate all of the static methods in PersistenceExtensions with calls to these (and most are only there for DSL, all the parameter variations). That new wrapper class would then be used for DSL. The methods would then be renamed for DSL in this wrapper class to avoid the naming conflict caused by Xtend. Is this what you mean?

And if I understand it well (I don’t use jruby myself), just like in DSL and unlike JS, persistence methods are directly on the item in jruby. Also there, it is not immediately clear where the response comes from if there is some overlap in the function. So something similar needs to be put in place there.
This leaves us with renaming in a complex way for DSL and jruby anyway. So why not strive for consistent method naming that also clearly conveys what it does and make it explicit in the name when the response will come from persistence, at the same time encouraging to use the item property and not use persistence to get this response in most cases?
Again, I am not happy myself in changing names of existing methods, but I do think that in this case, the advantages may outweigh the inconvenience.

[mherwege]

Here's an idea: how about removing them from persistence.

That’s very drastic and completely ignores use cases where it might actually be useful to get this info from persistence. Although I must admit that all use cases I have will be served much better with the new item level methods and I will not use the PersistenceExtension methods for this.

Dropping or renaming will have the same effect.
In rules DSL or jruby, this will not require a change to the code (if we change the PersistenceExtensions method name, or use a wrapper class, the effect will be the same). For JS (which I use), because it makes the call to a persistence extension method explicit through a separate object, it will always require a change in the script from using the persistence method to using the item property, independent of the solution chosen for this.

[jimtng]

Whilst in JRuby the PersistenceExtension methods are attached directly to the item object, this is done selectively and explicitly by the helper library. It is easy to ensure that the call goes to the GenericItem instead of PersistenceExtensions and it can also rename the methods from PersistenceExtensions if needed. So there's zero issues with JRuby.

With my proposal for dropping the methods from PersistenceExtensions:

• It would not affect DSL and it would eliminate the ambiguity
• No problems with JRuby, although we'll make adjustments in the helper library to deal with it. Zero impacts to user scripts.
• JSScripting would need to add a compatibility layer in ItemPersistence.lastChange to call item.getLastChange for >= OH4.3, with a deprecation warning, and call PersistenceExtensions.lastChange for < OH4.3. This is what JRuby library would do too.

Sure, we'll lose the functionality from persistence, but would anyone really miss them?

A wrapper class cannot remove the need for the various permutations of arguments for every method, because that would be API breaking. However, It would double the maintenance chore for PersistenceExtensions, which is already huge as it is, as I'm sure @mherwege is well aware.

Dropping/renaming them would also be API-breaking, I guess.

[mherwege]

Sure, we'll lose the functionality from persistence, but would anyone really miss them?

The one thing we would need to solve at the same time then is to restore lastChange, lastUpdate and previousState from persistence at startup. I know how to do it, and it is possible to enhance mapDB to support it. In that case, I don't see a disadvantage going that way.

I was already working on a PR to change the persistence extension methods. One thing I stumbled upon is that we do not only have lastChange, lastUpdate and previousState methods, but also nextChange, nextUpdate and nextState methods. While we could keep the naming for these as is, it would not be symetrical. And that makes it harder than necessary to 'know' what the method name should be. I therefore intended to create the symetrical names and deprecate (without removing) the old ones. But it is another set of methods in these Persistence Extensions to maintain, and it is huge enough as it is.

I would like some input from a core maintainer here. Would it be OK to drop some methods from persistence if they get replaced by equivalent item methods offering more consistent results? We believe the API changes will be minimal, no impact on DSL, and can be hidden from the end users by changing the jruby and JS helper libraries.

[jimtng]

I've just realised, it's not that simple, because we have variants of these methods that take an extra argument for serviceId.

In light of this, perhaps the easiest avenue is to rename the new GenericItem methods. It would avoid all compatibility problems. The only catch is, to benefit from the new methods, users would need to modify their scripts.

However, I haven't got an idea what the new method names should be.

[jimtng]

Proposal for naming the GenericItem methods in this PR: getLastStateChange, getLastStateUpdate, and getLastState.

[mherwege]

I've just realised, it's not that simple, because we have variants of these methods that take an extra argument for serviceId.

Indeed. But then again, the reason to sometime do that is to force use of another persistence service to get these values if we know the default service (like e.g. RDB4J) does not provide accurate enough info. If it is being used that way, I would argue it would still be better to change the script and use the item method. The helper methods could ignore serviceId. The Persistence Extensions could actually also call the item methods and ignore the serviceId in that case. I would prefer it over making the name of the new methods more complex.

[mherwege]

And there is another complicating factor. The PersistenceExtensions previousState methods return a HistoricItem, not a State. Changing that will force changes in DSL rules anyway.

[jimtng]

So do you agree that renaming the new methods is a better idea?

[mherwege]

So do you agree that renaming the new methods is a better idea?

Yes, I do in the end.

[jimtng]

@mherwege Perhaps with no naming changes in persistence, at least now instead of returning null, the persistence method can call the Item's method to fulfil the request when such data is not available from the persistence service.

Or another strategy is to try Item's method first, and if it's null (when it was first loaded and still UNDEF), then it can query the persistence service.

This will solve the issues linked in the first post above, without requiring users to change their existing scripts!

[mherwege]

@mherwege Perhaps with no naming changes in persistence, at least now instead of returning null, the persistence method can call the Item's method to fulfil the request when such data is not available from the persistence service.

Or another strategy is to try Item's method first, and if it's null (when it was first loaded and still UNDEF), then it can query the persistence service.

This will solve the issues linked in the first post above, without requiring users to change their existing scripts!

Agreed. That makes sense.

[jimtng]

I've renamed the methods to getLastStateChange, getLastStateUpdate, and getLastState to avoid conflicts with the PersistenceExtensions methods. There should be no issues going forward now, I hope.

I don't think these values need to be "restored on startup" like persistence. If people need that restoration feature, they can call the equivalent PersistenceExtensions method.

I am still not 100% satisfied with the naming, but I can't think of any better names.

[andrewfg]

not 100% satisfied with the naming, but I can't think of any better names

You could simply call the original ones 'xyz' and the new ones 'xyz2' ...

[mherwege]

I don't think these values need to be "restored on startup" like persistence. If people need that restoration feature, they can call the equivalent PersistenceExtensions method.

Not required, true. But that means a user has to explicitely create startup rules if they want to do this, or have a check in every rule if the values are there to retrieve from persistence if not. So at some point I would like to have it.

I am still not 100% satisfied with the naming, but I can't think of any better names.

Same for me. That's part of the reason why renaming in the PersistenceExtensions class was attractive. It then clearly conveyed the purpose. Putting '2' behind the name does not solve that either. By lack of better idea, I think what you propose is fine.

[mherwege]

I have created:

• Persistence restore lastState and lastStateChange on startup #4463: makes it possible to restore lastState, lastStateUpdate and lastStateChange on startup
• [mapdb] Store and restore lastState, lastStateChange and lastChangeUpdate openhab-addons#17820: enhanced mapDB to store these values

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/ideas-and-discussion-what-features-do-you-want-in-openhab-5-0/160573/161