Add defaultUnit metadata for NumberItem

[J-N-K]

Closes #3247

Before merging we should discuss if we remove the "derive unit from state description" code. Looking at discussion here, it seems that the majority would prefer to keep item-unit and representation (state description) completely separate. I'm fine with that, but it might be breaking for existing installations.

Signed-off-by: Jan N. Klug [email protected]

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/uom-default-units-and-consequences/142360/53

[spacemanspiff2007]

Thanks for your quick implementation.
Would it be possible to use a dedicated public nullable variable on the NumberItem instead pushing this into the metadata?
UoM is only supported there so this only makes sense in the context of the NumberItem.

I made some other suggestions in the issue which I currently do not see addressed in this PR.
Will these be addressed later and this is intended more as a quick fix?

[J-N-K]

I decided to use metadata instead of your approach because it requires no changes to other components. Metadata is also already available in the REST API.

[spacemanspiff2007]

I get that it's easier to implement with metadata but this brings lots of other metadata related issues (see some of the issues I've created concerning metadata).

This is more a strategic decision:
If UoM is an openHAB core functionality and openHAB builds on that then it should be implemented in a non-"hacky" way.
Because the time invested to change other components will be saved countless times compared to a quick fix and the issues down the line.
Then unit would be the same as e.g. label or category who both are a core functionality (of an item), too.

[J-N-K]

In general this configuration should not be needed at all. The state description is used for display purposes and values send to an UoM item should have a unit. This is merely for bindings that should be updated to provide good units. Bindings with a generic approach (like MQTT or HTTP) allow configuring units in the channel configuration.

[spacemanspiff2007]

This is merely for bindings that should be updated to provide good units.

There are also devices which change the Units based on its configuration. For the binding maintainer it's impossible to provide a good unit because the device internal configuration might be different per user and the device internal default might be different per country.

Also what you are describing is a unit transformation "on the channel".
But this is not what this issue is about, because the request I raised in the issue is to represent the item.state in a different unit than the system default.
My request is to explicitly set a different unit with scaling on the item so the item state is always represented in that unit (and scale).
Example:

Number:°C    TestTempC     "Temperature [%.1f °C]"     <-- Item unit explicitly set to °C
Number:°F    TestTempF     "Temperature [%.1f °C]"     <-- Item unit explicitly set to °F


Command "20°C" to both items
 -> TestTempC: 20°C
 -> TestTempF: 68°F

Command "77°F" to both items
->  TestTempC: 25°C
 -> TestTempF: 77°F

---

In general this configuration should not be needed at all.

I disagree - this is actually a very important feature and without it the UoM doesn't make (fully) sense at all.
Often it's clever and better when the item state is not in the default unit,
e.g.

• kg/m3 (default) vs μg/m3 (desired) for particle density
• m (default) vs km (desired) for e.g. distance from home
• J (actual default) vs (kWh) (current default) for Energy

While from a pure openHAB internal perspective there isn't much difference but it changes when you interact with the outside world, e.g.:

• APIs
• Databases
• External Charting
• External Rule Engines

There it makes sense to supply the value in a predefined defined unit and this unit very often does not align with the system default. And the supplied value is the item state.

Example:
Let's assume that J is the default for Energy again and not kWh.
I have a device that reports kWh. OpenHAB converts it to J. Now I want to chart in Grafana, but the chard is broken. So I add an outgoing transformation to kWh for Grafana. Then I want to call a script and pass the energy consumption. Again I add an outgoing transformation to kWh. End of the month I want to upload the value to my Energy Provider.
Again I add an outgoing transformation to kWh just so I can make an http call.

Instead of adding the countless transformations it would be much more elegant if I could tell openHAB in which unit the value should be represented. The system default might often be a good fit but just as often it is not.

Since the users know best they should not be patronized and be free to chose the appropriate representation (as in item state representation).

[J-N-K]

Internally a BigDecimal is used, so it shouldn't matter what unit is actually stored (even if you try to express length at atomic scale in km) - as long as it is compatible with the dimension (i.e. you don't try to set a value of 5.0 l to a Number:Length item).

We have four different cases:

1. A binding provides a value including unit (e.g. Netatmo) and is linked to a dimensionless item. The unit is stripped, as dimensionless items only hold a plain number.
2. A binding provides a value including unit (e.g. Netatmo) and is linked to a dimension item. We don't need a default unit here. If the measurement system is different (e.g. your locale provides m but the value is ft), then the value is converted to the unit of you locale. If the unit is in the same measurement system, then nothing is converted (e.g. m and km).
3. A binding provides a value without unit and is linked to a dimensionless item. In this case the value is just set.
4. A binding provides a value without unit and is linked to a dimension item. This is the only case where we need to figure out what unit we need to use. This is then either the defaultUnit introduced here, the state description (debatable if this should be removed) or the system's default unit for that dimension.

All of this has nothing to do with the representation in the UI or external APIs:

• If you need to use the value with an external API, you should convert the value to the unit you need by using .toUnit on the state.
• If you want a different representation in the UI, use the state description (for decoupling reasons I would therefore suggest to remove the state description from the NumberItem.getUnit code).

Persistence should (and actually is) storing and restoring the value in the unit provided by the item's .getUnit method. As long as the precision of the stored value is not limited, there is no need to change that value. If you want to use Grafana or something else to externally access the value, then you can of course set the defaultUnit which makes sure the values are stored in that unit.

I fail to see how we patronize users here: they can do as they like, stick with what we offer or set their own.

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/uom-default-units-and-consequences/142360/72

[ccutrer]

I'm satisfied with this. It provides my core goal - allowing a unit to be defined for all conversion use cases we've discussed, separately from the state description which is the default way UIs will present an item.

As for retaining state description as a source for the item's unit - you're right that it would be cleanest to not. It would be a bit of a breaking change to remove it. I guess I don't have strong feelings either way. I know users tend to dislike breaking changes where they have to take action, but as a developer it's often worth the energy to make the breaking change instead of having to maintain cruft in perpetuity. It just depends on how much cruft it is; this seems like not much.

There also the link to #3132, where one could argue that because state descriptions can come from bindings, they're not just for presentation purposes.

@spacemanspiff2007: can you point to some of the issues you've had with metadata in the past? I know I've had issues with things stored in metadata (like state description!) persisting after the item has been removed, but those all seem to be sorted out now, and well understood. It's pretty widely used now by several features, and most areas (MainUI, .items files, JSR223 scripting languages) are able to present metadata as if it were just part of the item. It's simply an implementation detail that it's stored completely separately.

[spacemanspiff2007]

We still talk about different things and your response doesn't fit to my question.
I'll try again so I hope you'll get a better understanding what I am suggesting.
This issue has nothing to do with units reported by binding or how values are displayed, this is purely an item state representation issue, e.g. how the item state gets represented and scaled internally.

What I suggest:

Number:m     Test_m     "Temperature [%.1f cm]"     <-- Item unit set to meters, GUI representation in cm
Number:km    Test_km     "Temperature [%.1f cm]"    <-- Item unit set to kilometers, GUI representation in cm


Command "5m" to both items (5 Meters)
 -> Test_m : 5m
 -> Test_km: 0.005km

Command "5mi" to both items (5 Miles)
 -> Test_m : 8046.72m
 -> Test_km: 8.04672km

This would allow me as a user to normalize values.
E.g. an item state would always be meters, even though I posted kilometers to the item.
That way I can chose with the item definition e.g. how values will be persisted and external systems will always be scaled correctly.

Currently the value would change from "999 m" to "1 km" and external systems often just ignore the unit (see the J vs kWh persistence issue). That means an external system that expects m would now think it's 1m instead of 1000m.

[ccutrer]

We still talk about different things and your response doesn't fit to my question. I'll try again so I hope you'll get a better understanding what I am suggesting. This issue has nothing to do with units reported by binding or how values are displayed, this is purely an item state representation issue, e.g. how the item state gets represented and scaled internally.

What I suggest:

Number:m     Test_m     "Temperature [%.1f cm]"     <-- Item unit set to meters, GUI representation in cm
Number:km    Test_km     "Temperature [%.1f cm]"    <-- Item unit set to kilometers, GUI representation in cm


Command "5m" to both items (5 Meters)
 -> Test_m : 5m
 -> Test_km: 0.005km

Command "5mi" to both items (5 Miles)
 -> Test_m : 8046.72m
 -> Test_km: 8.04672km

This would allow me as a user to normalize values. E.g. an item state would always be meters, even though I posted kilometers to the item. That way I can chose with the item definition e.g. how values will be persisted and external systems will always be scaled correctly.

Currently the value would change from "999 m" to "1 km" and external systems often just ignore the unit (see the J vs kWh persistence issue). That means an external system that expects m would now think it's 1m instead of 1000m.

Oh yikes, I didn't even know it did that. 95% of the time I work with unit conversions between systems. This definitely does seem like an issue for persistence. But it's a separate issue, not to be addressed by this PR.

[mstormi]

This would allow me as a user to normalize values.

Yes, but there is no convincing point in why a user would want to be doing that.
For display, there's patterns to automatically do it.
And as a programmer, no matter if rules or binding, you are free to apply whatever scale conversion you think you need, at any time.

Offering that capability to users would be a very bad idea.
It will make the number of functional combinations explodes, and with it the potential number of mistakes users will be making in configuring their systems. More choices aren't always better, here's it's the opposite.

Currently the value would change from "999 m" to "1 km"

Why would it ? A counter in some binding is associated with a base unit, and that will not change. 999+1 is 1000 not 1 so you will get 1000 m.

[rkoshak]

linked to a dimensionless item.

It might avoid confusion to use "unitless" instead of dimensionless since we have an actual unit called "Dimensionless".

then you can of course set the defaultUnit which makes sure the values are stored in that unit.

This would be set with separate Item metadata, some other mechanism, or is there already a mechanism I don't know about? Read the thread Rich! It's metadata.

E.g. an item state would always be meters, even though I posted kilometers to the item.
That way I can chose with the item definition e.g. how values will be persisted and external systems will always be scaled correctly.

Maybe now I'm confused. Is this not the defaultUnit @J-N-K mentioned above? Am I misinterpreting what the following means?

If you want to use Grafana or something else to externally access the value, then you can of course set the defaultUnit which makes sure the values are stored in that unit.

[mstormi]

Maybe now I'm confused. Is this not the defaultUnit @J-N-K mentioned above? Am I misinterpreting what the following means?

I think a lot of misunderstanding is about the question what a "unit" is.
m is a unit, foot is another one.
km and miles are NOT units of their own - they're the same as m and feet, they just have scale prefixes.

[spacemanspiff2007]

Yes, but there is no convincing point in why a user would want to be doing that.

Yes - there is. And that is interaction with other systems.
As I stated other systems currently just strip the external unit.
And don't get me started on the RestAPI:
A script that you wrote today will tomorrow be off by factor 1000 because the state representation suddenly change from m to km.
If it's impossible to set normalization and define that e.g. this value will always be in m then it's just a matter of time when things will fail.

Offering that capability to users would be a very bad idea.

I strongly disagree.
On the one hand you accept that openHAB users are clever enough to choose the unit but on the other hand they aren't clever enough to pick an internal normalization which according to you doesn't matter anyway?
Which way is it?
If it doesn't matter then it shouldn't matter if the user preference is e.g. m, km or μm and normalization should be possible.
If it matters then it's even more important that the user can select the appropriate normalization

And a nice side effect to a normalization field would be that for everyone it's very clear how e.g. this value would have to be persisted, passed around through the rest api, etc..

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/uom-default-units-and-consequences/142360/115

[J-N-K]

There is very surprising code in CommunicationManager: If the item ins plain Number and a DecimalType command is sent to a channel with an accepted item-type Number:foo, the CommunicationManager tries to guess the correct unit from the accepted item-type. I don't see any good reason to do so and believe that is failing quite often. I would vote for removal. @openhab/core-maintainers @openhab/add-ons-maintainers, WDYT?

[ccutrer]

Oooh, this might explain how the change to color-temperature-abs's channel type to a Number:Temperature caused bindings to receive QuantityType, even though the Item was still just a Number.

[J-N-K]

@ccutrer I must admit that I was not aware of that, too. And I think it's also very surprising and unnecessary. If a binding knows how to handle a value without unit, it can implement code for DecimalType, if not, it should only implement QuantityType.

[ccutrer]

Agreed. Or the user can add a profile that can do conversions. CommunicationManager shouldn't be adding extra logic here (besides instantiating and applying profiles).

[spacemanspiff2007]

I've given this issue some more thoughts and I still think we should not go with metadata but rather with a field directly on the item.

1. With metadata we use two places where store the same information

Number:Length     MyItemName     "MyLabel [%.1f cm]"  {defaultUnit=m}

With metadata defaultUnit=m the user already specifies that it's a length, there is no need for Number:Length because it can be inferred from defaultUnit.

Nicer would be:

Number:m     MyItemName     "MyLabel [%.1f cm]"

or if the user doesn't care about unit and normalization:

Number:Length     MyItemName     "MyLabel [%.1f cm]"

Then there is one way/value only which defines what happens.

2. With a field on the item there is no lifecycle issue

Define Number:Length with {defaultUnit=m}.
Then change item type to Number:Temperature. Now you have a Temperature Item with defaultUnit=m.
Then make it a String. Now you have a String with a defaultUnit=m
Or define a Number:Temperature with {defaultUnit=°F}.
Then change it to a String.
Then change it back. Now it'll display everything in °F even though you would have wanted the system default.

You have to keep both values in sync, which can lead to confusion and hard to track user and code issues .

Additionally metadata is not loaded together with items, so it might be that the item is already loaded and the metadata is still missing for example on startup (see my other issue to provide the item metadata together with the ItemAddedEvent).
What will happen on restore from persistence? Is it ensured that the metadata is always there especially with fast persistence services such as mapdb or rrd4j.

3. Units and normalization can be directly validated

This can directly be validated by the *.items file parser

Number:m     MyItemName     "MyLabel [%.1f cm]"

While through the GUI it's possible to present the user a dropdown menu for normalization metadata, it's still possible that with the RestAPI/Rules/etc an unsupported or invalid unit can be set.

Or defining a String item with defaultUnit=Cats/m³ and then changing it to a Number:Volume

---

@J-N-K : There has been this pr and other issues. Would you want me to do a write up in a fresh issue how I have envisioned this feature so it's complete in one place. Or do you still think this should be more of a quick fix?

[J-N-K]

I din't think that's the way to go and also very confusing. Items have a type. That type is Number or Dimmer, which define what this item is for. The Number type can be extended by the dimension of the number that is stored. That dimension is Length, Temperature or something else, not a specific unit. Also defaultUnit is optional, in most cases (like Temperature, which is probably the most used dimension) the majority of users will be happy with the system default.

String item's don't use the defaultUnit, it's unused metadata without any effect in that case. If you configure a Number:Length item with a defaultUnit of °F, a warning is logged and the metadata is discarded.

That aside: We don't support (and have never supported) changing item types, for managed things they need to be removed and re-added, and the same happens internally when the type is changed in the file. Additionally, if you delete a managed item, associated metadata is also removed, so the described scenario cannot happen. If someone wants to make everything more complicated than necessary and use the file-based item method, he should take care of removing metadata himself.

Regarding lifecycle: managed:metadata should be added to the start level 20 requirements, then we are safe, since the restore-on-startup happens after the ready marker for that level has been set.

[spacemanspiff2007]

The Number type can be extended by the dimension of the number that is stored. That dimension is Length, Temperature or something else, not a specific unit.

I think that's a misunderstanding:
I still want the dimension of the item be "Length". But the dimension "length" can be derived from the normalization value e.g. m or km. It's like if I say to you "It's 5km" then you immediately know I am talking about a distance. I don't have to say "It's a length and it's 5km".
If the user doesn't care about the normalization it's "Number:Length", if the user cares it's "Number:km".

---

If you configure a Number:Length item with a defaultUnit of °F, a warning is logged and the metadata is discarded.

It should rather be rejected with http status 422 at least for the RestAPI

---

It still think the "Number:Length" syntax in the item type was a mistake and should have never happened because it's not a different type but as you say yourself an extension of the Number Item.
It's an additional information used only by the number item and not a different type.

But if I follow the argumentation that Number:Length is an extension of Number then it should be Number:Length:km since the normalization value is an extension of Number:Length.

---

That aside: We don't support (and have never supported) changing item types, for managed things they need to be removed and re-added, and the same happens internally when the type is changed in the file. Additionally, if you delete a managed item, associated metadata is also removed, so the described scenario cannot happen.

I have a testcase in HABApp which does exactly that:
It changes a Number to a String and then further to a DateTime. This works from the openHAB side without issues.

Additionally, if you delete a managed item, associated metadata is also removed, so the described scenario cannot happen.

Was this introduced in 3.4? The last time I checked (I think it was 3.3 but I am not sure) metadata was kept.

[mstormi]

I'd think another field for scaling ain't necessary and like J-N-K I think it would be very confusing.
You need to think of the (current) unit as a two-component combo: it's a scale prefix plus a real unit.
In kWh the k has nothing to do with the physical unit but only with the value. It is a scale prefix and Wh is the real unit, same with n or k in nm or km.
Whenever the real unit is defined (and it now is with the 3.4 change), setting the value alone is sufficient.
If you always have to additionally read and interpret another field to contain the scale prefix, that will be very confusing, will raise yet more user expectations and, worst of all, will create a new source for errors of various sorts in programming and configuring.

[spacemanspiff2007]

If you always have to additionally read and interpret another field to contain the scale prefix, that will be very confusing, will raise yet more user expectations and, worst of all, will create a new source for errors of various sorts in programming and configuring.

You are not disagreeing but affirming my argument.

If the user has to define unit and scale prefix in different locations it's very confusing and inconsistent.
I'd like openHAB to internally use a field for the unit and scale prefix.
Only then it's possible to build it in a way that there is only one place for the user to define the unit and optionally the scale prefix.

[J-N-K]

If we go that way, then we should remove different item-types completely and only have a metadata unit. If set, that is a dimensional item, if not, then it is a plain number. This can be automatically migrated for managed items but will be a breaking change for textual configuration.

[mstormi]

You are not disagreeing but affirming my argument.

Err, no. I'm against introducing another field.

If the user has to define unit and scale prefix in different locations it's very confusing and inconsistent.

Today a user cannot explicitly define the scale via field, the only means to explicitly set scale today is by setting
value, scale prefix and unit in one go.
Which is absolutely fine as it is the natural, intuitive way therefore scale prefixes should only be settable in this context, never all by themselves, because many users will quickly forget they did.
That'll lead to all sorts of confugsion, interpretation and programming errors for many many users. With almost everybody in fact that does not have advanced individual requirements like yourself.

If we go that way, then we should remove different item-types completely and only have a metadata unit.

I agree with that statement in itself, but totally disagree with that plan. It would mean to needlessly destroy & rebuild a 5yr old established mechanism. You will create pain with and upset a lot of users.

[spacemanspiff2007]

If we go that way, then we should remove different item-types completely and only have a metadata unit. If set, that is a dimensional item, if not, then it is a plain number. This can be automatically migrated for managed items but will be a breaking change for textual configuration.

That would only be consistent and fine with me. 👍
I'll gladly provide a python script that'll do the migration for textual items.

---

That'll lead to all sorts of confugsion, interpretation and programming errors for many many users. Everybody in fact that does not have advanced individual requirements like yourself.

You wrote that the internal state representation and scale doesn't matter for the end user.
So I fail to see how this will result in "all sorts of confusion, interpretation and programming errors for many many users".
Either the internal representation of the item state doesn't matter then it's no problem to set a user defined scale or the internal representation does matter and then it's even more important that the user can pick the scale for the internal representation.

Based on your comment it seems you've not yet understood what the goal and the issue actually is. Please read the whole issue and the linked issues.

[mstormi]

You wrote that the internal state representation and scale doesn't matter for the end user.

That just applies as long as there is no means for the user to affect it
Which there would be if you introduced a scale field. Once it's there it will be used, intentionally or by chance.

[J-N-K]

@rkoshak I would like your opinion here. Would you agree that it would be beneficial to have only „Number“ items and make them handle units if a unit=m metadata is set? I believe if a user configures an item, he has an idea about the unit, much more than a dimension (especially for dimensions like Intensity or VolumetricFlowRate).

[ccutrer]

Just pitching in an additional point here:

"scale" isn't always a simple multiplication or division. My example is mired/mirek, often used in lightbulb color temperatures (more often in calculations and bindings, than user facing). It's an "inverse mega-kelvin", or 1/MK, or MK^-1. I don't have any strong recommendations here, besides that it's natural for a user to want to seamlessly convert between mired and kelvin the same way they would want to seamlessly convert between Celsius and Fahrenheit. As of now, this is implemented as QuantityType.toInvertedUnit(), since the dimension as actually changing (from Temperature to Temperature^-1 or vice versa) and toUnit() cannot change dimension. I'm not sure if this sort of conversion should be handled more generally, but wanted to keep it in people's minds as they're discussing things.

[J-N-K]

@ccutrer That would not be affected by any of these proposals. IMO mired/K is a special case (I know of no other dimension with such units) and already handled well.

[rkoshak]

If set, that is a dimensional item, if not, then it is a plain number. This can be automatically migrated for managed items but will be a breaking change for textual configuration.

I kind of like this idea. It certainly simplifies the whole units concepts and configuration. There will be some significant impacts to the UI and we'd have to determine the impacts on the bindings. But I like this. It's elegant.

But are we then saying here that if the binding sends a value and we've not defined unit the units will be stripped? I think I'm OK with that. This makes the adoption of units something the users get to opt into rather than something they must use. I wish they were implemented that way in the first place.

And/or, should the bindings be allowed to set the unit? If yes, the user should be able to override it. If not it would be a pretty big change from now and we'll have to prepare the way for users to know and understand the new behavior.

But then the unit will become the unit used everywhere, scale and all. If the Item receives an update different from that, it's converted to unit (assuming the different unit is compatible).

This feels consistent and gives the end user more control over the whole situation. However, I suspect a bunch of users will say "forget that, I ain't using UoM, it's complicated" if the default becomes no units.

[spacemanspiff2007]

But are we then saying here that if the binding sends a value and we've not defined unit the units will be stripped? I think I'm OK with that.

Yes - the unit should be stripped. Or to rephrase:
If no unit is provided it is assumed that the value is in the currently configured unit.

And/or, should the bindings be allowed to set the unit?

No. The unit is an item property and thus the "item definition".
The binding can send values with a unit but does not change/interact with the item unit itself.

However, I suspect a bunch of users will say "forget that, I ain't using UoM, it's complicated" if the default becomes no units.

Just like today where they just can use "Number".
But I think with the state normalization the UoM functionality and the benefit of UoM will become more clear which gives the users an incentive to use it.

[J-N-K]

Bindings can provide state descriptions, they should not interfere with item properties.

[J-N-K]

Closed, will be superseded by a PR depending on the outcome of #3282.