Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

units need updating #4

Closed
Anto79-ops opened this issue Mar 20, 2023 · 31 comments
Closed

units need updating #4

Anto79-ops opened this issue Mar 20, 2023 · 31 comments
Labels
bug Something isn't working

Comments

@Anto79-ops
Copy link
Contributor

Hi,

This is a minor thing, but just to be consistent, it would be good to update the units to the proper ones

shenxn/ha-dyson#137

thanks

@dotvezz
Copy link
Member

dotvezz commented May 4, 2023

Okay, I've got conversion factors for everything except Formaldehyde. It's all so straightforward I actually feel silly. Spoiler: There never were any base units to begin with!

  • NO2: 0.1
  • VOC: 0.1.
    • That's it. All the chemistry math I've been having to remember has been for naught lol.
    • So that means it's not base units of ppm or µg/m3. NO2 and VOC are actually presenting the exact reading given from the app, with the decimal removed.
    • A value of 0020 is a 2 in the app, 0085 is an 8 on the app, halfway to a 9. There's no unit at all, just an index value.
  • PM10: 1
  • PM2.5: 1
    • like I said before, it's an exact match from MQTT to the app. So the conversion factor is 1. This doesn't need any change in the integration.

I'll push an update soon that updates all the units and also sets the correct device class so Formaldehyde gives you a graph.

@dotvezz
Copy link
Member

dotvezz commented May 4, 2023

https://github.com/libdyson-wg/ha-dyson/releases/tag/v0.18.1 updates all the units and removes the µg/m³ unit labels from VOC, NO2 and Formaldehyde.

  • For Formaldehyde, we'll probably need to add the mg/m³ unit label once the conversion math is known. When you get a chance, can you screenshot the graph and app again to help figure out the conversion factor?
  • For NO2 and VOC,I kind of want to add some unit label that explains it's a 0-10 index. We can add "(0-10)" for example, but there's gotta be a better way.

@Anto79-ops
Copy link
Contributor Author

Anto79-ops commented May 4, 2023

thanks for the latest update to address the units issue!

Some error logs in HA though.

Logger: homeassistant.components.sensor
Source: components/sensor/__init__.py:648
Integration: Sensor (documentation, issues)
First occurred: 3:51:41 PM (10 occurrences)
Last logged: 3:51:49 PM

Entity sensor.pure_cool_link_volatile_organic_compounds_2 (<class 'custom_components.dyson_local.sensor.DysonVOCSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.
Entity sensor.pure_cool_link_particulates_2 (<class 'custom_components.dyson_local.sensor.DysonParticulatesSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('pm1') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.
Entity sensor.pure_hot_cool_new_volatile_organic_compounds (<class 'custom_components.dyson_local.sensor.DysonVOCSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.
Entity sensor.pure_hot_cool_new_nitrogen_dioxide (<class 'custom_components.dyson_local.sensor.DysonNO2Sensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('nitrogen_dioxide') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.
Entity sensor.pure_hot_cool_new_formaldehyde_2 (<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.

and some more:

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:405
Integration: Sensor (documentation, issues)
First occurred: 3:55:17 PM (8 occurrences)
Last logged: 3:55:17 PM

The unit of sensor.pure_hot_cool_nitrogen_dioxide is changing, got multiple {'µg/m³', None}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (µg/m³). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.pure_cool_link_volatile_organic_compounds_2 is changing, got multiple {'µg/m³', None}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (µg/m³). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.pure_cool_link_particulates_2 is changing, got multiple {'µg/m³', None}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (µg/m³). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.pure_hot_cool_new_volatile_organic_compounds is changing, got multiple {'µg/m³', None}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (µg/m³). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
The unit of sensor.pure_hot_cool_new_nitrogen_dioxide is changing, got multiple {'µg/m³', None}, generation of long term statistics will be suppressed unless the unit is stable and matches the unit of already compiled statistics (µg/m³). Go to https://my.home-assistant.io/redirect/developer_statistics to fix this

@dotvezz
Copy link
Member

dotvezz commented May 4, 2023

using native unit of measurement 'None' . . . expected one of ['µg/m³']

Yeah me too 😅.

@Anto79-ops
Copy link
Contributor Author

Anto79-ops commented May 4, 2023

ha, ok, unfortunatly its not collecting stats anymore because of the second issue. So I can not show you the plots to compare for formaldehyde.

ITs almost like it a unit has to be invented....ie. DU Dyson Units....I may revert back to 0.18.0

@dotvezz
Copy link
Member

dotvezz commented May 4, 2023 via email

@dotvezz
Copy link
Member

dotvezz commented May 5, 2023

Sorry, my last comment was sent from my phone over email so I wasn't able to go into much detail. You can go to your Home Assistant's /developer-tools/statistics page, search for "Dyson", and reset the statistics. This is what I would recommend.

That said, I think it's best for me to actually create new sensor entities so that it tracks them on a new graph instead of needing a reset. I'll pull down the 0.18.1 release (since it turned out to be akin to a breaking change) and do a 0.19.0 release with new sensors so it doesn't freak out Home Assistant.

  • Volatile Organic Compounds -> rename to "Volatile Organic Compounds Index"
  • Nitrogen Dioxide -> rename to "Nitrogen Dioxide Index"
  • Particulates -> rename to "Air Quality Index"
    • This matches what's in the app IIRC (I'll double check) and I actually think Home Assistant supports AQI natively, so that'll be nice.

@dotvezz
Copy link
Member

dotvezz commented May 5, 2023

@Anto79-ops I've released v0.19.0 which should take care of your issue with graphs and units as well.

Whew, this has been a big learning journey. We're almost there.

@Anto79-ops
Copy link
Contributor Author

@dotvezz awesome just updated to version 19. I will keep you posted!

@Anto79-ops
Copy link
Contributor Author

Anto79-ops commented May 6, 2023

Hey, I have some updates. I really like this more intuitive way of showing the levels. NO2, VOC and PM seems to updating accordingly.

But you're right, its almost there. Formaldehyde needs a bit more tweaking.

First, it seems its not updating as much as it used to. Maybe its rounding to 0.1, its almost like it is stuck at 0.2, for example:

image

and my second machine, did have a quick register at 0.3

image

However, when comparing the data with he app, it does fluctuate quite a bit as shown here

image

In you original fix of this, it up was updating every few minutes, like going to 5 to 12 So I wonder if its just a rounding issue.

there is a log error on every restart for both my machines, could it be related?

Logger: homeassistant.components.sensor
Source: components/sensor/__init__.py:648
Integration: Sensor (documentation, issues)
First occurred: 4:19:54 AM (2 occurrences)
Last logged: 4:20:02 AM

Entity sensor.pure_hot_cool_new_formaldehyde_2 (<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.
Entity sensor.pure_hot_cool_new_formaldehyde (<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.

What do you think is happening with the formaldehyde?

Nice update! Just needs a final tweak and I think you got this. Thank you!

@dotvezz
Copy link
Member

dotvezz commented May 6, 2023

Thanks for the info! Turns out I've committed a stupid and set the Formaldehyde sensor to track Nitrogen Dioxide.

@dotvezz
Copy link
Member

dotvezz commented May 6, 2023

Okay, I've deleted the 0.19.0 release and replaced it with v0.19.1. For the log error, we'll knock it out once we know the conversion factor.

@Anto79-ops
Copy link
Contributor Author

Awesome! Just updated and it is giving those whole numbers as it did originally. Nice!

image

Now, I compare the plots with the app and we can figure out the conversion factor....or is the index scale of 1 to 100? It might make sense that is because my formaldehyde barely goes about 15 (Always in the green region). Time will tell i guess.

I still get those invalid unit errors in the log, but guess if we don't know what this units are, we'll figure it out.

@Anto79-ops
Copy link
Contributor Author

Here's some data:

image

and the dyson app

Screenshot_20230506-200851

you can see, it seems that it generally tracks it very well.

They app seems to also do some smoothing or averaging. Also, reporting intervals on the app (every 15 minutes) are not as much as HA

But, as a rough estimate, im getting a conversion factor of

at 2:30
HA 12 DA 0.029

at 4:45pm
HA 11, DA, 0.030

at 5 pm:
HA 9, DA 0.027

I'd say, its generally a conversion factor of "0.00271" based on the average of these 3 points:

0.00242
0.00300
0.00273

@Anto79-ops
Copy link
Contributor Author

so, if you use the HA scale, something above 0.300 mg/m3 in the DA or 110 in HA would be considered poor.
image

so, question is now, what are these units so that HA does not complain about missing units all the time.

Logger: homeassistant.components.sensor
Source: components/sensor/__init__.py:648
Integration: Sensor (documentation, issues)
First occurred: 4:19:54 AM (2 occurrences)
Last logged: 4:20:02 AM

Entity sensor.pure_hot_cool_new_formaldehyde_2 (<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.
Entity sensor.pure_hot_cool_new_formaldehyde (<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'None' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.

@Anto79-ops
Copy link
Contributor Author

Anto79-ops commented May 7, 2023

the more research I do on this, the more I'm starting to think the native formaldehyde component is ppb

Dyson says they use/develop a "solid-state formaldehyde sensor" that uses "electrochemical" method for precise formaldehyde detection against other VOC gases

HP09-variant-feature-HCHO-sensor

and

image

their marketing photo above, looks very similar to this sensor:

https://www.mouser.ca/new/sensirion/sensirion-sfa30-formaldehyde-sensor-module/

which does output ppb

But, looking at the tech spec of the Sensiron SFA30 on page 11, it talks about how it calculates ppb and its based on a reciprocal factor of 5. I'm trying to play with the math to see if things match.

I'm not saying that the Dyson uses the Sensiron sensor (it very well could be using it), but this sensor is exactly the Dyson Marketing suggests.

https://www.farnell.com/datasheets/3171435.pdf

@dotvezz dotvezz mentioned this issue May 9, 2023
@dotvezz
Copy link
Member

dotvezz commented Jun 7, 2023

I'm seeing there's also another formaldehyde sensor field in the MQTT data, labeled hchr as opposed to hcho. I decompiled the Dyson Android App (oh the rabbit holes we go down) and saw hchr referred to as "high res formaldehyde" and I wonder if that will give us something that we can understand. @Anto79-ops would you be willing to help me test since I don't have any Formaldehyde devices?

@Anto79-ops
Copy link
Contributor Author

@dotvezz

You bet!

@Anto79-ops
Copy link
Contributor Author

I just realized that the machine itself also outputs the number that's in the app. Is it possible that this the hchr as opposed to hcho (raw value)

20230703_084637.jpg

I'm seeing there's also another formaldehyde sensor field in the MQTT data, labeled hchr as opposed to hcho. I decompiled the Dyson Android App (oh the rabbit holes we go down) and saw hchr referred to as "high res formaldehyde" and I wonder if that will give us something that we can understand. @Anto79-ops would you be willing to help me test since I don't have any Formaldehyde devices?

@dotvezz
Copy link
Member

dotvezz commented Jul 4, 2023

Interesting! I got your email (thanks!). I've got one more thing that could be awesome to get your help with. I created a tool called dyson-mqtt-listen that just spits out all the MQTT data from a dyson device. Could you let it run for a while, and paste the results here or in a gist? I'm curious about the hchr and hcho field values and how much they correlate with each other, with the app, etc...

https://github.com/dotvezz/dyson-mqtt-listen

@Anto79-ops
Copy link
Contributor Author

for sure!

I will run this and get that data for your.

Cool diagnostics tool

@Anto79-ops
Copy link
Contributor Author

Anto79-ops commented Jul 5, 2023

@dotvezz BOOM! You got it sir, hchr/1000 is the app and machine LCD screen value in mg/m3

21:20:31: Connected to device...
21:20:31: Subscribed to 527E/[redacted]/status/current
21:20:31: Subscribed to 527E/[redacted]/command
21:20:31: Press Ctrl+C to exit.
21:20:53|527E/[redacted]/status/current: {"msg":"ENVIRONMENTAL-CURRENT-SENSOR-DATA","time":"2023-07-05T03:20:55.000Z","data":{"tact":"2979","hact":"0049","pm25":"0001","pm10":"0000","va10":"0016","noxl":"0002","p25r":"0001","p10r":"0002","hcho":"0017","hchr":"0043","sltm":"OFF"}}
21:21:05|527E/[redacted]/status/current: {"msg":"CURRENT-STATE","time":"2023-07-05T03:21:07.000Z","mode-reason":"LAPP","state-reason":"MODE","rssi":"-40","channel":"11","fqhp":"90808","fghp":"53408","product-state":{"fpwr":"ON","auto":"ON","oscs":"ON","oson":"ON","nmod":"OFF","rhtm":"ON","fnst":"FAN","ercd":"NONE","wacd":"NONE","nmdv":"0004","fnsp":"AUTO","bril":"0002","corf":"ON","cflr":"INV","hflr":"0056","cflt":"SCOF","hflt":"GCOM","sltm":"OFF","osal":"0180","osau":"0180","ancp":"0090","hmod":"OFF","hmax":"2942","tilt":"OK","hsta":"OFF","psta":"CLNG","fdir":"ON"},"scheduler":{"srsc":"0000000000000000","dstv":"0001","tzid":"0001"}}
21:21:05|527E/[redacted]/status/current: {"msg":"ENVIRONMENTAL-CURRENT-SENSOR-DATA","time":"2023-07-05T03:21:07.000Z","data":{"tact":"2979","hact":"0049","pm25":"0000","pm10":"0000","va10":"0016","noxl":"0002","p25r":"0001","p10r":"0001","hcho":"0017","hchr":"0043","sltm":"OFF"}}
21:21:23|527E/[redacted]/status/current: {"msg":"ENVIRONMENTAL-CURRENT-SENSOR-DATA","time":"2023-07-05T03:21:25.000Z","data":{"tact":"2979","hact":"0049","pm25":"0000","pm10":"0000","va10":"0016","noxl":"0002","p25r":"0001","p10r":"0001","hcho":"0017","hchr":"0043","sltm":"OFF"}}
21:21:35|527E/[redacted]/status/current: {"msg":"CURRENT-STATE","time":"2023-07-05T03:21:37.000Z","mode-reason":"LAPP","state-reason":"MODE","rssi":"-36","channel":"11","fqhp":"90808","fghp":"53408","product-state":{"fpwr":"ON","auto":"ON","oscs":"ON","oson":"ON","nmod":"OFF","rhtm":"ON","fnst":"FAN","ercd":"NONE","wacd":"NONE","nmdv":"0004","fnsp":"AUTO","bril":"0002","corf":"ON","cflr":"INV","hflr":"0056","cflt":"SCOF","hflt":"GCOM","sltm":"OFF","osal":"0180","osau":"0180","ancp":"0090","hmod":"OFF","hmax":"2942","tilt":"OK","hsta":"OFF","psta":"CLNG","fdir":"ON"},"scheduler":{"srsc":"0000000000000000","dstv":"0001","tzid":"0001"}}
21:21:35|527E/[redacted]/status/current: {"msg":"ENVIRONMENTAL-CURRENT-SENSOR-DATA","time":"2023-07-05T03:21:37.000Z","data":{"tact":"2979","hact":"0049","pm25":"0000","pm10":"0000","va10":"0016","noxl":"0002","p25r":"0001","p10r":"0001","hcho":"0017","hchr":"0043","sltm":"OFF"}}
21:21:53|527E/[redacted]A/status/current: {"msg":"ENVIRONMENTAL-CURRENT-SENSOR-DATA","time":"2023-07-05T03:21:55.000Z","data":{"tact":"2979","hact":"0049","pm25":"0001","pm10":"0000","va10":"0016","noxl":"0002","p25r":"0001","p10r":"0002","hcho":"0017","hchr":"0042","sltm":"OFF"}}

@Anto79-ops
Copy link
Contributor Author

Anto79-ops commented Jul 5, 2023

@dotvezz I just noticed that that's also a payload for RSSI.

Anyway, you can add a Wi-Fi signal strength sensor diagnostic to the machines? Could be useful for those who have connection issues.

@dotvezz
Copy link
Member

dotvezz commented Jul 5, 2023

@dotvezz BOOM! You got it sir, hchr/1000 is the app and machine LCD screen value in mg/m3

YES!!! Finally! I smell a new release coming soon.

Also great eye catching the RSSI field. That's an awesome suggestion, I'll definitely do that.

dotvezz added a commit to dotvezz/libdyson-neon that referenced this issue Jul 6, 2023
Reference discussion link: libdyson-wg/ha-dyson#4

The `hcho` field in MQTT is of an unknown unit/scale and has no known
conversion factor to a known unit. However, it was recently discovered
that `hchr` (referred to as "High Res Formaldehyde" in the android app)
has a clean conversion factor of n/1000 to mg/m^3.
dotvezz added a commit to dotvezz/libdyson-neon that referenced this issue Jul 6, 2023
Reference discussion link: libdyson-wg/ha-dyson#4

The `hcho` field in MQTT is of an unknown unit/scale and has no known
conversion factor to a known unit. However, it was recently discovered
that `hchr` (referred to as "High Res Formaldehyde" in the android app)
has a clean conversion factor of n/1000 to mg/m^3.
dotvezz added a commit to libdyson-wg/libdyson-neon that referenced this issue Jul 6, 2023
Reference discussion link: libdyson-wg/ha-dyson#4

The `hcho` field in MQTT is of an unknown unit/scale and has no known
conversion factor to a known unit. However, it was recently discovered
that `hchr` (referred to as "High Res Formaldehyde" in the android app)
has a clean conversion factor of n/1000 to mg/m^3.
@dotvezz
Copy link
Member

dotvezz commented Jul 6, 2023

@Anto79-ops
Copy link
Contributor Author

Nice! Thanks @dotvezz

Look at this 😍

IMG_20230705_230818.jpg

@Anto79-ops
Copy link
Contributor Author

its still giving these log errors wonder if HA needs to change their device class definitions.....but happy to see the number is reporting correctly!

Logger: homeassistant.components.sensor
Source: components/sensor/__init__.py:659
Integration: Sensor (documentation, issues)
First occurred: 11:01:25 PM (2 occurrences)
Last logged: 11:01:26 PM

Entity sensor.pure_hot_cool_new_hcho (<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'mg/m³' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.
Entity sensor.pure_hot_cool_new_hcho_2 (<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'mg/m³' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.

@dotvezz
Copy link
Member

dotvezz commented Jul 6, 2023

(<class 'custom_components.dyson_local.sensor.DysonHCHOSensor'>) is using native unit of measurement 'mg/m³' which is not a valid unit for the device class ('volatile_organic_compounds') it is using; expected one of ['µg/m³']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.

Ugh, that seems like an overly-strict definition of "valid". I don't know exactly how I feel about conforming to it, tbh. I want to keep the scale consistent with what the Dyson app shows to avoid confusion.

But then, Dyson themselves aren't consistent either - particulate sensors are presented in µg/m3 while HCHO is presented in thousandths of mg/m3. This is kinda ridiculous on their part since 0.001 mg/m3 = 1 µg/m3.

I'm leaning toward keeping it at mg, but I'm worried about the logs bothering people. Does it spam your logs, or is it just on initialization?

@Anto79-ops
Copy link
Contributor Author

lol I know what you mean!! It nice the integration does exactly what the Dyson app/machine does....but on the other hand you get these. Which by the way, it does not spam the logs, just on every startup for every formaldehyde machine you have.

But I did a little bit of research on this, and there was other mentioning this for other devices.

Technically formaldehyde is a VOC BUT, it might need a class on its own like NO2, SO2, CO2 etc. which exists for those:

https://www.home-assistant.io/integrations/sensor#device-class

perhaps a PR to HA Core :)

@dotvezz
Copy link
Member

dotvezz commented Jul 6, 2023

Looking at other Formaldehyde PRs against HA Core (example: https://github.com/home-assistant/core/pull/75807/files) it looks like they just don't set the sensor class for formaldehyde, so it's defaulting to SensorDeviceClass.NONE. I'll drop a patch release that just does this too.

Fun fact: Looks like other HCHO sensors also use mg/m3 so I guess I shouldn't complain about Dyson.

@Anto79-ops
Copy link
Contributor Author

You did it again!

Just updated and now everything is working with no log issues.

Thank you for being an awesome and responsive developer. My hat to you.
@dotvezz

Please allow me the pleasure of closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

No branches or pull requests

2 participants