Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve description about numeric precision #259

Open
porres opened this issue Nov 25, 2024 · 5 comments
Open

improve description about numeric precision #259

porres opened this issue Nov 25, 2024 · 5 comments

Comments

@porres
Copy link
Collaborator

porres commented Nov 25, 2024

This is described in the manual in "2.4.1. Message types (selectors) and numerical precision"

I think it's worth noting that Pd uses '%g' and describe this better than it is right now. For reference, we have a detailed description here https://stackoverflow.com/questions/54162152/what-precisely-does-the-g-printf-specifier-mean/54162153#54162153 (thanks @ben-wes for the link).

note that [makefilename]'s help file could be improved, and I don't remember if I did it, but I think we can also improve and mention that's how things work in GUI boxes, maybe in their help file as well?

@porres
Copy link
Collaborator Author

porres commented Nov 25, 2024

I like the %g description from the python documentation

"General format. For a given precision p >= 1, this rounds the number to p significant digits and then formats the result in either fixed-point format or in scientific notation, depending on its magnitude.
The precise rules are as follows: suppose that the result formatted with presentation type 'e' and precision p-1 would have exponent exp. Then if -4 <= exp < p, the number is formatted with presentation type 'f' and precision p-1-exp. Otherwise, the number is formatted with presentation type 'e' and precision p-1. In both cases insignificant trailing zeros are removed from the significand, and the decimal point is also removed if there are no remaining digits following it.

Positive and negative infinity, positive and negative zero, and nans, are formatted as inf, -inf, 0, -0 and nan respectively, regardless of the precision.

A precision of 0 is treated as equivalent to a precision of 1. The default precision is 6."

@porres
Copy link
Collaborator Author

porres commented Nov 26, 2024

also, document range limit for %u and %x/%X in [makefilename]

@porres
Copy link
Collaborator Author

porres commented Nov 26, 2024

Ok, started imptoving at least the help file of [makefilename] in pure-data/pure-data#2486

@umlaeute
Copy link

I think it's worth noting that Pd uses '%g' and describe this better than it is right now.

i think this is not exactly worth it, as I'm afraid it will canonicalize the current behaviour (which really is just an implementation detail)

@porres
Copy link
Collaborator Author

porres commented Nov 26, 2024

Yeah, I'm really having second thoughts about this...

It's definitely worth in the [makefilename]. I mean, it was just wrong and needed fixing.

I thought the Manual could be wrong too, but it wasn't... the explanation wasn't all that great and I rephrased a bit and added more details. I think it definitely suits well the manual...

As for the other help files I'm kinda torn, and leaning to just removing the noise... since this is elsewhere. On the other hand, I always wanted to understand how the hell this thing actually worked and now that I finally internalized all the details I feel like exposing this everywhere.

thanks for the feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants