Skip to content

Commit

Permalink
Adding dynamic chromaloc correction
Browse files Browse the repository at this point in the history 
This became possible after mpv-player/mpv#15239
  • Loading branch information
@Artoriuz
Artoriuz committed Nov 2, 2024
1 parent dc42895 commit b792035
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 2 deletions.
10 changes: 9 additions & 1 deletion CfL_Prediction.glsl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.

//!PARAM chroma_offset_x
//!TYPE float
0.0

//!PARAM chroma_offset_y
//!TYPE float
0.0

//!HOOK CHROMA
//!BIND LUMA
//!BIND HOOKED
Expand All @@ -30,7 +38,7 @@
//!DESC Chroma From Luma Prediction (Downscaling Luma)

vec4 hook() {
return LUMA_texOff(0.0);
return LUMA_texOff(vec2(chroma_offset_x, chroma_offset_y));
}

//!HOOK CHROMA
Expand Down
10 changes: 9 additions & 1 deletion CfL_Prediction_Lite.glsl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.

//!PARAM chroma_offset_x
//!TYPE float
0.0

//!PARAM chroma_offset_y
//!TYPE float
0.0

//!HOOK CHROMA
//!BIND LUMA
//!BIND HOOKED
Expand All @@ -30,7 +38,7 @@
//!DESC Chroma From Luma Prediction (Downscaling Luma)

vec4 hook() {
return LUMA_texOff(0.0);
return LUMA_texOff(vec2(chroma_offset_x, chroma_offset_y));
}

//!HOOK CHROMA
Expand Down

12 comments on commit b792035

@Damole-wer
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello again) I would like to report that this commit has the same effect as 03b7f6a, which means it fails the RTINGS chroma subsampling test. This test is not a real world example, but it's very useful to check for chroma matching with luma, as it requires chroma placement to be perfect, in order to fully reconstruct the pattern. CfL is only able to do it at 0 offset, no matter what a chromaloc is.
I really think these kind offsets are only needed for spatial chroma scalers, they always slightly shift chroma to the left, resulting in a darkening or change in color slightly to the right. CfL with offsets has a similar effect.
I would like to ask what effect are you trying to achieve with this shader and how do you test which one is better, as it isn't really clear.

It would be nice if there were two versions of CfL:

  • The one without any limitations, that tries to reconstruct as much infromation as possible, even if it would result in artifacts. Those artifacts are the result of general video compression and only few pixels large anyway, you have to really pixel peep to notice it on a moving image. I personally like how the shader is able to reconstruct much more details and look more like the original image than regular chroma upscalers.
  • The optimised one, which has many customisation options to tweak the output for different people liking. The base quality would be safer, with optimisations to hide the artifacts, which could be details from the original image. The lite version isn't really needed, as the shader is already as fast as regular upscalers.

That's pretty much it. Hopefully, i'm not asking too much.
Thank you, for continuously updating this repo.

@Artoriuz
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shader is fine.

Depending on how you're converting the PNG to video you might be forcing centre chromaloc regardless of what's tagged or assumed by mpv. FFmpeg has only recently added toggles to control this: https://trac.ffmpeg.org/wiki/Scaling#Chromasamplelocation

Additionally, the bilinear downsampling step in the shader is very simple (by design). Shifting the sample location obviously means you can't weight all 4 input pixels the same and therefore the end result is worse. Left chromaloc is simply horrible and it's genuinely mind boggling that someone thought it would be a good idea to introduce a semi-random half-pixel shift to the left when downsampling chroma. With centre at 0.5x you have the equivalent of box, which is why it'll often work better.

@Damole-wer
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming -vf scale=out_chroma_loc is the correct way to specify output chromaloc in FFmpeg, i've re-done my testing. The command converting PNG to a video is ffmpeg -loop 1 -i chroma-444.png -vf scale=out_chroma_loc={chromaloc} -t 00:00:05 -c:v libx264 -crf 0 -pix_fmt yuv420p 420_{chromaloc}.mp4. Videos with all 6 available chromalocs have been created and then separately screenshot with both shaders (0_offset and dynamic_offset). Here is the archive: https://github.com/user-attachments/files/17622079/RTINGS.chromaloc.test.zip

Both shader versions are obviously the same at center. Both are able to reconstruct the pattern with top and bottom chromaloc, although not as good as center. When chromaloc gets to the left (topleft, left and bottomleft), dynamic offset breaks the pattern, while 0 offset is fine.

Considering the vast majority of videos are using the left chromaloc. An offset to the left breaks the chroma placement, while 0 offset has no such problems, I don't get the need for it in the first place.

@Artoriuz
Copy link
Owner Author

@Artoriuz Artoriuz commented on b792035 Nov 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dynamic offset breaks the pattern

It does not.

Assuming FFmpeg is actually respecting what you want to do, what "breaks the pattern" is most likely the bilinear downsampling part with the offset correction in place. You can force centre if you want but that means chroma will have a half-pixel shift on content with left chromaloc, which in my opinion is worse than having an unrealistic test pattern break.

Edit: You'd also have to check whether mpv respects the chromaloc tag or simply makes its own assumptions based on format and resolution.

Edit 2: I don't remember what exactly I used to double check whether the changes were working as expected, but I think these examples from Hibike make it clear enough.

1st image, cscale=lanczos:
Image

2nd image, cfl_prediction without chromaloc correction in the downsampling step (luma and chroma don't match):
Image

3rd image, cfl_prediction with all chromaloc corrections:
Image

4th image, luma for reference:
Image

If you cycle through these it should become pretty clear.

@Artoriuz
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also about "normal" vs "lite", in my opinion the simpler variant is more elegant and if I were to drop one of them it would be the other one =)

@Damole-wer
Copy link

@Damole-wer Damole-wer commented on b792035 Nov 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'd also have to check whether mpv respects the chromaloc tag or simply makes its own assumptions based on format and resolution.

Mpv recognised all the chromaloc tags, which can be seen in mpv shift+i first page stats as "Chroma Loc:", it just has different names for topleft, center and left.

chroma will have a half-pixel shift on content with left chromaloc, which in my opinion is worse than having an unrealistic test pattern break.

My point is, even if the 0 offset is not technically correct, it constantly allows the shader to produce an image closer to the original. The pattern is guaranteed to not be compressed beforehand and it suffers very much from chroma subsampling, I get that those points make it unrealistic, but it still goes through the same subsampling process as others, and chroma upscalers are meant to reverse that. If an upscaler doesn't do it properly here (the best case scenario for CfL, as it able to reconstruct the image almost perfectly), then I don't know how it would be better for not so obvious cases. There needs to be a real world example where the dynamic shift is closer maching the original, otherwise it's difficult to consider it producing better or more correct results.

these examples from Hibike make it clear enough

The no corrections one seems to produce sharper and closer to how the luma lines are, while corrected one is at the same offset as lanczos and produces fuzzier lines than with no correction. Also, comparing the chroma reconstruction while having an image with no colors is very difficult.

To summirise, I don't mind the change. It just needs a clear example with colors, if it is better at reconstructing the original. I thought it is necessary to discuss and document it, so other people confidently know how the shader changes in quality.

@deus0ww
Copy link

@deus0ww deus0ww commented on b792035 Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Damole-wer Are you using a luma scaler?

@Damole-wer
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you using a luma scaler?

Nope, i'm using pure CfL for chroma upscaling, the only differences are offsets. Sorry, I should've mentioned the video-unscaled=yes mpv option.

@Artoriuz
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Damole-wer Looked at this again today and I have some findings.

This is what chroma looks like in the clip you sent:

Image

This is what it looks like when I convert the 444 test image to a 420 avif:

Image

And this is what it looks like when I tried making a video out of it with ffmpeg -i chroma-444.png -vf scale=out_chroma_loc=left -pix_fmt yuv420p10 -c:v libx264 -preset placebo -qp 0 420_left.mp4:

Image

It's clear this is still centre so the shader will never work...

@Damole-wer
Copy link

@Damole-wer Damole-wer commented on b792035 Nov 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I guess, there is no reliable way to force a chromaloc... Or do you have a way? Edit: Also, isn't chroma by itself would always be at center?
Btw, is the first image from a youtube video or the archive i've sent later? Cuz it seems too compressed for crf 0. Please take a look in archive, if you didn't.
The youtube example is just a FFmpeg recording uploaded there, basically how it's done for everything, if youtube misstags videos, it's safe to say most videos on the internet are misstagged.
Overall, I'm thankfull for your cooperation. If you have a case when 0 offset fails on a properly tagged left chromaloc, I would have no other objections and we can close this disscussion. But this all was just for a sake being completely sure that the shader did not degrade in quality, and I'm glad with what's already here.

@Artoriuz
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, is the first image from a youtube video or the archive i've sent later?

From the archive.

Overall, I'm thankfull for your cooperation. If you have a case when 0 offset fails on a properly tagged left chromaloc, I would have no other objections and we can close this disscussion. But this all was just for a sake being completely sure that the shader did not degrade in quality, and I'm glad with what's already here.

I'm not completely happy with the shader in its current state either. As it stands, it's doing both the linear regression and its spatial filter fallback in the same shader pass. The problem is that while we need to correct the output of the spatial filter we don't really need to do anything to the regression. The fact it seems to work well enough as its stands is honestly coincidental.

@Jules-A
Copy link

@Jules-A Jules-A commented on b792035 Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it stands, it's doing both the linear regression and its spatial filter fallback in the same shader pass. The problem is that while we need to correct the output of the spatial filter we don't really need to do anything to the regression. The fact it seems to work well enough as its stands is honestly coincidental.

Hmm... I never thought to try do it separately, what ends up happening in that case?

Please sign in to comment.