Shaka stops display webvtt subtitles when webvtt cue-time is greater than 2^33 (~26h30m43s)

[chihchianglu-google]

Have you read the FAQ and checked for duplicate open issues?
Yes.

If the problem is related to FairPlay, have you read the tutorial?
Not related to FairPlay.

What version of Shaka Player are you using?
v4.7.11 (uncompiled)
Btw, I tried almost all the versions since v3, and none of them works.

Can you reproduce the issue with our latest release version?
Yes, v4.7.11 is the latest build / version as of 3/4/2024.

Can you reproduce the issue with the latest code from main?
Yes, I tried it on nightly build page (which should be the main).

Are you using the demo app or your own custom app?
I try it on demo page and nightly build page

If custom app, can you reproduce the issue using our demo app?
Not custom app.

What browser and OS are you using?
Chrome on MacOS 14.3.1 (23D60)

For embedded devices (smart TVs, etc.), what model and firmware version are you using?
Not on embedded devices.

What are the manifest and license server URIs?
HLS + FMP4: link
HLS + TS: link
DASH + FMP4: link

What configuration are you using? What is the output of player.getConfiguration()?
Simply paste the manifest links on shaka player demo page w/o changing any configuration.

What did you do?
Paste the HLS manifest links to shaka player demo page, and starting from 56~58 seconds, you will start seeing subtitles not showing (safari native player, hls.js, and bitmovin are all playing w/o issues).

What did you expect to happen?
Subtitles keeps showing throughout the whole stream w/o issues.

What actually happened?
The issue happens when webvtt cue-time is greater than 2^33 (roughly 26h30m43s), which is the mpegts wraparound time.
Currently, Shaka always applies this logic that tries to calculate and apply the offset if cue-time is greater than 2^33(roughly 26h30m43s), and it's not right, because in FMP4, there's no timestamp wraparound.
With this logic, the playback seeking time is wrong, thus no subtitles display after that point.

To make sure my assumption is correct:

1. I created a hacked live stream to reproduce the issue, by starting the stream with timestamp that's ~60 seconds smaller than 2^33, and after ~60 seconds, shaka fails to show subtitles while safari native player, hls.js, and bitmovin are all playing w/o issues. You should see the same behavior from the manifests I provided above.

2. DASH shouldn't run into the same issue because the shaka code I highlighted above only applies to HLS, and if you check the DASH manifest I provided above, it's indeed playing w/o issues.

Let me know if you need anything further from me, thanks.

[joeyparrish]

Confirmed repro. Thanks for the report! The stream URLs make this much easier to work on.

Currently, Shaka always applies this logic that tries to calculate and apply the offset if cue-time is greater than 2^33(roughly 26h30m43s), and it's not right, because in FMP4, there's no timestamp wraparound.

I believe you are correct. I suppressed the rollover logic in the debug console by setting shaka.text.VttTextParser.TS_ROLLOVER_ = Infinity, and I see subs throughout the HLS+MP4 clip.

I tried the HLS+TS clip, but it hangs at the rollover point with errors about bad timestamps in the media. There might be an issue with our transmuxer here.

[joeyparrish]

The HLS+TS clip plays in v4.4.0 and v4.5.0, but not in v4.6.0, v4.7.0, or v4.7.11 (latest).

In the versions where the HLS+TS doesn't hang at the rollover point, we still need the shaka.text.VttTextParser.TS_ROLLOVER_ = Infinity hack to suppress the rollover logic in the VTT parser, which makes sense because we're transmuxing TS to MP4 in Chrome. (We may still need the rollover logic for native TS playback.)

Probably the HLS+TS hang is caused by a transmuxer change introduced between v4.5.0 and v4.6.0. I'll narrow that down further.

[joeyparrish]

No bug fix releases were made to v4.5.x before v4.6.0 came out. So the culprit for the media hanging must be one of the changes in v4.6.0:

https://github.com/shaka-project/shaka-player/releases/tag/v4.6.0

[joeyparrish]

The transmuxer bug that impacts the HLS+TS stream begins with ae423b4, PR #5846. @avelad, I would love your input on this when you are available.

I can work on the VTT rollover, though. Making the rollover logic conditional is easy enough, but the condition is whether or not we're playing TS. Plumbing that knowledge into the parser may be ugly. But I'll figure it out.

[joeyparrish]

@chihchianglu-google, you can work around this issue with the following configuration:

player.configure('manifest.hls.sequenceMode', false);

This can cause issues with certain HLS content if the timestamps in the media are wrong, but I suspect the streams you are creating will be just fine.

Does this work for you as a workaround for now? Can you test this, and if it is robust enough for you, can you recommend this to your customers?

---

@avelad, related to this issue, we should discuss:

1. If we should always enable the transmuxer for TS content, because that would remove the need for new data plumbed to VTTTextParser and allow us to completely drop the rollover logic
2. What part of the TS transmuxer optimizations from fix: Fix nalu parsing and improve performance in the transmuxer #5846 could be causing the hang that appears in https://storage.googleapis.com/livestream-demo-output/ccl-webvtt-test/manifest-ts.m3u8 at the rollover point, around the 55-60 second mark, with errors about bad timestamps

Thanks!

[joeyparrish]

Lowering priority to P2 based on the configuration workaround, assuming acceptance of that workaround by the OP

[chihchianglu-google]

@joeyparrish , the config seems not working for me. Here's the player link.
I simply uncheck the config from UI, and I can see that reflect in the player URL above.
However, after that, all the subtitles are gone.

[avelad]

@avelad, related to this issue, we should discuss:

1. If we should always enable the transmuxer for TS content, because that would remove the need for new data plumbed to VTTTextParser and allow us to completely drop the rollover logic

I think it would be the best given the number of issues there are with the native TS.

2. What part of the TS transmuxer optimizations from [fix: Fix nalu parsing and improve performance in the transmuxer #5846](https://github.com/shaka-project/shaka-player/pull/5846) could be causing the hang that appears in https://storage.googleapis.com/livestream-demo-output/ccl-webvtt-test/manifest-ts.m3u8 at the rollover point, around the 55-60 second mark, with errors about bad timestamps

The problem is that we do not have pes-rollover implemented in the tsparser and tstransmuxer, we should implement it. I can work on it in 3 weeks, if that's okay with you.

[joeyparrish]

@joeyparrish , the config seems not working for me. Here's the player link. I simply uncheck the config from UI, and I can see that reflect in the player URL above. However, after that, all the subtitles are gone.

@chihchianglu-google, you are right. I was mistaken. I must have made a mistake while testing yesterday. Bumping back to P1.

@avelad wrote:

The problem is that we do not have pes-rollover implemented in the tsparser and tstransmuxer, we should implement it. I can work on it in 3 weeks, if that's okay with you.

@chihchianglu-google, the TS parser fix will have to wait until @avelad is available to work on it. Thanks for your understanding!

[joeyparrish]

I did some more digging and found that the timestamp logic was working in v4.2.0 - v4.2.2, but not in v4.2.3 or v4.3.0. This was the change responsible AFAICT: 3b9af2e, PR #4586, which mentions internal bug b/253104251 in the regression test.

If I remove the rollover logic entirely, that regression test will fail. It's not clear to me yet how to proceed. I don't have access to the original streams that led to that change, only the regression test that simulates them.

[chihchianglu-google]

Just my 2 cents.
I feel the issue of the webvtt in shaka is because it doesn't know the corresponding media (video/audio) segment timestamp. If it does, the calculation probably could be more generic w/o many corner cases because X-TIMESTAMP-MAP is just a timestamp mapping between vtt cue-time and media segment timestamp.

[joeyparrish]

Just my 2 cents. I feel the issue of the webvtt in shaka is because it doesn't know the corresponding media (video/audio) segment timestamp. If it does, the calculation probably could be more generic w/o many corner cases because X-TIMESTAMP-MAP is just a timestamp mapping between vtt cue-time and media segment timestamp.

You're not wrong, but it's more complicated than that for many reasons.

1. The same VTT file can be used in both HLS and DASH. In DASH, X-TIMESTAMP-MAP should be ignored. (In fact, in DASH, according to DASH InterOp Points (IOP), you should only ever use VTT segments wrapped in MP4, which have their own timestamps in the MP4.)
2. "Knowing" the media segment timestamp in a VTT parser is architecturally weird. The VTT parser shouldn't have to know things about video stream.
3. The "corresponding" media segment is ill-defined. Segments are not required to align at all. I could have 10s text segments and 2s video segments. Which of these 5 segments "corresponds" to this text segment? (Or the two durations could be co-prime, or the segments sizes could change on every single segment...)

We have HLS live streams in the demo app, but none of them have subtitles. So I don't have any real-world examples at hand to compare the formatting and timestamps of different live streaming solutions to vet any changes I make.

I'm looking more closely at the HLS spec so I can try to adjust the logic based on that.