[ViacomCBS] [XBOX] Clear preroll period to first encrypted content period failure.

[dsparacio]

Have you read the FAQ and checked for duplicate open issues?
Yes completely.

What version of Shaka Player are you using?
Tested Master branch, 2.5.14, 2.5.10, 2.4.7, 2.3.10 and 2.2.10 All of which had the exact same result. Could not get out of clear pre roll to content protected period.

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from master?
Yes

Are you using the demo app or your own custom app?
Demo raw simple setup. Page urls included in support email with content and la urls.

If custom app, can you reproduce the issue using our demo app?
yes

What browser and OS are you using?
XBOX Win/10 Spartan (2014–2019 Edge) non chromium build of edge which is what XBOX dev offers at this time.

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

Sandbox ID: XDKS.1
OS version: 10.0.19041.3068 (rs_xbox_release_2006.200610-0000)
OS edition: June 2020
Xbox Live device ID: FD00A956242E1869
Console ID: 74f233ae.f9265a03.c4fee763.851eb008.01
Serial number: 210633693016
Console type: Xbox One S
Model: 1681
Devkit type: Universal Windows App Devkit

Sandbox ID: XDKS.1
OS version: 10.0.19041.3068 (rs_xbox_release_2006.200610-0000)
OS edition: June 2020
Console ID: A7DCC4D8.6B218CCF.694B68FC.0D3DB308.01
Model: 1681
Console type: Xbox One S
Devkit type: Universal Windows App Devkit

Sandbox ID: XDKS.1
OS version: 10.0.19041.3068 (rs_xbox_release_2006.200610-0000)
OS edition: June 2020
Xbox Live device ID: FD00812AA7534FE0
Console ID: 4c279407.7e461c98.d56bb52f.dfb64007.01
Serial number: 051235192517
Console type: Xbox One X
Devkit type: Universal Windows App Devkit

What are the manifest and license server URIs?
Same as this ticket #2620
Sent over on [email protected]

What did you do?

Play from 0 start time and let preroll finish, on the first transition to a content period player stalls.

Non zero test are interesting.
If you skip the pre-roll with a non zero start time, all mid-roll DAI period transition thereafter works as expected.

What did you expect to happen?
The first content period initializes and plays out.

What actually happened?
The content period fails to init and play properly.

[dsparacio]

XBOX it is failing 100% on all boxes. We tested many versions and all fail exactly the same way.

I tried to apply some of your Tizen fixs I saw in master for Edge as well to just test on XBOX. For instance this block

// on one of these platforms. Instead, default stallSkip to 0 to force the
// stall detector to pause and play instead.
if (shaka.util.Platform.isWebOS() ||
shaka.util.Platform.isTizen() ||
shaka.util.Platform.isEdge() ||
shaka.util.Platform.isChromecast()) {
streaming.stallSkip = 0;
}
and the Endianness changes I did for comcast just to test..... All of which made no difference for XBOX.

[dsparacio]

Here is an image from xbox, much like the other samsung issue

[dsparacio]

We had to devs test on two of the mentioned XB1 above and here are the logs.
xbox_shaka_2514.txt

2.5.14-xbox.txt

---

this issue is related to #2620

[joeyparrish]

Looking at the content manifest, this appears to be a decryption failure, similar to what I found in #2722 (comment) .

If there's no PSSH in the manifest, then we rely on the encryption event, which according to #2722 may not always fire in Edge. But you do have a PSSH in your manifest.

One thing I notice is that your PSSH elements look like this:

<pssh xmlns="urn:mpeg:cenc:2013">

Instead of what I'm used to seeing, where the namespace is defined in the MPD element with xmlns:cenc="urn:mpeg:cenc:2013" and then the PSSH element looks like this:

<cenc:pssh>

I tested this in Chrome, and the XML parser in Chrome creates a node with namespaceURI set to "urn:mpeg:cenc:2013" as it should. If for some reason this were not the case in Tizen or XBox One, then that would cause the PSSH element to be skipped by the DASH parser, and the content would be played as though it were unencrypted, leading to what you see in the screenshot.

As an experiment, you could remove the check on namespaceURI in XmlUtils:

https://github.com/google/shaka-player/blob/e7965fe21962833955a4f98c11e87f8931b721fd/lib/util/xml_utils.js#L74-L75

If that fixes it, it tells us that the XML parser on Tizen and XBox is not compliant. Then we can start trying to come up with a workaround for those broken platforms. There is probably an alternate way to get the namespace URI.

Assuming the change I suggested above works, my first guess at a workaround would be to fall back to getAttribute(). I've checked that this fallback works on your content in Chrome, and so may also work on XBox and Tizen:

      return child instanceof Element && child.localName == name &&
          (child.namespaceURI == ns || child.getAttribute('xmlns') == ns);

If that works, we can make have that change in our next releases.

(There is one other instance of namespaceURI in MpdUtils, and we would need some compiler or linter check to ensure that we limit usage of namespaceURI to avoid regressions. Alternately, a polyfill could work, but I'm not sure how to polyfill something like this. Perhaps by monkey-patching the getter for Element.prototype.namespaceURI, but that could get complicated quickly.)

[dsparacio]

As an experiment, you could remove the check on namespaceURI in XmlUtils:

@joeyparrish I will set this test up today and get back to you asap. Great info thank you so much.

[dsparacio]

No luck but want to make sure I made the exact change you recommended.

From

shaka.util.XmlUtils.findChildrenNS = function(elem, ns, name) {
  return Array.prototype.filter.call(elem.childNodes, function(child) {
    return child instanceof Element && child.localName == name &&
        child.namespaceURI == ns;
  });
};

To

shaka.util.XmlUtils.findChildrenNS = function(elem, ns, name) {
  return Array.prototype.filter.call(elem.childNodes, function(child) {
    return child instanceof Element && child.localName == name;    
  });
};

Tested on both XBOX and Samsung with same green decode issue.

Also tried this on chrome in parallel which worked fine on chrome but had no affect on xbox

shaka.util.XmlUtils.findChildrenNS = function(elem, ns, name) {
  return Array.prototype.filter.call(elem.childNodes, function(child) {
    return child instanceof Element && child.localName == name &&
    (child.namespaceURI == ns || child.getAttribute('xmlns') == ns);
  });
};

[joeyparrish]

Thanks for trying. I'm embarrassed to realize that I should have read your logs. "Ignoring duplicate init data" in the log means it was at least trying to do a license exchange.

Sadly, I don't have any way to debug on an XBox right now.

Can you tell me if this is happening with any other PlayReady content in the demo app? If this is content-specific, that could be a useful clue.

Does it happen if you play an unencrypted version of the same content? That would tell us if the green garbage on the screen is the result of a decryption issue or a decoding issue.

Thanks!

[dsparacio]

Can you tell me if this is happening with any other PlayReady content in the demo app? If this is content-specific, that could be a useful clue.

It is not content specific or ad inventory specific but it is DAI specific. If we use our encode with no DAI SSAI, but same period structure and content protection etc. It plays with no issue on Both devices. XB1 and Samsung with no decode issue no green blocks.

Does it happen if you play an unencrypted version of the same content? That would tell us if the green garbage on the screen is the result of a decryption issue or a decoding issue.

We actually have tested this before and it works. So we have DAI stream but no DRM, same multiperiod structure etc, and it plays fine as well. So this is dectyption related I believe.

[joeyparrish]

If this is like the other issue you reported, where both DAI and DRM are necessary components, do you know what changes are made by the DAI system? Does it ingest encrypted segments, or does ingest clear segments and encrypt them for you at the end of the pipeline? If it ingests encrypted segments, does it ever have to split an encrypted segment to insert an ad?

[dsparacio]

Encrypted when it hits DAI. To my knowledge DAI just inserts AD periods and our content periods are untouched.

[joeyparrish]

If the ad periods are inserted at a segment boundary without modifying/splitting any of the segments, in theory you could cut out the ad periods and get playback equivalent to what you started with before DAI.

I'll try to come up with a means to do that. If we can make that happen in Chrome, we could send you a patch and you could try it on XBox to see what happens. It wouldn't be a solution, but it might give us another clue to figure out what is going wrong.

[dsparacio]

If the ad periods are inserted at a segment boundary without modifying/splitting any of the segments

Confirmed with our All Access team DAI does not modify any encrypted content segments at all.

[Aajax0]

I am seeing the same issue with our streams, when there is unencrypted period (ad preroll) before encrypted period (content). This only happens on Edge with PlayReady and Xbox One.
Anything I can try or provide to help with this issue?

[joeyparrish]

Hi everyone,

Sorry for the long delay with no updates.

We still don't have any bright ideas for how to debug this or what might be causing it. Our best guess is that something specific to DAI triggers a bug in the XBox or its browser or its PlayReady implementation, but we have no way to test that idea or dig deeper at the moment.

It would be ideal if Microsoft could debug this somehow, since they would have access to information from the browser and OS that we could never see.

[Aajax0]

Thanks @joeyparrish. I am talking to Microsoft about it, so will let you know if anything useful comes out of that.

[Aajax0]

Microsoft are saying that this is a limitation on their side in both Edge browser and Xbox, perhaps PlayReady in general. They determine the decoding pipeline method by first appended fragment to each source buffer. So if a stream starts with an unencrypted fragment (pre-roll advert for example) all encrypted content after the unencrypted part won't be decoded correctly. But no errors thrown, just green screen and no audio...

The way we are fixing this in our fork is that we find an init fragment of an encrypted period and when the stream starts from an unencrypted period, we append the encrypted init fragment first, then straight after that the correct unencrypted init fragment and continue as usual. This fixes the problem, but obviously it is a bit unusual solution.

@joeyparrish Please let me know if you are interested in a PR with this or if you'd rather stay away from this :)

[joeyparrish]

Yes, please send a PR! With appropriate comments, I'm sure there's a relatively clean way to make it work.

[dsparacio]

@davidbohunek David great info and news! We can test our content as well once PR is up. Thanks for sharing.

[agajassi]

@joeyparrish
Getting same 4001 XML Error even with TextDecoder polyfill added to the page.

[agajassi]

@joeyparrish
We tried Shaka version: v3.0.7-20-g66cdd44e-debug on Xbox. The one that does not require polyfill.

Got this error:
MEDIA_SOURCE_OPERATION_FAILED 3014 number. A MediaSource operation failed. error.data[0] is a MediaError code from the video element.

Full logs:
shaka_logs_new.txt

[dsparacio]

I built 3.0.x that resulted in v3.0.7-20-g66cdd44e-debug in logs. @joeyparrish Does all this look correct from your perspective? Does this log help at all. Let us know if you need us to collect or debug anything specifically.

[joeyparrish]

Thank you for the logs. I wish I could see what the MediaSource error was in error.data[0], but it doesn't appear to get logged. Is that something you can log for us?

I'm not sure what to do next. I think it's best if I revert the change in v3.0.x until we have this sorted out, and go ahead with the v3.0.8 release without this change.

I've started trying to replace our Xbox so we can deal with this in a more hands-on way, but apparently the original XB1 consoles are not being made or serviced, and the new ones are out of stock almost everywhere because of supply chain issues and import quarantines.

[dsparacio]

@joeyparrish if I can arrange an xbox to loan you for a few weeks would that help?

[joeyparrish]

I managed to find a used one that could be shipped to me by the end of the week. But thank you for the offer!

It will take some time to get set up for anything more useful than opening the browser, but I'll try to set up Visual Studio and dev mode soon.

[joeyparrish]

I have a hacky way to manually connect the Xbox to the Shaka automated tests. I found, to my disappointment, that the built-in browser doesn't seem to have PlayReady enabled. But I was able to build a UWP app with WebView and add a capability to the manifest to enable PlayReady there. Through that, I can run our automated tests and get console logs. But it's painful. I don't have time to build this all the way into our test lab with https://github.com/google/generic-webdriver-server yet, but I've got most of the pieces in place and it's good enough to allow me to do some basic debugging on the device.

I'm about out of time for today, but I may be able to get some useful info tomorrow.

That said, I think I still need to revert the workaround in the v3.0.x branch and get v3.0.8 out the door. I meant to have it done sooner. There are other fixes that need to be in an official release ASAP.

[joeyparrish]

My dev mode Xbox One refuses to play your content with my workaround, and the MS error code attached to the failure is 0x80004005, which seems to translate simply to E_FAIL. Not too helpful. :-)

But with the workaround disabled, it at least begins playback. So that means if I can figure out what is wrong with the manipulated init segments, I should get immediate feedback.

[riksagar]

Hey @joeyparrish , sorry if you already took this into account. You mentioned your Xbox1 is in DevMode. For DevMode you need to set you PR license to SL150.

[joeyparrish]

@riksagar, I'm not in control of the PR licenses coming from CBS's license service, but they seem to be fine in that regard.

Flipping the order of "encv"/"enca" and "avc1"/"mp4a" boxes seems to make my Xbox happy, although Chrome doesn't like it. Since this is a platform-specific workaround, that might not matter in the long run. However, I found that even with the workarounds disabled, the sample content sent to me by @dsparacio seems to play anyway.

So, it's possible that my Xbox simply doesn't exhibit the problem to begin with. @dsparacio, @agajassi, please try this small edit to the master branch in lib/media/content_workarounds.js to see if it helps:

-    const sourceBoxEnd = sourceBox.start + sourceBox.size;
+    const sourceBoxEnd = sourceBox.start;

For me, it at least fixes the E_FAIL error I saw before, but since I can't seem to reproduce the blocky-green artifacts of a failed decrypt either way, you'll have to let me know what that small change does for your devices.

[dsparacio]

Thanks @joeyparrish Will try in AM

[bcupac]

@joeyparrish @dsparacio
This issue is also reproducible for our streams when using Chromium Edge or old Edge with PlayReady or Windows 10 UWP which is basically old Edge.

[bcupac]

Tested windows 10 uwp, Xbox and chromium edge with this change.

-    const sourceBoxEnd = sourceBox.start + sourceBox.size;
+    const sourceBoxEnd = sourceBox.start;

Stream works OK in windows10 UWP and Xbox, but throws 3014 error on chromium edge.

with using old
const sourceBoxEnd = sourceBox.start + sourceBox.size
on chromium edge, no 3014 error but still has decoding green artifacts

[joeyparrish]

Stream works OK in windows10 UWP and Xbox, but throws 3014 error on chromium edge.

This workaround won't be applied on Chromium Edge, only on Xbox One and Tizen.

I'm glad to hear this fixes it for you on Xbox. I'll get this rolled into the next release.

[joeyparrish]

The existing workaround works on Tizen, but not Xbox One, and the modified workaround works on Xbox One, but not Tizen. So the next change to the code will have to be to make it platform-specific which order these boxes go in.

I've verified that with the new forthcoming workaround and the current platform rules for applying it (Xbox One & Tizen only), the content from CBS plays through at least 90s (preroll, one content period, midroll) for me on:

1. Chrome 88 on Windows 10 (workaround skipped)
2. Chromium-based Edge 88 on Windows 10 (workaround skipped)
3. Xbox One firmware 19041.5496 (November 2020) in UWP WebView (Legacy Edge 18) (workaround applied, encv box first)

In earlier test, these were also working:

1. Tizen 3 w/ PlayReady config (init segment workaround applied, avc1 box first)
2. Tizen 3 w/ Widevine config (init segment workaround applied, avc1 box first)

However, I can't reconfirm those, as our Tizen test infra seems to be broken at the moment. I can't load any external assets for some reason, and it complains about cross-origin access in spite of everything having explicit CORS headers allowing access. Not sure if it was broken by a TV firmware update or what, but the test infra itself hasn't changed.

[agajassi]

@joeyparrish

This is Agajan. I work with @dsparacio. I just tried your recent changes in v3.0.8-master and wanted to confirm that the underlying issue works on Xbox. Thank you for your help addressing this.

[joeyparrish]

We're very happy that we could address it at all. The change went out in the official v3.0.8 release, in case you want to stay on the release branch. Thanks for your feedback and for helping verify the work!

[dsparacio]

@joeyparrish much appreciated. Thanks for sticking with this one!

[joeyparrish]

Xbox One will soon become an officially-supported platform. If anyone is interested to see how, check out this PR for our new test infrastructure tools for Xbox: shaka-project/generic-webdriver-server#12

[avelad]

@dsparacio Do you know if this problem is exclusive to Xbox One or also happens in the Xbox Series S|X? I see that @joeyparrish 's code only applies to Xbox One. (https://github.com/google/shaka-player/blob/master/lib/util/platform.js#L97)

[dsparacio]

@avelad not that far with new Xbox so not sure. It does happen on a certain Samsung versions.

[joeyparrish]

The workaround is applied based on the UA string, so it's possible that it's too narrowly-targeted.

For what it's worth, I developed the workaround on a refurbished Xbox One X model. There, the workaround appeared to be unnecessary, but not harmful. I expect that excluding it from Series X and Series S models will be okay. If I'm wrong, it's a ~1-line fix.