Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks

Based on #1496

Problem
-------

We're currently trying to provide a complete[1] and easy to-use API for
DASH thumbnail tracks in the RxPlayer.

Today the proposal is to have an API called `renderThumbnail`, to which
an application would just provide an HTML element and a timestamp, and
the RxPlayer would do all that's necessary to fetch the corresponding
thumbnail and display it in the corresponding element.

The API is like so:
```js
rxPlayer.renderThumbnail({ element, time })
  .then(() => console.log("The thumbnail is now rendered in the element"));
```

This works and seems to me very simple to understand.

Yet, we've known of advanced use cases where an application might not
just want to display a single thumbnail for a single position. For
example, there's very known examples where an application displays a
window of multiple thumbnails at once on the player's UI to facilitate
navigation inside the content.

To do that under the solution proposed in #1496, an application could
just call `renderThumbnail` with several `element` and `time` values.

Yet for this type of feature, what the interface would want is not really
to indicate a `time` values, it actually wants basically a list of
distinct thumbnails around/before/after a given position.

By just being able to set a `time` value, an application is blind on
which `time` value is going to lead to a different timestamp (i.e. is
the thumbnail for the `time` `11` different than the thumbnail for the
`time` `12`? Nobody - but the RxPlayer - knows).

So we have to find a solution for this

[1] By complete, I here mean that we want to be able to handle its
complexities inside the RxPlayer, to ensure complex DASH situations like
multi-CDN, retry settings for requests and so on while still allowing
all potential use cases for an application.

Solution
--------

In this solution, I experiment with a second thumbnail API,
`getAvailableThumbnailTracks` (it already exists in #1496, but its role
there was only to list the various thumbnail qualities, if there are
several size for example). As this solution build upon yet stays
compatible to #1496, I chose to open this second PR on top of that
previous one.

I profit from the fact that most standardized thumbnail implementations I
know of (BIF, DASH) seem follow the principle of having evenly-spaced
(in terms of time) thumbnails (though I do see a possibility
for that to change, e.g. to have thumbnails corresponding to "important"
scenes instead, so our implementation has to be resilient).

So here, what this commit does is to add the following properties (all
optional) to a track returned by the `getAvailableThumbnailTracks` API:

  - `start`: The initial `time` the first thumbnail of that track will
    apply to

  - `end`: The last `time` the last thumbnail of that track will
    apply to

  - `thumbnailDuration`: The "duration" (in seconds) each thumbnail
    applies to (with the exception of the last thumbnail, which just
    fills until `end`)

Then, an application should have all information needed to calculate a
`time` which correspond to a different thumbnail.

Though this solution lead to a minor issue: by letting application make
the `time` operation themselves with `start`, `end`, `thumbnailDuration`
and so on, there's a risk of rounding errors leading to a `time`
which does not correspond to the thumbnail wanted but the one before or
after. To me, we could just indicate in our API documentation to
application developers that they should be extra careful and may add an
epsilon (or even choose a `time` in the "middle" of thumbnails each time)
if they want that type of thumbnail list feature.

Thoughts?

src/main_thread/api/public_api.ts

@@ -757,28 +757,51 @@ class Player extends EventEmitter<IPublicAPIEvent> {
    * Returns either an array decribing the various thumbnail tracks that can be
    * encountered at the given time, or `null` if no thumbnail track is available
    * at that time.
-   * @param {number} time - The position to check for thumbnail tracks, in
-   * seconds.
+   * @param {Object} arg
+   * @param {number|undefined} arg.time - The position to check for thumbnail
+   * tracks, in seconds.
+   * @param {string|undefined} arg.periodId
    * @returns {Array.<Object>|null}
    */
   public getAvailableThumbnailTracks({
     time,
+    periodId,
   }: {
-    time: number;
+    time: number | undefined;
+    periodId: string | undefined;
   }): IThumbnailTrackInfo[] | null {
     if (this._priv_contentInfos === null || this._priv_contentInfos.manifest === null) {
       return null;
     }
-    const period = getPeriodForTime(this._priv_contentInfos.manifest, time);
-    if (period === undefined || period.thumbnailTracks.length === 0) {
-      return null;
+    const { manifest } = this._priv_contentInfos;
+    let period;
+    if (time !== undefined) {
+      period = getPeriodForTime(this._priv_contentInfos.manifest, time);
+      if (period === undefined || period.thumbnailTracks.length === 0) {
+        return null;
+      }
+    } else if (periodId !== undefined) {
+      period = arrayFind(manifest.periods, (p) => p.id === periodId);
+      if (period === undefined) {
+        log.error("API: getAvailableThumbnailTracks: periodId not found");
+        return null;
+      }
+    } else {
+      const { currentPeriod } = this._priv_contentInfos;
+      if (currentPeriod === null) {
+        return null;
+      }
+      period = currentPeriod;
     }
     return period.thumbnailTracks.map((t) => {
       return {
         id: t.id,
         width: Math.floor(t.width / t.horizontalTiles),
         height: Math.floor(t.height / t.verticalTiles),
         mimeType: t.mimeType,
+        start: t.start,
+        end: t.end,
+        thumbnailDuration: t.thumbnailDuration,
       };
     });
   }

src/manifest/classes/__tests__/adaptation.test.ts

@@ -51,6 +51,9 @@ const minimalRepresentationIndex: IRepresentationIndex = {
   addPredictedSegments() {
     /* noop */
   },
+  getTargetSegmentDuration() {
+    return undefined;
+  },
   _replace() {
     /* noop */
   },

src/manifest/classes/__tests__/representation.test.ts

@@ -47,6 +47,9 @@ const minimalIndex: IRepresentationIndex = {
   canBeOutOfSyncError(): true {
     return true;
   },
+  getTargetSegmentDuration() {
+    return undefined;
+  },
   _replace() {
     return;
   },

src/manifest/classes/__tests__/update_period_in_place.test.ts

@@ -174,6 +174,9 @@ function generateFakeThumbnailTrack({ id }: { id: string }) {
     width: 200,
     horizontalTiles: 5,
     verticalTiles: 3,
+    start: 0,
+    end: 100,
+    thumbnailDuration: 2,
     index: {
       _update() {
         /* noop */
@@ -1433,6 +1436,9 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-2",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          thumbnailDuration: 2,
           width: 200,
         },
       ],
@@ -1443,6 +1449,9 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          thumbnailDuration: 2,
           width: 200,
         },
       ],
@@ -1510,6 +1519,9 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-2",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          thumbnailDuration: 2,
           width: 200,
         },
       ],
@@ -1520,6 +1532,9 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          thumbnailDuration: 2,
           width: 200,
         },
       ],
@@ -1588,6 +1603,9 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          thumbnailDuration: 2,
           width: 200,
         },
       ],
@@ -1660,6 +1678,9 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          thumbnailDuration: 2,
           width: 200,
         },
       ],

src/manifest/classes/period.ts

@@ -145,6 +145,9 @@ export default class Period implements IPeriodMetadata {
       width: thumbnailTrack.width,
       horizontalTiles: thumbnailTrack.horizontalTiles,
       verticalTiles: thumbnailTrack.verticalTiles,
+      start: thumbnailTrack.start,
+      end: thumbnailTrack.end,
+      thumbnailDuration: thumbnailTrack.thumbnailDuration,
     }));
     this.duration = args.duration;
     this.start = args.start;
@@ -305,6 +308,9 @@ export default class Period implements IPeriodMetadata {
         width: thumbnailTrack.width,
         horizontalTiles: thumbnailTrack.horizontalTiles,
         verticalTiles: thumbnailTrack.verticalTiles,
+        start: thumbnailTrack.start,
+        end: thumbnailTrack.end,
+        thumbnailDuration: thumbnailTrack.thumbnailDuration,
       })),
     };
   }
@@ -344,4 +350,28 @@ export interface IThumbnailTrack {
    * images contained vertically in a whole loaded thumbnail resource.
    */
   verticalTiles: number;
+  /**
+   * Starting `position` the first thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  start: number | undefined;
+  /**
+   * Ending `position` the last thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  end: number | undefined;
+  /**
+   * If set, all those thumbnail tracks' thumbnails are linked to the given
+   * duration of content in seconds, going from `start`, until `end`.
+   *
+   * E.g. with a `start` set to `10`, an `end` set to `17`, and a
+   * `thumbnailDuration` set to `2`, there should be 4 thumbnails:
+   * 1. Applying to 10-12 seconds
+   * 2. Applying to 12-14 seconds
+   * 3. Applying to 14-16 seconds
+   * 4. Applying to 16-17 seconds
+   *
+   * Set to `undefined` if a duration cannot be determined.
+   */
+  thumbnailDuration: number | undefined;
 }

src/manifest/classes/representation_index/static.ts

@@ -155,6 +155,24 @@ export default class StaticRepresentationIndex implements IRepresentationIndex {
     log.error("A `StaticRepresentationIndex` does not need to be initialized");
   }
 
+  /**
+   * Returns the `duration` of each segment in the context of its Manifest (i.e.
+   * as the Manifest anounces them, actual segment duration may be different due
+   * to approximations), in seconds.
+   *
+   * NOTE: we could here do a median or a mean but I chose to be lazy (and
+   * more performant) by returning the duration of the first element instead.
+   * As `isPrecize` is `false`, the rest of the code should be notified that
+   * this is only an approximation.
+   * @returns {number}
+   */
+  getTargetSegmentDuration(): { duration: number; isPrecize: boolean } | undefined {
+    return {
+      duration: Number.MAX_VALUE,
+      isPrecize: false,
+    };
+  }
+
   addPredictedSegments(): void {
     log.warn("Cannot add predicted segments to a `StaticRepresentationIndex`");
   }

src/manifest/classes/representation_index/types.ts

@@ -429,6 +429,29 @@ export interface IRepresentationIndex {
    */
   initialize(segmentList: ISegmentInformation[]): void;
 
+  /**
+   * Returns an approximate for the duration of that `RepresentationIndex`s
+   * segments, in seconds in the context of its Manifest (i.e. as the Manifest
+   * anounces them, actual segment duration may be different due to
+   * approximations), with the exception of the last one (that usually is
+   * shorter).
+   * @returns {number}
+   */
+  getTargetSegmentDuration():
+    | {
+        /** Approximate duration of any segments but the last one in seconds. */
+        duration: number;
+        /**
+         * If `true`, the given duration should be relatively precize for all
+         * segments but the last one.
+         *
+         * If `false`, `duration` indicates only a general idea of what can be
+         * expected.
+         */
+        isPrecize: boolean;
+      }
+    | undefined;
+
   /**
    * Add segments to a RepresentationIndex that were predicted after parsing the
    * segment linked to `currentSegment`.

src/manifest/classes/update_period_in_place.ts

@@ -75,6 +75,9 @@ export default function updatePeriodInPlace(
       oldThumbnailTrack.width = newThumbnailTrack.width;
       oldThumbnailTrack.horizontalTiles = newThumbnailTrack.horizontalTiles;
       oldThumbnailTrack.verticalTiles = newThumbnailTrack.verticalTiles;
+      oldThumbnailTrack.start = newThumbnailTrack.start;
+      oldThumbnailTrack.end = newThumbnailTrack.end;
+      oldThumbnailTrack.thumbnailDuration = newThumbnailTrack.thumbnailDuration;
       oldThumbnailTrack.cdnMetadata = newThumbnailTrack.cdnMetadata;
       if (updateType === MANIFEST_UPDATE_TYPE.Full) {
         oldThumbnailTrack.index._replace(newThumbnailTrack.index);
@@ -88,6 +91,9 @@ export default function updatePeriodInPlace(
         width: oldThumbnailTrack.width,
         horizontalTiles: oldThumbnailTrack.horizontalTiles,
         verticalTiles: oldThumbnailTrack.verticalTiles,
+        start: oldThumbnailTrack.start,
+        end: oldThumbnailTrack.end,
+        thumbnailDuration: oldThumbnailTrack.thumbnailDuration,
       });
     }
   }
@@ -105,6 +111,9 @@ export default function updatePeriodInPlace(
         width: t.width,
         horizontalTiles: t.horizontalTiles,
         verticalTiles: t.verticalTiles,
+        start: t.start,
+        end: t.end,
+        thumbnailDuration: t.thumbnailDuration,
       })),
     );
     oldPeriod.thumbnailTracks.push(...newThumbnailTracks);

src/manifest/types.ts

@@ -286,6 +286,30 @@ export interface IThumbnailTrackMetadata {
    * images contained vertically in a whole loaded thumbnail resource.
    */
   verticalTiles: number;
+  /**
+   * Starting `position` the first thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  start: number | undefined;
+  /**
+   * Ending `position` the last thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  end: number | undefined;
+  /**
+   * If set, all those thumbnail tracks' thumbnails are linked to the given
+   * duration of content in seconds, going from `start`, until `end`.
+   *
+   * E.g. with a `start` set to `10`, an `end` set to `17`, and a
+   * `thumbnailDuration` set to `2`, there should be 4 thumbnails:
+   * 1. Applying to 10-12 seconds
+   * 2. Applying to 12-14 seconds
+   * 3. Applying to 14-16 seconds
+   * 4. Applying to 16-17 seconds
+   *
+   * Set to `undefined` if a duration cannot be determined.
+   */
+  thumbnailDuration: number | undefined;
 }
 
 export interface ILoadedThumbnailData {

src/parsers/manifest/dash/common/indexes/base.ts

@@ -433,6 +433,29 @@ export default class BaseRepresentationIndex implements IRepresentationIndex {
     log.warn("Cannot add predicted segments to a `BaseRepresentationIndex`");
   }
 
+  /**
+   * Returns the `duration` of each segment in the context of its Manifest (i.e.
+   * as the Manifest anounces them, actual segment duration may be different due
+   * to approximations), in seconds.
+   *
+   * NOTE: we could here do a median or a mean but I chose to be lazy (and
+   * more performant) by returning the duration of the first element instead.
+   * As `isPrecize` is `false`, the rest of the code should be notified that
+   * this is only an approximation.
+   * @returns {number}
+   */
+  getTargetSegmentDuration(): { duration: number; isPrecize: boolean } | undefined {
+    const { timeline, timescale } = this._index;
+    const firstElementInTimeline = timeline[0];
+    if (firstElementInTimeline === undefined) {
+      return undefined;
+    }
+    return {
+      duration: firstElementInTimeline.duration / timescale,
+      isPrecize: false,
+    };
+  }
+
   /**
    * Replace in-place this `BaseRepresentationIndex` information by the
    * information from another one.

src/parsers/manifest/dash/common/indexes/list.ts

@@ -348,6 +348,25 @@ export default class ListRepresentationIndex implements IRepresentationIndex {
     log.warn("Cannot add predicted segments to a `ListRepresentationIndex`");
   }
 
+  /**
+   * Returns the `duration` of each segment in the context of its Manifest (i.e.
+   * as the Manifest anounces them, actual segment duration may be different due
+   * to approximations), in seconds.
+   *
+   * NOTE: we could here do a median or a mean but I chose to be lazy (and
+   * more performant) by returning the duration of the first element instead.
+   * As `isPrecize` is `false`, the rest of the code should be notified that
+   * this is only an approximation.
+   * @returns {number}
+   */
+  getTargetSegmentDuration(): { duration: number; isPrecize: boolean } | undefined {
+    const { duration, timescale } = this._index;
+    return {
+      duration: duration / timescale,
+      isPrecize: true,
+    };
+  }
+
   /**
    * @param {Object} newIndex
    */

src/parsers/manifest/dash/common/indexes/template.ts

@@ -507,6 +507,22 @@ export default class TemplateRepresentationIndex implements IRepresentationIndex
     log.warn("Cannot add predicted segments to a `TemplateRepresentationIndex`");
   }
 
+  /**
+   * Returns the `duration` of each segment in the context of its Manifest (i.e.
+   * as the Manifest anounces them, actual segment duration may be different due
+   * to approximations), in seconds.
+   * @returns {number}
+   */
+  getTargetSegmentDuration(): {
+    duration: number;
+    isPrecize: boolean;
+  } {
+    return {
+      duration: this._index.duration / this._index.timescale,
+      isPrecize: true,
+    };
+  }
+
   /**
    * @param {Object} newIndex
    */