FaceMesh faceOval points not in order around the face

[dano1234]

Here is a sketch with old ML5 facemesh making a mask
https://editor.p5js.org/dano/sketches/2ULaz2E0v

Here is a sketch with new ML5 facemesh making a mask
https://editor.p5js.org/dano/sketches/-3gKqxIZD

It appears that all the points for the "faceOval" property come in but they are not in a consecutive order around the face.

As ever thanks for this amazing library

[shiffman]

I took a quick look at this! I think the order of the points is coming directly from the tf.js model. This is the only documentation I could find.

We also recently merged #45 which added in bounding boxes and center points for all of the parts. I wonder if we should "re-sort" the faceOval points (or any other parts?) @gohai, is this something that you or any of the DURF students encountered?

I made this example many years ago to sort the vertices of a polygon in case it's helpful:

https://editor.p5js.org/codingtrain/sketches/GFlV5R9Ur

@ziyuan-linn maybe you want to take a look at this?

[ziyuan-linn]

Sounds great! I am happy to look into this.

I think the original facemesh came with an annotations object but the new one didn't, so we are now manually creating the annotations within ml5. Apparently, we forgot to take the order of the vertices into account. I will take a look at the old tfjs facemesh library to see how they generate the annotations object.

In addition to using a vertex-sorting algorithm, we could also consider manually creating a map so we can sort the vertices in linear time.

[gohai]

Thanks for the report @dano1234! This didn't come up, wasn't attempted, in the summer.

Turns out that MediaPipe, the new underlying model, returns points in a particular order, but for creating contours, we can't simply pair them up sequentially. Here is how Google's own implementation gets this done. And this is the source of these pairings. We should amend addKeypoints to take care of this for the contours it creates.

@ziyuan-linn @shiffman I'd be happy to address this Saturday morning (unless Peter already fixed this while I am writing this message 😅)

[gohai]

@dano1234 Should be fixed in upcoming test builds! (Here's a preview using a local ml5.js.)

We did change the API since the initial build: the points moved to .keypoints, so that every contour also has an (easier to use) .centerX, .centerY properties etc..

[dano1234]

Fantastic!