Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

What happened to the "skeleton" part of the bodyPose? #85

Closed
lindapaiste opened this issue Mar 9, 2024 · 4 comments · Fixed by #104
Closed

What happened to the "skeleton" part of the bodyPose? #85

lindapaiste opened this issue Mar 9, 2024 · 4 comments · Fixed by #104

Comments

@lindapaiste
Copy link
Contributor

I was looking at the migrating one of the examples that I wrote for the old poseNet model (ml5 #1386) and I realized that the new bodyPose model does not return the skeleton data -- telling us which pairs of keypoints should be connected. Should we add this in somehow? Do we get this data from the TensorFlow model?

We were previously getting these pairs from from a getAdjacentKeyPoints function on the TensorFlow posenet model.
https://github.com/ml5js/ml5-library/blob/f80f95aa6b31191ab7e79ff466e8506f5e48a172/src/PoseNet/index.js#L120

JSON from the old ml5 version:

{
    "pose": {
        "score": 0.9598300737493178,
        "keypoints": [
            {
                "score": 0.9982106685638428,
                "part": "nose",
                "position": {
                    "x": 234.7436927765724,
                    "y": 120.99073052313543
                }
            },
            {
                "score": 0.9967994689941406,
                "part": "leftEye",
                "position": {
                    "x": 244.4021276332989,
                    "y": 116.2922117478189
                }
            },
            {
                "score": 0.9952504634857178,
                "part": "rightEye",
                "position": {
                    "x": 225.1750576746603,
                    "y": 111.51711423109478
                }
            },
            {
                "score": 0.8623329997062683,
                "part": "leftEar",
                "position": {
                    "x": 260.36198021558477,
                    "y": 119.75604613189104
                }
            },
            {
                "score": 0.8424369692802429,
                "part": "rightEar",
                "position": {
                    "x": 208.37124743628596,
                    "y": 112.57503536239209
                }
            },
            {
                "score": 0.9977927207946777,
                "part": "leftShoulder",
                "position": {
                    "x": 268.2742455033477,
                    "y": 173.4225490288048
                }
            },
            {
                "score": 0.9977340698242188,
                "part": "rightShoulder",
                "position": {
                    "x": 185.7138446258664,
                    "y": 156.34151538893408
                }
            },
            {
                "score": 0.9940621256828308,
                "part": "leftElbow",
                "position": {
                    "x": 306.7087895137327,
                    "y": 228.746257499962
                }
            },
            {
                "score": 0.9967411160469055,
                "part": "rightElbow",
                "position": {
                    "x": 119.44014965606573,
                    "y": 195.0827477896724
                }
            },
            {
                "score": 0.9498299956321716,
                "part": "leftWrist",
                "position": {
                    "x": 352.618514064685,
                    "y": 194.567012682963
                }
            },
            {
                "score": 0.969875693321228,
                "part": "rightWrist",
                "position": {
                    "x": 123.3985939768038,
                    "y": 135.49304417413498
                }
            },
            {
                "score": 0.9958400130271912,
                "part": "leftHip",
                "position": {
                    "x": 247.96982743675147,
                    "y": 293.0668878221326
                }
            },
            {
                "score": 0.9959107041358948,
                "part": "rightHip",
                "position": {
                    "x": 178.35620396239287,
                    "y": 287.9360321282413
                }
            },
            {
                "score": 0.984212338924408,
                "part": "leftKnee",
                "position": {
                    "x": 247.36217650179736,
                    "y": 406.25373513782074
                }
            },
            {
                "score": 0.9769543409347534,
                "part": "rightKnee",
                "position": {
                    "x": 190.22527761496463,
                    "y": 411.3035043204341
                }
            },
            {
                "score": 0.8522323369979858,
                "part": "leftAnkle",
                "position": {
                    "x": 234.5086841509036,
                    "y": 518.1386816121261
                }
            },
            {
                "score": 0.9108952283859253,
                "part": "rightAnkle",
                "position": {
                    "x": 145.71858087680684,
                    "y": 455.45372270517316
                }
            }
        ],
        "nose": {
            "x": 234.7436927765724,
            "y": 120.99073052313543,
            "confidence": 0.9982106685638428
        },
        "leftEye": {
            "x": 244.4021276332989,
            "y": 116.2922117478189,
            "confidence": 0.9967994689941406
        },
        "rightEye": {
            "x": 225.1750576746603,
            "y": 111.51711423109478,
            "confidence": 0.9952504634857178
        },
        "leftEar": {
            "x": 260.36198021558477,
            "y": 119.75604613189104,
            "confidence": 0.8623329997062683
        },
        "rightEar": {
            "x": 208.37124743628596,
            "y": 112.57503536239209,
            "confidence": 0.8424369692802429
        },
        "leftShoulder": {
            "x": 268.2742455033477,
            "y": 173.4225490288048,
            "confidence": 0.9977927207946777
        },
        "rightShoulder": {
            "x": 185.7138446258664,
            "y": 156.34151538893408,
            "confidence": 0.9977340698242188
        },
        "leftElbow": {
            "x": 306.7087895137327,
            "y": 228.746257499962,
            "confidence": 0.9940621256828308
        },
        "rightElbow": {
            "x": 119.44014965606573,
            "y": 195.0827477896724,
            "confidence": 0.9967411160469055
        },
        "leftWrist": {
            "x": 352.618514064685,
            "y": 194.567012682963,
            "confidence": 0.9498299956321716
        },
        "rightWrist": {
            "x": 123.3985939768038,
            "y": 135.49304417413498,
            "confidence": 0.969875693321228
        },
        "leftHip": {
            "x": 247.96982743675147,
            "y": 293.0668878221326,
            "confidence": 0.9958400130271912
        },
        "rightHip": {
            "x": 178.35620396239287,
            "y": 287.9360321282413,
            "confidence": 0.9959107041358948
        },
        "leftKnee": {
            "x": 247.36217650179736,
            "y": 406.25373513782074,
            "confidence": 0.984212338924408
        },
        "rightKnee": {
            "x": 190.22527761496463,
            "y": 411.3035043204341,
            "confidence": 0.9769543409347534
        },
        "leftAnkle": {
            "x": 234.5086841509036,
            "y": 518.1386816121261,
            "confidence": 0.8522323369979858
        },
        "rightAnkle": {
            "x": 145.71858087680684,
            "y": 455.45372270517316,
            "confidence": 0.9108952283859253
        }
    },
    "skeleton": [
        [
            {
                "score": 0.9958400130271912,
                "part": "leftHip",
                "position": {
                    "x": 247.96982743675147,
                    "y": 293.0668878221326
                }
            },
            {
                "score": 0.9977927207946777,
                "part": "leftShoulder",
                "position": {
                    "x": 268.2742455033477,
                    "y": 173.4225490288048
                }
            }
        ],
        [
            {
                "score": 0.9940621256828308,
                "part": "leftElbow",
                "position": {
                    "x": 306.7087895137327,
                    "y": 228.746257499962
                }
            },
            {
                "score": 0.9977927207946777,
                "part": "leftShoulder",
                "position": {
                    "x": 268.2742455033477,
                    "y": 173.4225490288048
                }
            }
        ],
        [
            {
                "score": 0.9940621256828308,
                "part": "leftElbow",
                "position": {
                    "x": 306.7087895137327,
                    "y": 228.746257499962
                }
            },
            {
                "score": 0.9498299956321716,
                "part": "leftWrist",
                "position": {
                    "x": 352.618514064685,
                    "y": 194.567012682963
                }
            }
        ],
        [
            {
                "score": 0.9958400130271912,
                "part": "leftHip",
                "position": {
                    "x": 247.96982743675147,
                    "y": 293.0668878221326
                }
            },
            {
                "score": 0.984212338924408,
                "part": "leftKnee",
                "position": {
                    "x": 247.36217650179736,
                    "y": 406.25373513782074
                }
            }
        ],
        [
            {
                "score": 0.984212338924408,
                "part": "leftKnee",
                "position": {
                    "x": 247.36217650179736,
                    "y": 406.25373513782074
                }
            },
            {
                "score": 0.8522323369979858,
                "part": "leftAnkle",
                "position": {
                    "x": 234.5086841509036,
                    "y": 518.1386816121261
                }
            }
        ],
        [
            {
                "score": 0.9959107041358948,
                "part": "rightHip",
                "position": {
                    "x": 178.35620396239287,
                    "y": 287.9360321282413
                }
            },
            {
                "score": 0.9977340698242188,
                "part": "rightShoulder",
                "position": {
                    "x": 185.7138446258664,
                    "y": 156.34151538893408
                }
            }
        ],
        [
            {
                "score": 0.9967411160469055,
                "part": "rightElbow",
                "position": {
                    "x": 119.44014965606573,
                    "y": 195.0827477896724
                }
            },
            {
                "score": 0.9977340698242188,
                "part": "rightShoulder",
                "position": {
                    "x": 185.7138446258664,
                    "y": 156.34151538893408
                }
            }
        ],
        [
            {
                "score": 0.9967411160469055,
                "part": "rightElbow",
                "position": {
                    "x": 119.44014965606573,
                    "y": 195.0827477896724
                }
            },
            {
                "score": 0.969875693321228,
                "part": "rightWrist",
                "position": {
                    "x": 123.3985939768038,
                    "y": 135.49304417413498
                }
            }
        ],
        [
            {
                "score": 0.9959107041358948,
                "part": "rightHip",
                "position": {
                    "x": 178.35620396239287,
                    "y": 287.9360321282413
                }
            },
            {
                "score": 0.9769543409347534,
                "part": "rightKnee",
                "position": {
                    "x": 190.22527761496463,
                    "y": 411.3035043204341
                }
            }
        ],
        [
            {
                "score": 0.9769543409347534,
                "part": "rightKnee",
                "position": {
                    "x": 190.22527761496463,
                    "y": 411.3035043204341
                }
            },
            {
                "score": 0.9108952283859253,
                "part": "rightAnkle",
                "position": {
                    "x": 145.71858087680684,
                    "y": 455.45372270517316
                }
            }
        ],
        [
            {
                "score": 0.9977927207946777,
                "part": "leftShoulder",
                "position": {
                    "x": 268.2742455033477,
                    "y": 173.4225490288048
                }
            },
            {
                "score": 0.9977340698242188,
                "part": "rightShoulder",
                "position": {
                    "x": 185.7138446258664,
                    "y": 156.34151538893408
                }
            }
        ],
        [
            {
                "score": 0.9958400130271912,
                "part": "leftHip",
                "position": {
                    "x": 247.96982743675147,
                    "y": 293.0668878221326
                }
            },
            {
                "score": 0.9959107041358948,
                "part": "rightHip",
                "position": {
                    "x": 178.35620396239287,
                    "y": 287.9360321282413
                }
            }
        ]
    ]
}

JSON from the new ml5 version:

{
    "keypoints": [
        {
            "y": 116.29785878956318,
            "x": 235.0086233296345,
            "score": 0.7886537313461304,
            "name": "nose"
        },
        {
            "y": 109.22195221483707,
            "x": 247.31669480038673,
            "score": 0.8181501030921936,
            "name": "left_eye"
        },
        {
            "y": 104.87190327048302,
            "x": 227.84886195487582,
            "score": 0.7873461246490479,
            "name": "right_eye"
        },
        {
            "y": 121.65976269543171,
            "x": 258.62494331045247,
            "score": 0.8321802616119385,
            "name": "left_ear"
        },
        {
            "y": 107.55718423426151,
            "x": 212.17497975555892,
            "score": 0.8760896325111389,
            "name": "right_ear"
        },
        {
            "y": 168.72356361150742,
            "x": 268.7925813615936,
            "score": 0.8404656052589417,
            "name": "left_shoulder"
        },
        {
            "y": 152.61473408341408,
            "x": 178.79237256099265,
            "score": 0.8981107473373413,
            "name": "right_shoulder"
        },
        {
            "y": 226.47244331240654,
            "x": 299.3288135528564,
            "score": 0.7384112477302551,
            "name": "left_elbow"
        },
        {
            "y": 194.91725680232048,
            "x": 117.27373925680965,
            "score": 0.9358187913894653,
            "name": "right_elbow"
        },
        {
            "y": 202.76897567510605,
            "x": 348.5035130903893,
            "score": 0.8098330497741699,
            "name": "left_wrist"
        },
        {
            "y": 140.22100810706615,
            "x": 123.04792117826716,
            "score": 0.8564737439155579,
            "name": "right_wrist"
        },
        {
            "y": 299.423077493906,
            "x": 245.70130230225234,
            "score": 0.9197857975959778,
            "name": "left_hip"
        },
        {
            "y": 299.77109426259995,
            "x": 183.46146969451115,
            "score": 0.9340135455131531,
            "name": "right_hip"
        },
        {
            "y": 411.4370810389519,
            "x": 238.26926617278264,
            "score": 0.8583171963691711,
            "name": "left_knee"
        },
        {
            "y": 416.5619515180588,
            "x": 182.75880041810655,
            "score": 0.8068801164627075,
            "name": "right_knee"
        },
        {
            "y": 517.5497948527336,
            "x": 244.53618868110104,
            "score": 0.8860065937042236,
            "name": "left_ankle"
        },
        {
            "y": 449.2491074204445,
            "x": 139.96030905812054,
            "score": 0.15222936868667603,
            "name": "right_ankle"
        }
    ],
    "box": {
        "yMin": 0.1048629879951477,
        "xMin": 0.1899825632572174,
        "yMax": 0.9370519518852234,
        "xMax": 0.7449000477790833,
        "width": 0.5549174845218658,
        "height": 0.8321889638900757
    },
    "score": 0.7778568267822266,
    "id": 1,
    "nose": {
        "x": 235.0086233296345,
        "y": 116.29785878956318,
        "score": 0.7886537313461304
    },
    "left_eye": {
        "x": 247.31669480038673,
        "y": 109.22195221483707,
        "score": 0.8181501030921936
    },
    "right_eye": {
        "x": 227.84886195487582,
        "y": 104.87190327048302,
        "score": 0.7873461246490479
    },
    "left_ear": {
        "x": 258.62494331045247,
        "y": 121.65976269543171,
        "score": 0.8321802616119385
    },
    "right_ear": {
        "x": 212.17497975555892,
        "y": 107.55718423426151,
        "score": 0.8760896325111389
    },
    "left_shoulder": {
        "x": 268.7925813615936,
        "y": 168.72356361150742,
        "score": 0.8404656052589417
    },
    "right_shoulder": {
        "x": 178.79237256099265,
        "y": 152.61473408341408,
        "score": 0.8981107473373413
    },
    "left_elbow": {
        "x": 299.3288135528564,
        "y": 226.47244331240654,
        "score": 0.7384112477302551
    },
    "right_elbow": {
        "x": 117.27373925680965,
        "y": 194.91725680232048,
        "score": 0.9358187913894653
    },
    "left_wrist": {
        "x": 348.5035130903893,
        "y": 202.76897567510605,
        "score": 0.8098330497741699
    },
    "right_wrist": {
        "x": 123.04792117826716,
        "y": 140.22100810706615,
        "score": 0.8564737439155579
    },
    "left_hip": {
        "x": 245.70130230225234,
        "y": 299.423077493906,
        "score": 0.9197857975959778
    },
    "right_hip": {
        "x": 183.46146969451115,
        "y": 299.77109426259995,
        "score": 0.9340135455131531
    },
    "left_knee": {
        "x": 238.26926617278264,
        "y": 411.4370810389519,
        "score": 0.8583171963691711
    },
    "right_knee": {
        "x": 182.75880041810655,
        "y": 416.5619515180588,
        "score": 0.8068801164627075
    },
    "left_ankle": {
        "x": 244.53618868110104,
        "y": 517.5497948527336,
        "score": 0.8860065937042236
    },
    "right_ankle": {
        "x": 139.96030905812054,
        "y": 449.2491074204445,
        "score": 0.15222936868667603
    }
}
@lindapaiste
Copy link
Contributor Author

I'm looking at the @tfjs-model now and I'm not seeing any equivalent to the getAdjacentKeyPoints method. If we wanted to implement this ourselves, we can look at the source from the old PoseNet. It's pretty simple. We would need to:

  1. Define the pairings of connected points (names or indices) for each model
const connectedPartNames: StringTuple[] = [
  ['leftHip', 'leftShoulder'], ['leftElbow', 'leftShoulder'],
  ['leftElbow', 'leftWrist'], ['leftHip', 'leftKnee'],
  ['leftKnee', 'leftAnkle'], ['rightHip', 'rightShoulder'],
  ['rightElbow', 'rightShoulder'], ['rightElbow', 'rightWrist'],
  ['rightHip', 'rightKnee'], ['rightKnee', 'rightAnkle'],
  ['leftShoulder', 'rightShoulder'], ['leftHip', 'rightHip']
];

export const connectedPartIndices = connectedPartNames.map(
    ([jointNameA, jointNameB]) => ([partIds[jointNameA], partIds[jointNameB]]));

https://github.com/tensorflow/tfjs-models/blob/facabba571b36ad63e172649c2420cc002cb8a71/posenet/src/keypoints.ts

  1. Map these pairings to the detected points

  2. Remove any connections where one or both points was not found, or was low-confidence.

function eitherPointDoesntMeetConfidence(
    a: number, b: number, minConfidence: number): boolean {
  return (a < minConfidence || b < minConfidence);
}

export function getAdjacentKeyPoints(
    keypoints: Keypoint[], minConfidence: number): Keypoint[][] {
  return connectedPartIndices.reduce(
      (result: Keypoint[][], [leftJoint, rightJoint]): Keypoint[][] => {
        if (eitherPointDoesntMeetConfidence(
                keypoints[leftJoint].score, keypoints[rightJoint].score,
                minConfidence)) {
          return result;
        }


        result.push([keypoints[leftJoint], keypoints[rightJoint]]);


        return result;
      }, []);
}

https://github.com/tensorflow/tfjs-models/blob/facabba571b36ad63e172649c2420cc002cb8a71/posenet/src/util.ts#L23-L42

@lindapaiste lindapaiste mentioned this issue Mar 9, 2024
Draft
@shiffman
Copy link
Member

shiffman commented Mar 11, 2024 •
edited
Loading

Ah yes, I think we decided not to include this since it wasn't part of what the native model anymore. I can see how it could be nice to bring back! From my own experience, I very rarely had students use the skeleton connections, the actual points are almost always what they focused on. @MOQN I know you taught quite a bit with PoseNet, what do you think? Is it worth the extra effort and maintenance required to add this data into the model output?

@ziyuan-linn also, if you recall anything different from our conversations, let us know!

@ziyuan-linn
Copy link
Member

Yes, I also remember that we decided to not include it. If I recall correctly we were also planning on making an example p5 sketch that draws the skeleton connections manually.

@MOQN
Copy link
Member

MOQN commented Mar 13, 2024

I agree with the approach! Since many students tend to visualize the skeleton using their own aesthetics, providing an example of manually drawing individual skeletal parts could be helpful for our students.

@ziyuan-linn ziyuan-linn linked a pull request Mar 21, 2024 that will close this issue
bodyPose skeleton examples #104
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bodyPose skeleton examples ml5js/ml5-next-gen
4 participants
@shiffman @MOQN @lindapaiste @ziyuan-linn