Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix imageClassifier webgpu bug + add teachable machine image example #141

Merged
merged 4 commits into from
Jun 20, 2024

Conversation

ziyuan-linn
Copy link
Member

@ziyuan-linn ziyuan-linn commented Jun 18, 2024

This PR changes dataSync to data in the classifyInternal method in order to support webgpu backend. setBackend("webgl") is no longer needed. The end user API is not affected by this change.

This PR also adds a teachable machine image example as requested by @QuinnHe in ml5js/ml5-website-v02-docsify/issues/96. The only things I changed are removing setBackend("webgl") line and changing the color of the text for better visibility. This example should be live on the web editor once we merge this PR and run the p5 upload script.

@shiffman
Copy link
Member

This looks great! Picking up from #128, since Teachable Machine flips the webcam image during training let's include the flipped property in the example. Would you like to add this in before merging @ziyuan-linn? @sharellb you can use this as a model for the other examples if you want to work on those!

@QuinnHe
Copy link

QuinnHe commented Jun 18, 2024

Thanks Peter! This is awesome!

@ziyuan-linn
Copy link
Member Author

@shiffman @sharellb I included the flipped property for this example. For this example, only the p5 video had to be flipped. ImageClassifer doesn't deal with landmark points, so it does not need a flipped property. For a landmark model like handPose, just also add the flipped: true to its options object, and the landmark points should be flipped.

@shiffman
Copy link
Member

@shiffman @sharellb I included the flipped property for this example. For this example, only the p5 video had to be flipped. ImageClassifer doesn't deal with landmark points, so it does not need a flipped property. For a landmark model like handPose, just also add the flipped: true to its options object, and the landmark points should be flipped.

Ah, I see! If the p5.js flipped property is enabled, is the mirrored image actually passed to the model? Or would adding the flipped property additionally in ml5 enable that? While it will likely work regardless it would be good to be consistent in terms of what is visually displayed and what the model sees. The TM model will likely have been trained with flipped images so it may also be a bit more accurate by passing in flipped images.

@MOQN
Copy link
Member

MOQN commented Jun 19, 2024

If the p5.js flipped property is enabled, is the mirrored image actually passed to the model?

The original image is passed to the model.

@ziyuan-linn
Copy link
Member Author

@shiffman I did some quick testing and I think teachable machine actually trains on the original unmirrored webcam footage as well.

I trained a model with mirror camera enabled on teachable machine.
Screenshot 2024-06-19 152925

The same model loaded in to ml5 correctly classifies the underlying unmirrored video.
Screenshot 2024-06-19 153101

We probably don't have to change anything.

@shiffman
Copy link
Member

Oh, fantastic, thank you the thorough testing! Merging!

@shiffman shiffman merged commit d4d87bc into main Jun 20, 2024
@ziyuan-linn ziyuan-linn deleted the teachable-machine-imageClassifier-example branch July 3, 2024 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants