Discussion: ImageClassifier

[lindapaiste]

I'm opening this thread as a place for us to discuss ongoing work on the ImageClassifier model. Some of the tasks are:

• Decide where the user should specify the number of classes to return (the topk variable). Should it be when they create the model? When they call classify/classifyStart? Or both? In the new version of the detector models we eliminated the ability to pass options to the detect function. Following that example we would support ml5.imageClassifier('doodlenet', { topk: 5 }) but not myClassifier.classify(image, 5). Right now passing the option to ml5.imageClassifier() only works for mobilenet.
• Separate handling which only applies to one model type Refactor: breakup ImageClassifier ml5-library#1362
• Figure out where we want to handle conversion to Tensor, related to Allow tensor as input image ml5-library#1396
• Clean up preprocessing code using the utilities created in Fix darknet classifier, refactor preprocessing ml5-library#1398
• See if there are parts of the legacy code that don't make sense. For example we look for a property ml5Specs in the model.json file for custom models. That property would only be set if the user trained their model using the ml5 FeatureExtractor, which we haven't ported over here. But users may have trained models on the old version so I guess we should keep it?
• Look into what a model from TeachableMachine actually looks like and see if we are handling it properly. Is the image size always 224? Is our range of normalized values correct? Can we infer these values from the model in case the user has a custom trained model which doesn't match our expectations?
• Write unit tests for all model types which make sure that the model gives the correct prediction. There's a lot in https://github.com/ml5js/ml5-library/blob/b4960e9f8d5673830b6e149ad9da0e3548ec636a/src/ImageClassifier/index.test.js but even that is incomplete.

[jackbdu]

To add to this list: shall we allow relative path for loading custom Teachable Machine models? isHttpUrl() in ImageClassifier is helpful in validating urls, but it doesn't accept relative paths like ./my-model/mode.json, so one has to either upload their trained model on Teachable Machine and use the absolute url, or host the model locally and use an absolute url that only works locally (e.g. http://127.0.0.1:8080/my-model/mode.json). Or maybe I'm not using it properly?

[shiffman]

Hi @jackbdu, yes, I agree we should support loading of local model files! The neural network module has support for this (see this old tutorial that I believe should still work with ml5 1.0). I'll note that the p5 web editor opens up some additional complexity given how "local" files are stored via an S3 bucket.

I would suggest opening a new issue to track this funcitonality specifically. @OrpheasK, do you know what the current, expected behavior is for loading a model not from a URL?

[OrpheasK]

@shiffman I believe this is currently not supported (the ImageClassifier only handles URLs beyond the four model names) but I can look into this! I have only used a URL to load TM models but I can see how it makes sense to load locally as well, especially if it's a functionality supported by other modules.

[jackbdu]

I was also wondering what it would entail for the p5 web editor. A temperoray solution (live demo, source code) that worked for me for both local servers and GitHub pages (not for p5 web editor) is classifier = ml5.imageClassifier(`${window.location.href}my-tm-model/`);. Since Teachable Machine offers the option to download models, I was just exploring ways to use the downloaded models. If we do choose to support this, it would be ideal to find ways to support p5 web editor as well.