New Features, some fixes and more

[cvalenzuela]

Implementing some new and exciting changes!

Image Classification and Feature Extraction

I've re-written most of the ImageClassification class, removing the custom video element and added support for tf.js custom Mobilenet class. This classifier now requires the name of the model to load, per ml5js/ml5-examples#26.

Following #128, I also created a new class called FeatureExtractor. The transfer learning example has changed a little bit now can be implemented as following:

  // Extract the already learned features from Mobilenet
  const featureExtractor = new ml5.FeatureExtractor('Mobilenet', modelReady);
  // Create a new classifier using those features and give the video we want to use
  const classifier = featureExtractor.asClassifier(video);

Here is a complete example.

There's also a regression method:

  // Extract the features from Mobilenet
  const featureExtractor = new ml5.FeatureExtractor('Mobilenet', modelReady);
  // Create a new regressor using those features and give the video we want to use
  const regressor = featureExtractor.asRegressor(video);

Here is a complete example.

I originally kept the ImageClassifier with the transfer learning capabilities (following ml5js/ml5-examples#26), but ended up moving it into the FeatureExtractor class. I think this makes more sense, it's more transparent on what the functionallity of this method is actually doing and does not changes or complicate the API.

Other changes to these methods are:

• You can now specify the accuracy of Mobilenet to use
• Added eventListeners for the current video instead of creating new video elements
• The FeatureExtractor method can support other models for sound and text.

PoseNet

• Fixes loading video on poseNet: PoseNet should wait for video to be loaded before running pose estimation #137

StyleTransfer

• Added support for p5js image and video elements
• Added async supporto to the transfer method. So no need to call setTimeOut on the examples
• A video can be passed in the constructor

YOLO

• Hosting the weights here now
• Updating the custom video creating and cleanup

Other things:

• Some cleanup in the README.md

This PR in examples has all this changes implemented: https://github.com/ml5js/ml5-examples

[shiffman]

This is awesome! So excited about these new features. In terms of the naming and API language I think it would be good to get a small group of whoever is available and hash some of this out on Tuesday. Conceptually I think this works well, but I'm concerned it's a little wordy or overly technical in language. It would be nice to have a simple transfer learning example that feels very approachable. We may end up where this is now as it's accurate, but worth trying out some other ideas I think? Here is a long list of probably worse possibilities:

const features = new ml5.Features('MobileNet');
const classifier = features.classification();
const predictor = features.regression();

const classifier = new ml5.Features('MobileNet').classifier();

const features = new ml5.Features('MobileNet');
const classifier = new ml5.ImageClassifier(features);

const classifier = new ml5.ImageClassifier();
classifier.transfer('MobileNet');

const classifier = new ml5.ImageClassifier('MobileNet');
const customClassifier = classifier.retrain();

const classifier = new ml5.ImageClassifier('MobileNet').retrain();

const classifier = new ml5.ImageClassifier('MobileNet');
classifier.retrain();

const classifier = new ml5.ImageClassifier('MobileNet');
const predictor = classifier.retrain('regression');

const classifier = new ml5.ImageClassifier('MobileNet');
const customClassifier = classifier.transferLearning();

More soon!

[shiffman]

@cvalenzuela I wonder if we should separate out the FeatureExtractor class into a separate branch (here and in ml5js/ml5-examples#29). We can merge all the smaller fixes and updates, but give ourselves a bit more time to sort out what makes sense with new API features?

[cvalenzuela]

Yep, we could. Although if we are planning on resolving the new API by Tuesday we could just wait. I'm not committing more changes to this repo for now, just working on the website.

Let me know if you if you prefer to separate into two branches or wait until Tuesday!

[shiffman]

Yes, I think we can wait! I am just overly excited. 😄

[nsthorat]

I like "Features" or "FeatureExtractor" on MobileNet for transfer learning, and creating a classifier or regression object from those features rather than say something directly about transfer learning, but it depends on how much you really want to hide the details.

Is the plan to hard code a layer of MobileNet for feature extraction so the user doesn't have to care?

[cvalenzuela]

I think we should not hide all details, but we should have a simple way of extracting a model's features without worrying about the layers names, types, epochs, batches, etc. If the user want's to step in and change those value it is still possible, but the first interaction should be kept simple.

[shiffman]

@cvalenzuela, @yining1023, and I just met and we are happy with this sequence, but not 100% sure about the naming. Feedback welcome, merging merging merging this now!

const features = new ml5.FeatureExtractor('MobileNet');

const classifier = features.classification();
classifier.addImage(img1, 'cat');
classifier.addImage(img2, 'dog');
classifier.train();
classifier.classify(img3, gotResult);

const predictor = features.regression();
predictor.addImage(img1, 0.1);
predictor.addImage(img2, 0.9);
predictor.train();
predictor.predict(img2, gotResult);

Also some more advanced use cases:

// Specify the layer you want
const features = new ml5.FeatureExtractor('MobileNet', layerNum);

features.predict(img, gotResult);

// Actually look at the raw output of the layer
function gotResults(outputs) {
  console.log(outputs);
}