Real-time Interactive Training with ml5.NeuralNetwork examples

[shiffman]

→ Description 📝

This is a new example that I plan to demonstrate in the IMA course and I also hope to make a video tutorial about it. This is inspired by @fiebrink1's work on wekinator and course Machine Learning for Musicians and Artists.

This is a very simple demonstration where with training data as follows:

  let frequency = 440;
  notePlayer.data.addData([mouseX, mouseY], [frequency)]);

Then the neural network predicts a frequency based on mouse position and plays the note using a p5.Oscillator object.

→ Screenshots 🖼

Here is a GIF demonstration (but you cannot hear the sound of course.)

[shiffman]

video quality aside here's a version with sound https://twitter.com/shiffman/status/1178445523709255681

[shiffman]

@joeyklee here's an interesting bug with ml5.NeuralNetwork.data.normalize() that I discovered while creating a version of the Rebecca Fiebrink Eyeo 2018 demo in ml5.js. In some cases, I would like to specify the range for normalization myself (rather than auto-detect it). For example. in the case of pixels I know the range is 0-255. But if a certain pixel color remains constant in the training data, it will pick that color as the minimum and maximum. Perhaps this is ok and it should just result in a value of 0.5. However, instead the normalize() function fails and returns NaN.

Here is some simple code to demonstrate:

function setup() {
  const options = {
    inputs: 2,
    outputs: 1,
  }
  const neural = ml5.neuralNetwork(options);
  neural.data.addData([100, 50], [1]);
  neural.data.addData([100, 100], [2]);
  neural.data.addData([100, 200], [2]);
  neural.data.normalize();
  neural.data.normalizedData.tensors.inputs.print();
}

Tensor
    [[NaN, 0        ],
     [NaN, 0.3333333],
     [NaN, 1        ]]

A version of the Fiebrink pixels example in progress will be pushed to this branch momentarily.

[shiffman]

I filtered out pixel values that don't change (all the alphas) and the example is working, see: https://twitter.com/shiffman/status/1178868124730449920 for a video.

[joeyklee]

@shiffman - wow! Nice examples. I am planning to start a refactor of the neural net class to see if maybe we can clean up the code base as well as integrate these considerations (plus all those we learned through this process). I'll point you to the refactor soon!

[joeyklee]

@shiffman Just noting that I've updated these examples in this PR together with: #195 as it relates to ml5js/ml5-library#591

[joeyklee]

@shiffman - I think we can close this branch now since these examples are integrated as part of #195. Feel free to close if/when you're ready.

[shiffman]

Great! Closing. . .

[shiffman]

@joeyklee oops, i think this was merged before I created the face example. I'm just going to add it back in via another branch I'm working in.