Reproduce previous results from color classifier [DO NOT MERGE]

[shiffman]

→ Describe your Pull Request 📝

For some reason, the implementation in ml5 does not produce the same results as the example I made previously (see: 02_Color_Classifier).

This is the loss I see in my previous example with the same data:

Trying a bunch of different things here, I am never able to get the loss to improve beyond tiny increments with the ml5.js version. (example found in: ml5js/ml5-examples#164 with colorData.json rather than small.)

So far in this branch, I've tried matching the following differences:

1. Even though the output is a label and one hot encoded into a tensor, I previously cast it as float32 rather than int32. Not sure this actually matters?
2. I used sigmoid and sgd instead of relu and adam.

But so far no improvement.

[shiffman]

Ack, sorry for all the whitespace fixing up. The only things changed (so far) are:

NeuralNetwork.js:

    const outputTensor = tf.tensor(targets).cast('float32');

index.js:

      case 'classification': // Create a model for classification
        // set classification model parameters
        console.log(this.config);
        this.config.hiddenUnits = 16;
        this.config.activationHidden = 'sigmoid' // 'relu',
        this.config.activationOutput = 'softmax' // 'relu',
        this.config.modelLoss = 'categoricalCrossentropy'
        this.config.modelOptimizer = tf.train.sgd(DEFAULTS.learningRate); // tf.train.adam();

        return this.createModelInternal();

[shiffman]

Noting I've investigated this a bit further with a lot of manual copying of code from the previous example to override ml5 code and still the same issues. . It may be something is going on in normalize(), taking a look at that now!

[shiffman]

This should definitely not be merged in its current state! I commented out all of the code in NeuralNetworkData's normalize() method and manually normalized and made the tensors in a different way. See: b9b6596 I now get the proper result:

@joeyklee maybe we can look over the normalize() function together to see where something might have gone wrong?

[shiffman]

Getting closer, shapes match but values do not!

[shiffman]

I believe the culprit is:

      .flatten()
      .reshape([this.data.length, this.meta.inputUnits]);

If we start with:

[a,b,c], [d,e,f]

which becomes:

[a,d], [b,e], [c,f]

then after flattening it is:

[a,d,b,e,c,f]

which becomes. .

[a,d,b],[e,c,f]

All is out of order! Ding ding!

But my train ride is over and I can't do the next step of testing this for sure!!! Back soon!

[shiffman]

Confirmed!

I can work on this later!

[joeyklee]

@shiffman - ooooh! I see. Interesting. Thanks for doing the detective work on this. I can try to debug this, but if you also are in deep, please go ahead!

[shiffman]

Thanks @joeyklee this should be an easy-ish fix now that I know what the problem is. . It may make sense to take a more manual approach to finding the mins/maxes (and we can skip mins and maxes for one hot encoded data) in order to avoid having to reshape the data back and forth?

I am focused on reviewing blog posts and other content for tomorrow's class but plan to work on this more afterwards!

[joeyklee]

@shiffman - I think I managed to fix this in #575 - Maybe have a look first there to see if this resolves this issue? thanks!

[joeyklee]

@shiffman - Thanks for the notes you've made here. I will close this now since these insights have been integrated into the current generic-neural-network branch.