Scala Module MNIST MLP example not working

[gigasquid]

Description

Running the Scala example for Module MNIST is not working due to the new requirement in DataDesc from this PR b95ae7c#diff-196b8fe8b507164f1c13b556f178b7b2R233

Environment info (Required)

Mac OSX
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)

MXNet commit hash: 4044eb4

Build config:
USE_BLAS = openblas
ADD_CFLAGS += -I/usr/local/opt/openblas/include
ADD_LDFLAGS += -L/usr/local/opt/openblas/lib
ADD_LDFLAGS += -L/usr/local/lib/graphviz/

Error Message:

15:05 $ ./mnist_mlp.sh 
[main] INFO MXNetJVM - Try loading mxnet-scala from native path.
[main] INFO MXNetJVM - Try loading mxnet-scala-osx-x86_64-gpu from native path.
[main] INFO MXNetJVM - Try loading mxnet-scala-osx-x86_64-cpu from native path.
[main] WARN MXNetJVM - MXNet Scala native library not found in path. Copying native library from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH), or specifying by Java cmd option -Djava.library.path=[lib path].
[main] INFO org.apache.mxnet.util.NativeLibraryLoader - Replaced .dylib with .jnilib
[main] INFO org.apache.mxnet.util.NativeLibraryLoader - Loading libmxnet-scala.jnilib from /lib/native/ copying to mxnet-scala
[15:05:57] src/io/iter_mnist.cc:110: MNISTIter: load 60000 images, shuffle=1, shape=(10,784)
[15:05:58] src/io/iter_mnist.cc:110: MNISTIter: load 10000 images, shuffle=1, shape=(10,784)
[main] INFO org.apache.mxnetexamples.module.MnistMlp - Run intermediate level api from beginning.
[main] ERROR org.apache.mxnetexamples.module.MnistMlp - requirement failed: number of dimensions in shape :2 with shape: (10,784) should match the length of the layout: 4 with layout: NCHW
java.lang.IllegalArgumentException: requirement failed: number of dimensions in shape :2 with shape: (10,784) should match the length of the layout: 4 with layout: NCHW
    at scala.Predef$.require(Predef.scala:224)
    at org.apache.mxnet.DataDesc.<init>(IO.scala:233)
    at org.apache.mxnet.DataDesc$$anonfun$ListMap2Descs$1.apply(IO.scala:256)
    at org.apache.mxnet.DataDesc$$anonfun$ListMap2Descs$1.apply(IO.scala:256)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at org.apache.mxnet.DataDesc$.ListMap2Descs(IO.scala:256)
    at org.apache.mxnetexamples.module.MnistMlp$.runIntermediateLevelApi(MnistMlp.scala:52)
    at org.apache.mxnetexamples.module.MnistMlp$.main(MnistMlp.scala:150)
    at org.apache.mxnetexamples.module.MnistMlp.main(MnistMlp.scala)
 --batch-size N : the batch size for data iterator
 --data-dir VAL : the input data directory
 --num-epoch N  : number of training epoches
[Thread-0] INFO org.apache.mxnet.util.NativeLibraryLoader - Deleting /var/folders/cj/s4l2n2kn0ld5km979f23h83m0000gn/T/mxnet4424169528991588780/mxnet-scala
[Thread-0] INFO org.apache.mxnet.util.NativeLibraryLoader - Deleting /var/folders/cj/s4l2n2kn0ld5km979f23h83m0000gn/T/mxnet4424169528991588780

Steps to reproduce

(Paste the commands you ran that produced the error.)

• cd scala-package/examples/scripts/module
• ./mnist_mlp.sh

What have you tried to solve it?

1. I removed the require from https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L233 and recompiled. Test suite passed and example ran fine.

I also pinged @nswamy about it. He confirmed that the require should be there and that the examples should be updated.

[lanking520]

Hi @gigasquid , I am also facing the same issues when I try to deal with RNN example, I have add some fix here: #11753. Please see if this helps this problem as well

[lanking520]

This problem resolved by introducing __undefined__ field in the DataDesc. Please check the PR.

[gigasquid]

Thanks @lanking520 - Can this be closed now?

[lanking520]

@gigasquid Yes, this problem resolved!

[gigasquid]

Thanks!