Check list for Theano 1.0.0

[junpenglao]

Some user complain about recent slowdown of PyMC3, which I think might be theano 1.0 related. So here is an (incomplete) list of the things we should check/fix before upgrade to theano 1.0

• Fix Travis test (Unpin theano #2777)
• Benchmark potential compiling slowdown
• Check theano.clone and theano.scan related code, especially in VI (@ferrine)
• Remove theano op that are now availble in the release (e.g., https://github.com/pymc-devs/pymc3/blob/master/pymc3/distributions/dist_math.py#L240-L241 cc @aseyboldt) (Refactor Normal and MvNormal #2847)

Please add/edit if I miss anything.

[pwl]

Just for completeness, I include the link to the discussion on compilation slowdown. In my case the combination of pymc3 3.1 with theano 1.0 worked just fine (fast compilation), only after upgrading to pymc3 3.2 I noticed a big (10x) compilation slowdown. Now I have to wait minutes for my model to compile, while it was seconds in pymc3 3.1.

[pwl]

Not only the compilation is slower but it generates slower code altogether (I mean ADVI here). With pymc3 3.1 on a GPU I'm getting 134 it/s, after upgrading to 3.2 it's down to 109 it/s. On top of that the value of the loss function differs by an order of magnitude between versions (although the VI results do not). That's on a modified version of the VAE from here, perhaps the slowdowns are visible there too.

I wanted to test it on the original VAE notebook but for some reason the GPU version doesn't work directly on 3.1 (despite changing the local_RVs to a tuple according to the previous API). But still you can still see the difference in the loss function by running it on a CPU. Unfortunately I don't have much time to prepare a clean-cut test case.

[twiecki]

@pwl Can you check if the compilation slowdown is also present with 3.1 under theano 1.0? I.e. is it theano or pymc3.

[pwl]

All tests were done with theano 1.0.

[twiecki]

OK, so just to make absolutely sure: compilation times under theano 1.0 are much slower with pymc 3.2 than 3.1? Is this for ADVI only for also for NUTS?

[pwl]

Yes, it's compilation times for a GPU with pymc3 3.1 vs 3.2, both with theano 1.0, running ADVI. I can't say anything about NUTS.

[junpenglao]

With pymc3 3.1 on a GPU I'm getting 134 it/s, after upgrading to 3.2 it's down to 109 it/s.

This is normal right? as in 3.2 the VI module is much more complex.

[junpenglao]

FYI @pwl the advi slowdown has been fixed #2759

[junpenglao]

Theano 1.0.0 is set for the project, however, the remaining two points should still be addressed.

[junpenglao]

#2847 WIP for fixing the last point