[MXNET-524] Broadcast like operator

[ifeherva]

Description

Operator, which can output a broadcasted array for the given target. This allows easier broadcasting and hybridization.

[lanking520]

Thanks for your contribution! @zhanghang1989, @eric-haibin-lin for review

[taliesinb]

There is a generalization that would be extremely useful for this operator to have. The generalization is very similar to one that was discussed at https://discuss.mxnet.io/t/reshaping-broadcasting-without-hardcoding-target-dimensions/851/6 (you can skip to the last 4 comments, the thread contains an irrelevant proposal although the motivation is relevant).

In short, the generalization would allow only specific dimensions to be copied from the 'other' tensor. For example:

input.shape = (1, 2, 1, 3)
other.shape = (5, 6, 7, 8)
output = broadcast_like(input, other, input_axes:(0,2), other_axes:(1,3))
output.shape = (6, 2, 8, 3)

In other words, what's happening here is that the you can pick exactly which axes of the other tensor you want to use to "fill in" axes of the input tensor. This is how broadcast_axes works, except instead of providing the values via a size parameter, you are providing them from specific axes in the other tensor.

The reason this is so valuable is that it is common to have another tensor that contains the dimension you want to broadcast amongst a set of irrelevant dimensions. There is simply no other way of "extracting" the relevant dimension from elsewhere in the net, so currently you have to hardcode that dimension into a parameter list, which forces expensive workarounds like bucketing where otherwise cheap reshaping would work to make a net that is compatible with multiple sequence lengths, for example.

The current behavior of broadcast_like in the PR would be consistent with this generalization if the default value of input_axis is the empty tuple, which means "all axes".

[zhanghang1989]

LGTM

[szha]

also need to add an entry at https://github.com/ifeherva/incubator-mxnet/blob/00eeeca61c9f052f3e85d4febe43130ed5669e61/docs/api/python/symbol/symbol.md#expanding-elements-1. Same goes with ndarray.

[ifeherva]

Done

[ifeherva]

@taliesinb Very interesting proposal indeed. The implementation is quite straighforward and I am happy to do it if this is something that is planned to happen. Is there a JIRA ticket open for this? I propose to have it in a separate PR.

[taliesinb]

@ifeherva if you're enthusiastic about this proposal that's great! yes, another PR might make sense. i'm not aware of a JIRA ticket, but the design of reshape_like is very similar to this proposal and that design was proposed by @piiswrong with the goal of solving the same kind of problem (my colleague @sbodenstein is going to submit a PR for that reshape_like extension in the next few days).

[ifeherva]

@taliesinb Great! Once that one is merged I can adapt broadcast_like as well.