Sequence encoder decoder

[basma-b]

I am using your decoder to implement my sequence encoder/decoder but actually i don't know how can I do to get the decoder output the same shape as my input. My input is (None, MAX_SEQ_LENGTH) these are my dialogue turns that I encode and the decode to get the next dialogue turn which has the same shape i.e (None, MAX_SEQ_LENGTH) although the decoder returns 3D dimension tensor because of return_sequences=True.

Can you help me please to do this ?

[basma-b]

Here is a small code

``

seq2seq = Sequential() # my turn shape=(None, MAX_SEQUENCE_LENGTH)
seq2seq.add(Embedding(output_dim=args.emb_dim,
                        input_dim=MAX_NB_WORDS,
                        input_length=MAX_SEQUENCE_LENGTH,
                        weights=[embedding_matrix],
                        mask_zero=True,
                        trainable=True))
seq2seq.add(LSTM(units=args.hidden_size, return_sequences=True))
seq2seq.add(AttentionDecoder(args.hidden_size, args.emb_dim)) # the decoded shape=(None, MAX_SEQUENCE_LENGTH, args.emb_dim)
seq2seq.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])

[zafarali]

Why is the shape (None,..)? Keras takes into account the batch size internally!

…

On Wed, Dec 6, 2017 at 4:51 AM Basma BOUSSAHA ***@***.***> wrote: Here is a small code `` seq2seq = Sequential() # my turn shape=(None, MAX_SEQUENCE_LENGTH) seq2seq.add(Embedding(output_dim=args.emb_dim, input_dim=MAX_NB_WORDS, input_length=MAX_SEQUENCE_LENGTH, weights=[embedding_matrix], mask_zero=True, trainable=True)) seq2seq.add(LSTM(units=args.hidden_size, return_sequences=True)) seq2seq.add(AttentionDecoder(args.hidden_size, args.emb_dim)) # the decoded shape=(None, MAX_SEQUENCE_LENGTH, args.emb_dim) seq2seq.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc']) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGAO_O4C9qLLJtzjULrdjmtd30vAw4yOks5s9o3agaJpZM4Q32jX> .

[basma-b]

No sure no, this is what is actually been done inside. It's just to explain but you can ignore it ..

[zafarali]

Ok. Have you tried using the graph API instead of sequential?

…

On Wed, Dec 6, 2017 at 7:24 AM Basma BOUSSAHA ***@***.***> wrote: No sure no, this is what is actually been done inside. It's just to explain but you can ignore it .. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGAO_GySyyD_4_avlFLNQKn1a4Y7ym3mks5s9rHLgaJpZM4Q32jX> .

[basma-b]

No, but can you give me more hints please ?

[zafarali]

A good intro is here: https://keras.io/getting-started/functional-api-guide/ I believe that the sequential API is just there for legacy purposes and should not be used anymore. This is an example of me using the graph API: https://github.com/datalogue/keras-attention/blob/master/models/NMT.py Does that help?

…

On Wed, Dec 6, 2017 at 7:27 AM Basma BOUSSAHA ***@***.***> wrote: No, but can you give me more hints please ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGAO_L9-xfV_sq9X7D8rCQnDi6GERcBsks5s9rJhgaJpZM4Q32jX> .

[basma-b]

Well I think the problem was not clear, but actually now I found a way to transform my decoder output to classes instead of probabilities using argmax. So now my output has the shape (MAX_SEQUENCE_LENGTH, ) the same as my inputs (I removed the batch size here).

Now I have another question: since my outputs are classes instead of probabilities which loss function could I use in this case.
example :
output labels = [0, 0, 1, 12, 165, 3]
predicted_labels = [0, 0, 13, 166, 3]

[zafarali]

Since the output is from a categorical distribution you should use the categorical cross entropy. You need the probabilities for this though. You willl need to convert your predicted labels into one hot vectors representing the class

…

On Wed, Dec 6, 2017 at 8:42 AM Basma BOUSSAHA ***@***.***> wrote: Well I think the problem was not clear, but actually now I found a way to transform my decoder output to classes instead of probabilities using argmax. So now my output has the shape (MAX_SEQUENCE_LENGTH, ) the same as my inputs (I removed the batch size here). Now I have another question: since my outputs are classes instead of probabilities which loss function could I use in this case. example : output labels = [0, 0, 1, 12, 165, 3] predicted_labels = [0, 0, 13, 166, 3] — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGAO_NNTqee_uCa_mVO9L3cey_rkKzj8ks5s9sPrgaJpZM4Q32jX> .

[basma-b]

You willl need to convert your predicted labels into one hot vectors representing the class

And how can I do this ?

[Ap1075]

For one hot encoding you could use the to_categorical function in keras, look it up in the documentation. To link your encoder to the attention decoder, you could use the RepeatVector layer to change the dimensions according to your requirements. Hope this helps.