Memory Usage Problem

[NikcPoon]

Hi Kelvin

Thans for the great work! I try the default editor on personal dataset and it works well. However during the training I found that the memory usage increasingly goes up, which leads to the experiment being killed after some training steps. Is it because I run the model in a wrong way?(I just prepare the dataset and use main.py in editor model fold.) And I want to know how to fix this problem.
Thanks

[YadiLao]

Hi NikcPoon,
I try to run default editor as well, but I encounter some bugs when training. Can you tell me the way you run?

[NikcPoon]

@YadiLao
Hi YadiLao, I just run the main.py in edit_model/. I think it will be helpful if you provide some Traceback messages

[Vonzpf]

@NikcPoon
Hi NikcPoon, I had finished the training on onebillion dataset. My memory usage is stable during the training and it only increases quickly in data reading process.
I have a question that how do you begin the test? Dose the training contain results？
I just found some outputs in neural-editor-data/edit_runs/0/stdout.txt like this:

SOURCE: an e - mail message seeking comment from the club was not immediately returned .
INSERT: [u"'s", u'group', u'publicist', u'sent', u'to']
DELETE: [u'club', u'comment', u'from', u'message', u'seeking']
TARGET: an e - mail sent to the group 's publicist was not immediately returned .
(0.02) an e - mail message seeking comment from the company was not immediately answered .
(0.02) an e - mail message seeking comment from the company was not immediately returned .
(0.02) an e - mail message seeking comment was not immediately returned .
(0.01) an e - mail message seeking comment from was not immediately returned .
(0.01) an e - mail message seeking comment from the company was not immediately returned .

[NickPoon]

@Vonzpf
Hi Vonzpf. If I remember correctly, the training will log some result from random pick data. Actually I wrote my own test script. The general idea is that you can use editor.edit() to do inference.

[Vonzpf]

@NikcPoon I understand it. Thank you for your help!