Ablation test results do not match with those presented in the paper

[duanqn]

Hi,
I am trying to verify the experiment results presented in the original paper. I made some changes to the code in order to perform the ablation test. My code is on https://github.com/duanqn/PRPN/tree/LMparse
diff: master...duanqn:LMparse
I'm doing the first ablation test, which is to remove parsing network and replace structured attention with normal ones.
I have run the original code on Penn Treebank dataset, getting 61.53 perplexity (which is similar to 61.98 in the paper). After removing the parsing network, I got 62.18 test PPL which is different from 64.42 in the paper.
Right now I'm not sure if I modified the network correctly. Could you please show me the right way to change the structure of the network?

Thanks,
Qingnan

[justinchiu]

I wonder if the ablation results would be less finicky on a larger corpus which isn’t so regularization dependent