Performance Gap (significantly lower than reported)

[convnets]

Hi,

I have tried to reproduce the reported result. However, my results are lower than the paper claimed. The results are shown below:

Evaluating on val_seen env ...
Epoch: [205][1/16]      Time 44.433 (44.433)    Loss 0.0000 (0.0000)
Epoch: [205][2/16]      Time 41.156 (42.794)    Loss 0.0000 (0.0000)
Epoch: [205][3/16]      Time 40.692 (42.093)    Loss 0.0000 (0.0000)
Epoch: [205][4/16]      Time 49.232 (43.878)    Loss 0.0000 (0.0000)
Epoch: [205][5/16]      Time 56.059 (46.314)    Loss 0.0000 (0.0000)
Epoch: [205][6/16]      Time 50.356 (46.988)    Loss 0.0000 (0.0000)
Epoch: [205][7/16]      Time 38.051 (45.711)    Loss 0.0000 (0.0000)
Epoch: [205][8/16]      Time 38.715 (44.837)    Loss 0.0000 (0.0000)
Epoch: [205][9/16]      Time 37.932 (44.070)    Loss 0.0000 (0.0000)
Epoch: [205][10/16]     Time 38.539 (43.516)    Loss 0.0000 (0.0000)
Epoch: [205][11/16]     Time 38.241 (43.037)    Loss 0.0000 (0.0000)
Epoch: [205][12/16]     Time 47.775 (43.432)    Loss 0.0000 (0.0000)
Epoch: [205][13/16]     Time 54.741 (44.302)    Loss 0.0000 (0.0000)
Epoch: [205][14/16]     Time 52.321 (44.875)    Loss 0.0000 (0.0000)
Epoch: [205][15/16]     Time 38.056 (44.420)    Loss 0.0000 (0.0000)
Epoch: [205][16/16]     Time 37.984 (44.018)    Loss 0.0000 (0.0000)
| nav_error: 3.8490198931233213 | oracle_error: 2.3369494706951466 | steps: 6.107843137254902 | lengths: 10.45375464930404 | spl: 0.6103185273231977 | success_rate: 0.6460784313725491 | oracle_rate: 0.7294117647058823
Evaluating on val_unseen env ...
Epoch: [205][1/37]      Time 38.728 (38.728)    Loss 0.0000 (0.0000)
Epoch: [205][2/37]      Time 36.856 (37.792)    Loss 0.0000 (0.0000)
Epoch: [205][3/37]      Time 36.631 (37.405)    Loss 0.0000 (0.0000)
Epoch: [205][4/37]      Time 37.941 (37.539)    Loss 0.0000 (0.0000)
Epoch: [205][5/37]      Time 36.848 (37.401)    Loss 0.0000 (0.0000)
Epoch: [205][6/37]      Time 36.870 (37.312)    Loss 0.0000 (0.0000)
Epoch: [205][7/37]      Time 37.395 (37.324)    Loss 0.0000 (0.0000)
Epoch: [205][8/37]      Time 36.739 (37.251)    Loss 0.0000 (0.0000)
Epoch: [205][9/37]      Time 37.376 (37.265)    Loss 0.0000 (0.0000)
Epoch: [205][10/37]     Time 36.645 (37.203)    Loss 0.0000 (0.0000)
Epoch: [205][11/37]     Time 36.693 (37.156)    Loss 0.0000 (0.0000)
Epoch: [205][12/37]     Time 37.355 (37.173)    Loss 0.0000 (0.0000)
Epoch: [205][13/37]     Time 36.837 (37.147)    Loss 0.0000 (0.0000)
Epoch: [205][14/37]     Time 37.657 (37.184)    Loss 0.0000 (0.0000)
Epoch: [205][15/37]     Time 37.261 (37.189)    Loss 0.0000 (0.0000)
Epoch: [205][16/37]     Time 36.708 (37.159)    Loss 0.0000 (0.0000)
Epoch: [205][17/37]     Time 36.177 (37.101)    Loss 0.0000 (0.0000)
Epoch: [205][18/37]     Time 37.224 (37.108)    Loss 0.0000 (0.0000)
Epoch: [205][19/37]     Time 36.919 (37.098)    Loss 0.0000 (0.0000)
Epoch: [205][20/37]     Time 36.471 (37.067)    Loss 0.0000 (0.0000)
Epoch: [205][21/37]     Time 37.841 (37.103)    Loss 0.0000 (0.0000)
Epoch: [205][22/37]     Time 36.392 (37.071)    Loss 0.0000 (0.0000)
Epoch: [205][23/37]     Time 37.118 (37.073)    Loss 0.0000 (0.0000)
Epoch: [205][24/37]     Time 37.091 (37.074)    Loss 0.0000 (0.0000)
Epoch: [205][25/37]     Time 36.840 (37.064)    Loss 0.0000 (0.0000)
Epoch: [205][26/37]     Time 36.788 (37.054)    Loss 0.0000 (0.0000)
Epoch: [205][27/37]     Time 37.286 (37.062)    Loss 0.0000 (0.0000)
Epoch: [205][28/37]     Time 36.880 (37.056)    Loss 0.0000 (0.0000)
Epoch: [205][29/37]     Time 37.105 (37.058)    Loss 0.0000 (0.0000)
Epoch: [205][30/37]     Time 37.046 (37.057)    Loss 0.0000 (0.0000)
Epoch: [205][31/37]     Time 43.863 (37.277)    Loss 0.0000 (0.0000)
Epoch: [205][32/37]     Time 44.296 (37.496)    Loss 0.0000 (0.0000)
Epoch: [205][33/37]     Time 38.315 (37.521)    Loss 0.0000 (0.0000)
Epoch: [205][34/37]     Time 38.055 (37.537)    Loss 0.0000 (0.0000)
Epoch: [205][35/37]     Time 37.899 (37.547)    Loss 0.0000 (0.0000)
Epoch: [205][36/37]     Time 47.541 (37.825)    Loss 0.0000 (0.0000)
Epoch: [205][37/37]     Time 54.212 (38.267)    Loss 0.0000 (0.0000)
| nav_error: 6.141163837885611 | oracle_error: 3.6298467382790323 | steps: 6.171988080034057 | lengths: 10.225189720171494 | spl: 0.3966670362847731 | success_rate: 0.44785014899957426 | oracle_rate: 0.5798212005108557

In the paper, Table 2 (row number 7), the expected result should be
val_seen (NE | SR | OSR | SPL): 3.23 | 0.70 | 0.78 | 0.66 val_unseen(NE | SR | OSR | SPL): 5.04 | 0.57 | 0.70 | 0.51. However, I obtained val_seen SPL 0.61, 5% lower and val_unseen SPL 0.396, 12% lower.

My configurations are posted as follows:

# Name                    Version                   Build  Channel
python                    3.8.2                hcf32534_0
pytorch                   1.4.0           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
numpy                     1.18.1           py38h4f9e942_0
networkx                  2.4                      pypi_0    pypi
torchvision               0.5.0                py38_cu101    pytorch

[ZhangTianrong]

I am facing the same problem here yielding almost same perforance with you. Have you found any clue?

[rmant]

Same here

[convnets]

I am facing the same problem here yielding almost same perforance with you. Have you found any clue?

I still do not have any clue. If you find the answer, can you share with us here ? This is really wired.

[convnets]

Same here

If you find the answer, can you share with us here ? This is really wired.