Not quite matching Mogrifier evaluation results

[jkkummerfeld]

Hi,
I tried training and evaluating the Mogrifier LSTM on the standard PTB data using the commands described here:

https://github.com/deepmind/lamb/tree/master/lamb/experiment/mogrifier

The output contains:

I0427 15:09:37.012961 140302239332160 evaluation.py:73] final valid_det_t0.8800000000000001 xe: 3.96009 (73760.0), batches: 1054
I0427 20:18:13.086667 139838128674624 evaluation.py:255] final valid_mca_d0.6_t0.92 xe: 3.94770 (73760.0), batches: 1054 (ns=200)

Converting the cross-entropy values to perplexities I get:
e^3.96009 = 51.8
e^3.94770 = 52.5

The paper reports 51.4 and 52.1 respectively. At 0.4, the difference could just be due to some uncontrolled source of randomness, but I wanted to check if I was missing anything.

I set up the code as described in the repository. To provide full information about what I have installed, I've included the results from conda list below. I'm running with a single Titan X, which has 12 Gb of RAM (I did not need to change max_time_steps for it to run).

Thanks!

# packages in environment at /home/jkummerf/anaconda3/envs/tfp3.7:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_tflow_select             2.1.0                       gpu
absl-py                   0.9.0                    py37_0
astor                     0.8.0                    py37_0
blas                      1.0                         mkl
c-ares                    1.15.0            h7b6447c_1001
ca-certificates           2020.1.1                      0
certifi                   2020.4.5.1               py37_0
contextlib2               0.6.0.post1              pypi_0    pypi
cudatoolkit               10.0.130                      0
cudnn                     7.6.5                cuda10.0_0
cupti                     10.0.130                      0
dm-sonnet                 1.36                     pypi_0    pypi
dm-tree                   0.1.4                    pypi_0    pypi
gast                      0.2.2                    py37_0
google-pasta              0.2.0                      py_0
grpcio                    1.27.2           py37hf8bcb03_0
h5py                      2.10.0           py37h7918eee_0
hdf5                      1.10.4               hb1b8bf9_0
intel-openmp              2020.0                      166
keras-applications        1.0.8                      py_0
keras-preprocessing       1.1.0                      py_1
lamb                      1.0                       dev_0    <develop>
ld_impl_linux-64          2.33.1               h53a641e_7
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libprotobuf               3.11.4               hd408876_0
libstdcxx-ng              9.1.0                hdf63c60_0
markdown                  3.1.1                    py37_0
mkl                       2020.0                      166
mkl-service               2.3.0            py37he904b0f_0
mkl_fft                   1.0.15           py37ha843d7b_0
mkl_random                1.1.0            py37hd6b4f25_0
ncurses                   6.2                  he6710b0_0
numpy                     1.18.1           py37h4f9e942_0
numpy-base                1.18.1           py37hde5b4d6_1
openssl                   1.1.1g               h7b6447c_0
opt_einsum                3.1.0                      py_0
pip                       20.0.2                   py37_1
protobuf                  3.11.4           py37he6710b0_0
python                    3.7.7           hcf32534_0_cpython
readline                  8.0                  h7b6447c_0
scipy                     1.4.1            py37h0b6359f_0
semantic-version          2.8.4                    pypi_0    pypi
setuptools                46.1.3                   py37_0
six                       1.14.0                   py37_0
sqlite                    3.31.1               h62c20be_1
tabulate                  0.8.7                    pypi_0    pypi
tensorboard               1.15.0             pyhb230dea_0
tensorflow                1.15.0          gpu_py37h0f0df58_0
tensorflow-base           1.15.0          gpu_py37h9dcbed7_0
tensorflow-estimator      1.15.1             pyh2649769_0
tensorflow-gpu            1.15.0               h0d30ee6_0
termcolor                 1.1.0                    py37_1
tk                        8.6.8                hbc83047_0
webencodings              0.5.1                    py37_1
werkzeug                  0.16.1                     py_0
wheel                     0.34.2                   py37_0
wrapt                     1.12.1           py37h7b6447c_1
xz                        5.2.5                h7b6447c_0
zlib                      1.2.11               h7b6447c_3