For experiment Runners in experiments\43_rigorous_benchmark\runners, all methods expect a decoded dictionary as input. Writing the encoded database to an h5 file and decoding per batch would alleviate memory limitations, enable practical training time comparisons, and allow comparison between raw and compressed h5 files for experimental compression values.
quicker memory fix
The dictionary approach allows for quicker general tests, but all decoded datasets are loaded into memory at once before running the first test. Reranging tests so that they are run after their respective decodings will ensure that only the necessary data is in memory for each test.
For experiment Runners in experiments\43_rigorous_benchmark\runners, all methods expect a decoded dictionary as input. Writing the encoded database to an h5 file and decoding per batch would alleviate memory limitations, enable practical training time comparisons, and allow comparison between raw and compressed h5 files for experimental compression values.
quicker memory fix
The dictionary approach allows for quicker general tests, but all decoded datasets are loaded into memory at once before running the first test. Reranging tests so that they are run after their respective decodings will ensure that only the necessary data is in memory for each test.