Hi,
Thank you very much for this work.
I am trying to assess the contamination between different benchmarks. For example, between mmlu training set and mmlu-pro validation set (and vice versa). However, different benchmarks might have different format i.e different keys. And we can set only a global text_keys arg. Thus we run into the assertion error
Is there any way to set different text keys/key currently?
Hi,
Thank you very much for this work.
I am trying to assess the contamination between different benchmarks. For example, between mmlu training set and mmlu-pro validation set (and vice versa). However, different benchmarks might have different format i.e different keys. And we can set only a global text_keys arg. Thus we run into the assertion error
Is there any way to set different text keys/key currently?