MSA-4B Memory File Construction for RULER and LongBench Evaluation

[q1lin570]

I encountered difficulties when using MSA-4B for RULER and LONGBENCH evaluations. How should the memory_file be constructed?

In the MSA dataset, the data is divided into mdata (memory) and qdata (query). The mdata is already segmented into documents, which makes it straightforward to feed into the model. However, in benchmarks like RULER and LONGBENCH, the memory is provided as a single long context. How do you split this long context into documents? I believe it can have an impact on the results.

[c925777075]

We use a method of first dividing the document into chunks according to rules, and then merging adjacent chunks based on their embedding similarity.