FrozenRecord was never meant to handle "large" datasets. However it's not always evident what a large dataset is, and it's also easy for a dataset to grow over time and severely impact performance.
It's also often not obvious when an index could help.
Reject slow queries in test
For the discoverability of performance issues, I think we should add an option "slow query" limit, that can be turned on in test environments. e.g. FrozenRecord.max_scan_size = 1000 that would immediately raise if you do a query that end up scanning more than 1k records.
This should prompt owners to either find a proper index, or to use something other than FrozenRecord.
That one is totally trivial to implement.
Sqlite backend?
For cases where there's just no possible indexes, we could explore a more efficient storage. I've seen some app generate static sqlite dbs on deploy, the usability was a bit shit, but I think we could have a FrozenRecord backend that pretty much does this automatically with little to no change to the interface.
That one need some research, it may or may not work.
cc @alexcwatt
FrozenRecord was never meant to handle "large" datasets. However it's not always evident what a large dataset is, and it's also easy for a dataset to grow over time and severely impact performance.
It's also often not obvious when an index could help.
Reject slow queries in test
For the discoverability of performance issues, I think we should add an option "slow query" limit, that can be turned on in test environments. e.g.
FrozenRecord.max_scan_size = 1000that would immediately raise if you do a query that end up scanning more than 1k records.This should prompt owners to either find a proper index, or to use something other than FrozenRecord.
That one is totally trivial to implement.
Sqlite backend?
For cases where there's just no possible indexes, we could explore a more efficient storage. I've seen some app generate static sqlite dbs on deploy, the usability was a bit shit, but I think we could have a FrozenRecord backend that pretty much does this automatically with little to no change to the interface.
That one need some research, it may or may not work.
cc @alexcwatt