Problem
prefetch_parquet_file_metadata_for_ir fetches parquet footers from S3 at the start of every query execution, even for files that were already read in a previous iteration of the same benchmark or a previous query against the same dataset. The CachedParquetInfo objects (which include the parsed FileMetaData and pre-built HybridScanMetadata, and open kvikio handles as of #23317) are discarded when the query finishes.
From profiling Q1 at SF300 on a g7e.8xlarge:
It takes a second to prefetch the parquet metadata on both iterations. But the second iteration should not pay that cost.
Proposed Fix
Add a session-level cache of CachedParquetInfo objects on the engine. Key the cache on (path, etag) where the etag is the S3 object version identifier propagated from polars via #22734.
Related
Problem
prefetch_parquet_file_metadata_for_irfetches parquet footers from S3 at the start of every query execution, even for files that were already read in a previous iteration of the same benchmark or a previous query against the same dataset. TheCachedParquetInfoobjects (which include the parsedFileMetaDataand pre-builtHybridScanMetadata, and open kvikio handles as of #23317) are discarded when the query finishes.From profiling Q1 at SF300 on a g7e.8xlarge:
It takes a second to prefetch the parquet metadata on both iterations. But the second iteration should not pay that cost.
Proposed Fix
Add a session-level cache of
CachedParquetInfoobjects on the engine. Key the cache on(path, etag)where the etag is the S3 object version identifier propagated from polars via #22734.Related
Scannode in cudf-polars #22734 object metadata (size, etag) from polars