Avoid SQL query pessimization when auxiliary database(s) used

[wrenoud]

• AI (Copilot or something similar) supported my development of this PR. See our policy about AI tool use. Use of AI tools must be indicated.
• Added clear title that can be used to generate release notes

Investigating the performance of cs2cs when an auxiliary database is used, the query run by AuthorityFactory::Private::createPropertiesSearchUsages was found to be particularly slow. Reviewing the results of EXPLAIN QUERY PLAN on the query it was found that intermediate results from the VIEWs used to UNION the primary and auxiliary databases were being MATERIALIZE'ed (see details below). This was in affect duplicating the tables in memory for the query.

Details

  2,   0,   0, CO-ROUTINE usage
  3,   2,   0, COMPOUND QUERY
  4,   3,   0, LEFT-MOST SUBQUERY
  7,   4,  43, SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?)
 37,   3,   0, UNION ALL
 39,  37, 216, SCAN db_1.usage
 69,   0,   0, MATERIALIZE extent
 71,  69,   0, COMPOUND QUERY
 72,  71,   0, LEFT-MOST SUBQUERY
 74,  72, 135, SCAN db_0.extent
 93,  71,   0, UNION ALL
 95,  93, 215, SCAN db_1.extent
117,   0,   0, MATERIALIZE scope
119, 117,   0, COMPOUND QUERY
120, 119,   0, LEFT-MOST SUBQUERY
122, 120,  97, SCAN db_0.scope
132, 119,   0, UNION ALL
134, 132, 215, SCAN db_1.scope
146,   0, 196, SCAN usage
167,   0,   0, BLOOM FILTER ON extent (code=? AND auth_name=?)
188,   0,  53, SEARCH extent USING AUTOMATIC COVERING INDEX (code=? AND auth_name=?)
201,   0,   0, BLOOM FILTER ON scope (code=? AND auth_name=?)
213,   0,  53, SEARCH scope USING AUTOMATIC COVERING INDEX (code=? AND auth_name=?)
245,   0,   0, USE TEMP B-TREE FOR ORDER BY

I first attempted to use temporary tables for extent and scope, which incurred an overhead of table creation, but significantly improved the query time. But @rouault pointed out that possibly the ORDER BY was the cause for the MATERIALIZE, and it was. That is what is implemented here. The sort is done after the query. You can see the new EXPLAIN QUERY PLAN in the details below.

Details

  1,   0,   0, COMPOUND QUERY
  2,   1,   0, LEFT-MOST SUBQUERY
  3,   2,   0, COMPOUND QUERY
  4,   3,   0, LEFT-MOST SUBQUERY
  5,   4,   0, COMPOUND QUERY
  6,   5,   0, LEFT-MOST SUBQUERY
 11,   6,  43, SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?)
 30,   6,  39, SEARCH db_0.extent USING PRIMARY KEY (auth_name=? AND code=?)
 36,   6,  34, SEARCH db_0.scope USING PRIMARY KEY (auth_name=? AND code=?)
 62,   5,   0, UNION ALL
 67,  62,  43, SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?)
 86,  62,  39, SEARCH db_0.extent USING PRIMARY KEY (auth_name=? AND code=?)
 92,  62,  45, SEARCH db_1.scope USING PRIMARY KEY (auth_name=? AND code=?)
118,   5,   0, UNION ALL
119, 118,   0, COMPOUND QUERY
120, 119,   0, LEFT-MOST SUBQUERY
125, 120,  43, SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?)
144, 120,  45, SEARCH db_1.extent USING PRIMARY KEY (auth_name=? AND code=?)
150, 120,  34, SEARCH db_0.scope USING PRIMARY KEY (auth_name=? AND code=?)
176, 119,   0, UNION ALL
181, 176,  43, SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?)
200, 176,  45, SEARCH db_1.extent USING PRIMARY KEY (auth_name=? AND code=?)
206, 176,  45, SEARCH db_1.scope USING PRIMARY KEY (auth_name=? AND code=?)
232, 119,   0, UNION ALL
233, 232,   0, COMPOUND QUERY
234, 233,   0, LEFT-MOST SUBQUERY
235, 234,   0, COMPOUND QUERY
236, 235,   0, LEFT-MOST SUBQUERY
240, 236, 216, SCAN db_1.usage
256, 236,  39, SEARCH db_0.extent USING PRIMARY KEY (auth_name=? AND code=?)
262, 236,  34, SEARCH db_0.scope USING PRIMARY KEY (auth_name=? AND code=?)
288, 235,   0, UNION ALL
292, 288, 216, SCAN db_1.usage
308, 288,  39, SEARCH db_0.extent USING PRIMARY KEY (auth_name=? AND code=?)
314, 288,  45, SEARCH db_1.scope USING PRIMARY KEY (auth_name=? AND code=?)
340, 235,   0, UNION ALL
341, 340,   0, COMPOUND QUERY
342, 341,   0, LEFT-MOST SUBQUERY
346, 342, 216, SCAN db_1.usage
362, 342,  45, SEARCH db_1.extent USING PRIMARY KEY (auth_name=? AND code=?)
368, 342,  34, SEARCH db_0.scope USING PRIMARY KEY (auth_name=? AND code=?)
394, 341,   0, UNION ALL
398, 394, 216, SCAN db_1.usage
414, 394,  45, SEARCH db_1.extent USING PRIMARY KEY (auth_name=? AND code=?)
420, 394,  45, SEARCH db_1.scope USING PRIMARY KEY (auth_name=? AND code=?)

I also added a minor performance improvement in DatabaseContext::Private::attachExtraDatabases, the columns were being aggregated into a string multiple times per table for each database. I switch to doing it once and reusing the joined columns string.

Results

This is for Windows 11, Intel Core Ultra 7, running the following command:

echo 45 0 | cs2cs EPSG:4236 EPSG:32601

Below you can see for various values of environment variable PROJ_AUX_DB the per-query average over 4 runs, with 11 calls per run. empty.db is initialized with no entries, and nsrs_proj.db is from https://github.com/jjimenezshaw/NSRS-2022-PROJ

PROJ_AUX_DB	averge query master (4 runs, n=44)	averge query this PR (4 runs, n=44)
(unset)	0.06 ms	0.06 ms
empty.db	3.72 ms	0.25 ms
nsrs_proj.db	4.61 ms	0.57 ms
empty.db;nsrs_proj.db	5.21 ms	1.02 ms

[rouault]

LGTM. Can you rework the commit history to blend together the fixup commit and its original one?

[wrenoud]

Can you rework the commit history to blend together the fixup commit and its original one?

Done! Thanks @rouault

[rouault]

Thanks @wrenoud !