24.8.14 Backport of #79147 - Remove guard pages from fibers and alternative stack for signals in threads

[mkmkme]

Note for the reviewer

This PR originally changed two files and I had conflicts in both of them. One of them, FiberStack.cpp, was missing from the original 24.8 branch because it was added later.

Initially I landed the changes on FiberStack.h itself and introduced the missing memoryGuardInstall / memoryGuardRemove functions to Common/memory.h, creating Common/memory.cpp as well. This, however, broke the build for a missing jemalloc.h because FiberStack.h got an include for Common/memory.h:

In file included from /home/mk/git/altinity-clickhouse-24.8-79147/src/Core/ExternalTable.cpp:18:
In file included from /home/mk/git/altinity-clickhouse-24.8-79147/src/Core/ExternalTable.h:3:
In file included from /home/mk/git/altinity-clickhouse-24.8-79147/src/Client/Connection.h:11:
In file included from /home/mk/git/altinity-clickhouse-24.8-79147/src/IO/ReadBufferFromPocoSocketChunked.h:4:
In file included from /home/mk/git/altinity-clickhouse-24.8-79147/src/IO/ReadBufferFromPocoSocket.h:5:
In file included from /home/mk/git/altinity-clickhouse-24.8-79147/src/Common/AsyncTaskExecutor.h:5:
In file included from /home/mk/git/altinity-clickhouse-24.8-79147/src/Common/FiberStack.h:7:
/home/mk/git/altinity-clickhouse-24.8-79147/src/Common/memory.h:13:14: fatal error: 'jemalloc/jemalloc.h' file not found
   13 | #    include <jemalloc/jemalloc.h>
      |              ^~~~~~~~~~~~~~~~~~~~~
1 error generated.

Because of that, I decided to split FiberStack to a header and implementation file as it is in the upstream. When I did that, I got multiple compiler errors about missing fmt and formatReadable* functions because the files using those functions previously got those from transitive includes by FiberStack.h. After I moved most of inclusions to FiberStack.cpp, the functions became undefined.

Before adding the extra includes to Keeper.cpp, ClientBase.cpp, Connection.cpp and AsyncTaskExecutor.cpp I verified those includes existed in the upstream's versions of those files at the moment of the original PR.

Unfortunately all of this led to the situation where the PR got quite large in comparison to the original one.

Changelog category (leave one):

• CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Remove guard pages for threads and async_socket_for_remote/use_hedge_requests. Change the allocation method in FiberStack from mmap to aligned_alloc. Since this splits VMAs and under heavy load vm.max_map_count can be reached (ClickHouse#79147 by @CheSema)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[altinity-robot]

This is an automated comment for commit 26c354c with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure

Successful checks

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	✅ success
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Grype Scan altinityinfra/clickhouse-keeper:2035-24.8.14.10546.altinitytest	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Grype Scan altinityinfra/clickhouse-server:2035-24.8.14.10546.altinitytest-alpine	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Grype Scan altinityinfra/clickhouse-server:2035-24.8.14.10546.altinitytest	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Ready for release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release aes_encryption	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release aggregate_functions_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release aggregate_functions_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release aggregate_functions_3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release alter_attach_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release alter_attach_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release alter_move	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release alter_replace	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release atomic_insert	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release base_58	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release benchmark_aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release benchmark_gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release benchmark_minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release clickhouse_keeper_failover	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release clickhouse_keeper_no_ssl_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release clickhouse_keeper_no_ssl_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release clickhouse_keeper_ssl_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release clickhouse_keeper_ssl_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release datetime64_extended_range	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release disk_level_encryption	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release dns	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release engines	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release example	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release extended_precision_data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release functions	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release iceberg_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release iceberg_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release jwt_authentication	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release kafka	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release kerberos	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release key_value	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release ldap_authentication	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release ldap_role_mapping	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release lightweight_delete	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release memory	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release parquet	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release parquet_aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release parquet_minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release part_moves_between_shards	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release rbac_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release rbac_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release rbac_3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_aws_s3_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_aws_s3_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_azure_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_azure_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_export_part	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_export_partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_gcs_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_gcs_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_minio_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_minio_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release s3_minio_3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release selects	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release session_timezone	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release settings	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release ssl_server_1	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release ssl_server_2	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release ssl_server_3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release swarms	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release tiered_storage_local	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release tiered_storage_minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release tiered_storage_s3amazon	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release tiered_storage_s3gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release version	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release window_functions	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Sign release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	✅ success

[ilejn]

Honestly I am a bit shocked because I expected something like this https://github.com/Altinity/ClickHouse/tree/backports/24.8.14/79147_tiny
Kind of a sketch, requires minor fixes, but it is compilable.

This is a compact change, it keeps ::munmap and everything that is natural for 24.8

Anyway, the PR looks good to me.

[mkmkme]

Honestly I am a bit shocked because I expected something like this https://github.com/Altinity/ClickHouse/tree/backports/24.8.14/79147_tiny

This is a compact change, it keeps ::munmap and everything that is natural for 24.8

Anyway, the PR looks good to me.

I was considering that, but wanted to keep it as close to upstream as possible.
We can take your approach instead since you've done it anyway :)

[ilejn]

I was considering that, but wanted to keep it as close to upstream as possible. We can take your approach instead since you've done it anyway :)

We already have this upstream PR in 25.3 #917
Since this backport is in line with 917, lets go further.

[CarlosFelipeOR]

❌ QA: Build fails — needs a fix before further verification

Verdict: Not approvable in its current state. This PR does not build on either architecture, so no functional or regression testing can run yet.

Build failure

Both BuilderDebRelease (amd64) and BuilderDebAarch64 (arm64) fail identically; everything downstream (stateless, integration, regression, etc.) is SKIPPED as a result.

src/Common/AsyncTaskExecutor.h:126:1: error: unknown type name 'String'; did you mean 'std::string'?
  126 | String getSocketTimeoutExceededMessageByTimeoutType(AsyncEventTimeoutType type, Poco::Timespan timeout, const String & socket_description);

Logs: release · aarch64

Root cause

AsyncTaskExecutor.h uses String (line 126) but obtained it only transitively through FiberStack.h. This PR empties FiberStack.h (moving its includes into the new FiberStack.cpp), so AsyncTaskExecutor.h no longer sees the String alias. The <fmt/format.h> include added to AsyncTaskExecutor.cpp covers the fmt::format usage but not String — and since the declaration lives in the header, the fix belongs in the header.

Suggested fix

// src/Common/AsyncTaskExecutor.h
#include <base/types.h>   // provides the `String` alias (using String = std::string)

This keeps the header self-contained and fixes every consumer in the ExternalTable.h → Connection.h → … → AsyncTaskExecutor.h chain, not just the .cpp.

Backport review (otherwise looks good)

Differential vs upstream ClickHouse#79147: the core logic of FiberStack.cpp and ThreadStatus.cpp matches the upstream final state exactly. The larger diff is justified — it reconstructs prerequisites the 24.8 branch lacked (splitting FiberStack into .h/.cpp, backporting the memoryGuardInstall/memoryGuardRemove infra into memory.cpp).

---

Root-cause diagnosis and the suggested fix were produced with AI-assisted analysis (static; confirmed by inspecting the header includes, not yet by a local rebuild).

[mkmkme]

@CarlosFelipeOR,

Thanks! I've addressed this in 26c354c