chore(docs): add server tuning & Linux optimization

content/docs/binary-protocol/commands.mdx

@@ -80,7 +80,7 @@ LEAVE_CONSUMER_GROUP = 605                # operation 149
 SYNC_CONSUMER_GROUP = 606                 # non-replicated
 ```
 
-`FLUSH_UNSAVED_BUFFER` still decodes on the wire, but the server has no on-demand flush primitive and answers every call with `FeatureUnavailable`. Per-topic durability is configured with the `enforce_fsync` [topic option](/docs/server/topic-options) instead.
+`FLUSH_UNSAVED_BUFFER` still decodes on the wire, but the server has no on-demand flush primitive and answers every call with `FeatureUnavailable`. Per-topic message completion is configured with the `durability` [topic option](/docs/server/topic-options).
 
 ## Payloads
 
@@ -178,7 +178,7 @@ Patch semantics: absent option keys are left unchanged.
 [stream_id: Identifier][partitions_count: u32][name_len: u8][name: N][options block to end]
 ```
 
-The fixed fields are the shape of the operation: which stream, how many partitions, what name. Every topic setting (`compression_algorithm`, `message_expiry`, `max_topic_size`, `segment_size`, `enforce_fsync`, `messages_required_to_save`, `size_of_messages_required_to_save`, `preallocate_segments`) rides the options block. See [Topic options](/docs/server/topic-options). `partitions_count` is an argument, not a setting: it is consumed at admission and never persisted as an option.
+The fixed fields are the shape of the operation: which stream, how many partitions, what name. Every topic setting (`compression_algorithm`, `message_expiry`, `max_topic_size`, `segment_size`, `durability`, `consumer_offset_durability`, `messages_required_to_save`, `size_of_messages_required_to_save`, `preallocate_segments`) rides the options block. See [Topic options](/docs/server/topic-options). `partitions_count` is an argument, not a setting: it is consumed at admission and never persisted as an option.
 
 **Delete topic. Code: 303.**
 
@@ -262,7 +262,7 @@ The 16-byte prefix is followed by a stream of [batch records](/docs/binary-proto
 [stream_id: Identifier][topic_id: Identifier][partition_id: u32][fsync: u8]
 ```
 
-Parses, but the server always answers `FeatureUnavailable`: there is no on-demand flush primitive. Use the per-topic `enforce_fsync` option for durability guarantees.
+Parses, but the server always answers `FeatureUnavailable`: there is no on-demand flush primitive. Select `durability=persisted` at topic creation to require recoverable stable-storage copies before successful message completion.
 
 ### Consumer offsets
 

content/docs/binary-protocol/encodings.mdx

@@ -114,7 +114,7 @@ Semantics:
 - **Update** requests are patches: keys absent from the block are left alone, never reset. A client built before a key existed cannot erase it.
 - Unknown keys are rejected at the wire edge, never silently skipped.
 
-The catalog is discoverable at runtime with `DESCRIBE_OPTIONS` (code 13), payload `[scope: u8]` with scope `1` = topic, `2` = stream, `3` = user (HTTP: `GET /options/topic`). Today only topics have keys. The stream and user catalogs are empty, and any key sent for them is rejected. The topic catalog (`segment_size`, `enforce_fsync`, `message_expiry`, `max_topic_size`, and the rest) with defaults and constraints is documented on the [Topic options](/docs/server/topic-options) page. `UpdateTopic` accepts only `compression_algorithm`, `message_expiry`, and `max_topic_size`. The storage-layout knobs are create-only.
+The catalog is discoverable at runtime with `DESCRIBE_OPTIONS` (code 13), payload `[scope: u8]` with scope `1` = topic, `2` = stream, `3` = user (HTTP: `GET /options/topic`). Today only topics have keys. The stream and user catalogs are empty, and any key sent for them is rejected. The topic catalog (`segment_size`, `durability`, `consumer_offset_durability`, `message_expiry`, `max_topic_size`, and the rest) with defaults and constraints is documented on the [Topic options](/docs/server/topic-options) page. `UpdateTopic` accepts only `compression_algorithm`, `message_expiry`, and `max_topic_size`. Both durability policies and the storage-layout knobs are create-only.
 
 ## Compression
 

content/docs/clustering/deploy.mdx

@@ -85,13 +85,13 @@ Start each replica in a separate terminal. Use a different data path for every p
 
 ```bash
 # Replica 0
-IGGY_SYSTEM_PATH=local_data/node-0 ./target/debug/iggy-server --replica-id 0
+IGGY_PATH=local_data/node-0 ./target/debug/iggy-server --replica-id 0
 
 # Replica 1
-IGGY_SYSTEM_PATH=local_data/node-1 ./target/debug/iggy-server --replica-id 1
+IGGY_PATH=local_data/node-1 ./target/debug/iggy-server --replica-id 1
 
 # Replica 2
-IGGY_SYSTEM_PATH=local_data/node-2 ./target/debug/iggy-server --replica-id 2
+IGGY_PATH=local_data/node-2 ./target/debug/iggy-server --replica-id 2
 ```
 
 The exported settings must be present in all three terminals.
@@ -103,7 +103,7 @@ In cluster mode:
 - `ports` is the single source of listener ports: every enabled transport needs an explicit per-node port, otherwise the server **refuses to start**
 - `tcp_replica` carries replica-to-replica consensus traffic and is **always required**
 - `ip` must be a **literal IP address**. Use `advertised_address` when clients can't reach it (see [Configuration](/docs/clustering/configuration))
-- use a different `system.path` for each process on the same host
+- use a different root `path` for each process on the same host
 
 ## Spanning multiple hosts
 

content/docs/clustering/durability.mdx

@@ -0,0 +1,137 @@
+---
+title: Cluster Durability
+description: "Quorum completion, persisted prepare history, and the failures each topic policy covers."
+---
+
+Iggy uses [Viewstamped Replication](/docs/clustering/vsr) to commit partition
+operations. The topic's `durability` and `consumer_offset_durability`
+options select the storage guarantee required for message production and
+explicit offset changes, respectively. Both default independently to
+`replicated`. Both continue to write data to disk.
+
+For the creation options and single-node behavior, see
+[Durability](/docs/server/durability).
+
+## What an acknowledgement means
+
+For an operation that waits for completion:
+
+| Policy | Required before success |
+| --- | --- |
+| `replicated` | VSR quorum commit and local application, without an additional stable-storage barrier |
+| `persisted` | VSR quorum commit backed by recoverable stable-storage copies at the required quorum, followed by local application |
+
+The policy changes what a replica must retain before its acknowledgement can
+count toward commit. It does not replace quorum commit with a primary-only
+disk write.
+
+Iggy's replication quorum is not a strict majority for every group size:
+
+| Replicas | Replication quorum | View-change quorum |
+| --- | --- | --- |
+| 1 | 1 | 1 |
+| 2 | 2 | 2 |
+| 3 | 2 | 2 |
+| 4 | 2 | 3 |
+| 5 | 3 | 3 |
+| 6 | 3 | 4 |
+
+Except for two replicas, the replication quorum is `min(ceil(n / 2), 3)`
+and the view-change quorum is `n - replication_quorum + 1`. Two replicas
+require both for either quorum. The quorums therefore intersect. These are
+the [implemented quorum rules](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/consensus/src/impls.rs),
+not a configurable acknowledgement count.
+
+## Replicated completion
+
+With both policies set to `replicated`, partition prepares stay off the
+disk prepare-WAL path. Replicas retain the prepares in memory, and committed
+messages reach segment files through ordinary flush scheduling.
+
+Success therefore does not prove that the operation reached stable storage
+on any replica. Replication protects against failures while sufficient peers
+retain the history and can form the quorums needed for recovery and progress.
+For example, a healthy three-replica group can continue after one replica
+fails.
+
+Failure independence matters. A shared power loss, an OOM cascade, or a
+crash triggered on every replica can destroy multiple volatile copies at
+once. A process-only failure can lose messages still in the processes'
+journals, even while the kernels and their page caches remain alive.
+Replication count alone does not establish a bound on that loss.
+
+## Persisted completion
+
+When either policy is `persisted`, each multi-replica partition uses a
+bounded on-disk prepare WAL. The WAL records full prepares, including message
+payloads. A prepare requiring persistence cannot release its `PrepareOk`
+until its history and durable frontier are recoverable.
+
+Prepares can be forwarded while local persistence is pending. Once enough
+replicas have met the required barrier, the operation can commit and the
+primary can apply it and reply. An acknowledged message may still be in
+the prepare WAL rather than in a segment file; its recovery does not depend
+on first reaching a segment flush threshold.
+
+The WAL also retains predecessors across message and offset operations.
+Consequently:
+
+- `durability=persisted` does not make an offset response persisted when
+  `consumer_offset_durability=replicated`.
+- `consumer_offset_durability=persisted` with replicated messages still
+  journals message payloads. A durable offset's predecessor history must
+  remain recoverable.
+- A shared barrier can also persist co-batched operations with the weaker
+  policy. That incidental persistence does not strengthen what their earlier
+  `replicated` acknowledgements promised.
+- Only the `replicated` / `replicated` combination avoids the disk prepare
+  WAL entirely.
+
+WAL history is reclaimed only after the materialized segment and offset
+state needed to replace it has been synchronized. The server's
+`[partition] wal_bytes_max` setting, default `256 MiB`, bounds active WAL
+and queued/in-flight prepare bytes per partition. Capacity pressure causes
+checkpointing and backpressure; it does not downgrade a persisted operation.
+Temporary rewrites need additional disk space.
+
+## Recovery and failure limits
+
+A restarting replica loads segment and offset state and, when enabled,
+reconciles the durable prepare WAL with that state. It also restores durable
+consensus state and follows the recovery protocol before serving as a healthy
+replica. It does not treat every locally recovered prepare as committed.
+
+Missing required history, checksum failures, or contradictory materialized
+state must not become an empty healthy partition. Recovery can fence a
+partition and repair it from peers. Storage errors withhold successful
+completion; a locally failed application of a committed operation fences the
+partition and initiates server shutdown.
+
+`persisted` protects acknowledged operations across process and power
+failures when the required storage copies remain intact and the storage
+honors synchronization. Recovery and availability still require the
+protocol's quorums. It cannot protect against destruction of all durable
+copies, nor does it prevent configured retention or explicit deletion.
+
+## Choosing the policies
+
+Use `replicated` when the workload accepts the risk of losing acknowledged
+operations after correlated failures in exchange for avoiding a required
+storage barrier at completion. Isolate replicas across failure domains; see
+[Deployment](/docs/clustering/deploy).
+
+Use `durability=persisted` when acknowledged messages must be recoverable
+after volatile copies are lost. Select
+`consumer_offset_durability=persisted` independently when explicit offset
+stores and deletes need the same guarantee. Ordinary flush thresholds can
+remain at their defaults.
+
+Neither policy turns HTTP `ack=none` or a poll's auto-commit into an awaited
+durable result. See [Which responses prove completion](/docs/server/durability#which-responses-prove-completion).
+
+The storage mechanisms are implemented in the
+[partition persistence worker](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/partitions/src/persistence.rs)
+and [prepare journal](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/journal/src/partition_journal.rs).
+The [crash-recovery tests](https://github.com/apache/iggy/blob/97f7b0c0335f81691bef34f923f21d783016a6a4/core/integration/tests/cluster/crash_durability.rs)
+exercise acknowledged persisted messages and offsets below ordinary flush
+thresholds, on both a singleton and a three-replica cluster.

content/docs/clustering/meta.json

@@ -1,4 +1,4 @@
 {
   "title": "Clustering",
-  "pages": ["vsr", "deploy", "configuration", "security", "client-failover"]
+  "pages": ["vsr", "durability", "deploy", "configuration", "security", "client-failover"]
 }

content/docs/introduction/architecture.mdx

@@ -118,8 +118,10 @@ Iggy uses a custom memory pool with 28 buckets holding buffer sizes from 4 KiB t
 
 Messages flow through a multi-stage write pipeline:
 
-1. Messages arrive on the owning shard and are buffered in the partition journal
-2. A flush is triggered when either the message count threshold or the size threshold is reached - both are **per-topic options** set at topic creation (defaults: 1024 messages, 1 MiB)
-3. The `MessagesWriter` uses **vectored I/O** with up to 1024 buffers per syscall
-4. `fsync` per write is a per-topic option (`enforce_fsync`) for durability guarantees
-5. When a segment reaches the topic's segment size (default 1 GiB), it is **sealed** and a new segment is created
+1. Messages arrive on the owning shard and are buffered in the partition journal.
+2. Partition VSR replicates prepares. With `durability=persisted`, a multi-replica group requires recoverable prepare-WAL copies at the replication quorum before commit.
+3. Committed operations are applied before success is returned. A singleton with `durability=persisted` synchronizes local segment state before replying.
+4. Ordinary segment writes use per-topic count and byte thresholds (defaults: 1024 messages, 1 MiB); required persistence, capacity pressure, and lifecycle work can flush earlier. The `MessagesWriter` uses **vectored I/O** with up to 1024 buffers per syscall.
+5. When a segment reaches the topic's segment size (default 1 GiB), it is **sealed** and a new segment is created.
+
+Message `durability` and `consumer_offset_durability` default independently to `replicated`. Both policies write data to disk; `persisted` adds a stable-storage requirement at completion. See [Durability](/docs/server/durability).

content/docs/introduction/concepts.mdx

@@ -36,7 +36,7 @@ The stream is a logical concept, and you might think of it as a **namespace**. F
 
 The topic is also the logical concept, which is a part of the stream. The topic is identified by its unique ID. You could think of topic as an entity being responsible for storing the specific type of the records. For example, you could have a topic for the user events, and another topic for the order events, etc.
 
-The messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have the **retention policy** assigned, which means that the records are being deleted automatically once they are older than the specified retention period. Topics also support maximum size limits and per-topic durability options (`segment_size`, `enforce_fsync`, flush thresholds) set at creation, plus a `compression_algorithm` option (a placeholder today: no compression is applied yet).
+The messages are not being stored in the topic directly, but rather in the **partitions**, which are assigned to the topic. The topic can have one or more partitions assigned, that could help achieve higher parallelism and throughput. The topic can also have the **retention policy** assigned, which means that the records are being deleted automatically once they are older than the specified retention period. Topics also support maximum size limits and per-topic storage options (`segment_size`, `durability`, `consumer_offset_durability`, and flush thresholds) set at creation, plus a `compression_algorithm` option (a placeholder today: no compression is applied yet). Both [durability policies](/docs/server/durability) independently default to `replicated`.
 
 ## Partition
 

content/docs/introduction/getting-started.mdx

@@ -33,7 +33,7 @@ docker run --rm \
   apache/iggy:latest
 ```
 
-`SYS_NICE`, the seccomp setting and the memlock limit are all required by `io_uring` and the thread-per-core architecture; see [Docker & Helm](/docs/server/docker) for the details. `IGGY_TCP_ADDRESS` is needed because the server binds to `127.0.0.1` inside the container by default, which a published port cannot reach. `IGGY_NODE_ADVERTISED_ADDRESS` is needed because that wildcard leaves the server with no address to give clients, and it refuses to start rather than publish one nobody can dial. Here the port is published to the host, so `localhost` is that address. Setting the root credentials explicitly means the username and password used later in this guide will work.
+The capabilities, seccomp setting, and memlock limit form a permissive development setup. Production deployments can use narrower syscall permissions and a finite memory budget; see [Docker & Helm](/docs/server/docker#why-these-capabilities) for the details. `IGGY_TCP_ADDRESS` is needed because the server binds to `127.0.0.1` inside the container by default, which a published port cannot reach. `IGGY_NODE_ADVERTISED_ADDRESS` is needed because that wildcard leaves the server with no address to give clients, and it refuses to start rather than publish one nobody can dial. Here the port is published to the host, so `localhost` is that address. Setting the root credentials explicitly means the username and password used later in this guide will work.
 
 Alternatively, build from source by cloning the [repository](https://github.com/apache/iggy) and running:
 
@@ -201,7 +201,7 @@ async fn init_system(client: &IggyClient) {
 }
 ```
 
-Every field of `TopicCreateOptions` left as `None` resolves to the **server defaults**. It's also where the per-topic durability knobs live: `segment_size`, `enforce_fsync` and the flush thresholds (`messages_required_to_save`, `size_of_messages_required_to_save`) can all be set at topic creation.
+Optional fields of `TopicCreateOptions` left as `None` resolve to the **server defaults**. The `durability` and `consumer_offset_durability` fields use `Durability::Replicated` or `Durability::Persisted` and default independently to `Replicated`. They select the completion guarantee for messages and explicit offset changes. Segment size and the flush thresholds (`messages_required_to_save`, `size_of_messages_required_to_save`) are separate creation options. See [Durability](/docs/server/durability).
 
 Finally, let's send some messages into our stream. We will implement the basic loop with an interval between each iteration to simulate publishing the batch of messages. Since the streaming server works directly with the binary data, and couldn't care less about the (de)serialization format for the message payload, it's really up to you, how to efficiently stream the messages for your use case.