From e8759bcb2573f1d33caae2b118c4a996b08c892e Mon Sep 17 00:00:00 2001 From: Yidian Sun Date: Thu, 20 Aug 2026 18:21:18 +0800 Subject: [PATCH 1/5] chore(ET-6463): add large object snapshot doc --- .../migration/dms_agent_reference.mdx | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx index f227d26e6f..a946c63871 100644 --- a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx +++ b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx @@ -151,6 +151,74 @@ Oracle examples: - Disable network encryption negotiation: `oracle.net.encryption_client=REJECTED` - Set session program name: `v$session.program=DMSAgent` +## Postgres Large object migration parameters + +### LO_SNAPSHOT_ENABLED + +Feature gate for the large object snapshot. When false (the default), no large object related +operations run: the source is not scanned, no partitions are computed, and nothing is uploaded +or published. Set to true to opt into the large object snapshot. + +### LO_SNAPSHOT_PARALLELISM + +Number of parallel workers used for the large object snapshot. +For example the LO_SNAPSHOT_PARALLELISM = 20 +If the agent mode is reader, there will be 20 workers COPY OUT the large object from source database and upload to object storage in parallel. +If the agent mode is writer, there will be 20 workers download the large object and COPY IN to the target database. + +Recommandation: +Our performance benchmark test environment setting: +Deploy Source postgres database(8vCPU, 32G Mem) and Reader mode agent(16vCPU, 16G Mem) in one i4i.8xlarge EC2 +Deploy Target epas database(16vCPU, 32G Mem) and Writer mode agent(4vCPU, 16G Mem) in another one i4i.8xlarge EC2 + +Tables setting: +Large objects counts: 6400 +Large objects size: 32MB +Large objects expected partion size: 1024MB +Forign key paraent table size: 100000rows +Forign key child table size: 10000000rows +Integer Non-unique index size: 10000000rows + +Large object agent setting: +Reader mode agent parallelism: 24 +Writer mode agent parallelism: 32 + +Result: +E2E duration | Speed(MB/s) | Reader Peak CPU | Reader Peak Mem | Writer Peak CPU | Writer Peak Mem | Source Peak CPU | Source Peak Mem | Target Peak CPU | Target Peak Mem +8min30s 401.5686275 47.90% 35.10% 49% 12.80% 38% 75.10% 39.50% 25% + +### LO_SNAPSHOT_PARTITION_TARGET_MB + +All the large object will be splited to different partition. This option is the your expected partion size in MB. + +### LO_SNAPSHOT_PARTITION_TARGET_LO_COUNT + +All the large object will be splited to different partition. This option is the your expected large object count for each partion. +Agent will compute the partition number using both LO_SNAPSHOT_PARTITION_TARGET_MB and LO_SNAPSHOT_PARTITION_TARGET_LO_COUNT then +chose the larger one. + +### LO_SNAPSHOT_COMPRESSION_RATIO + +Compression ratio used to estimate logical (decompressed) volume. Intentionally unset by default: +when empty, the logical volume is estimated by sampling large objects at startup. When pinned, the +value is used directly with no sampling. + +### LO_SNAPSHOT_SAMPLE_LO_COUNT + +Number of large objects to decompress when estimating the compression ratio. A fixed count +(not a percentage) bounds the sampling cost regardless of total LO volume (default: 200) +Larger LO_SNAPSHOT_SAMPLE_LO_COUNT will increase the estimating time. + +### LO_SNAPSHOT_SAMPLE_TIMEOUT_SECONDS + +Statement timeout (seconds) for the sampling query for estimating the compression ratio (default: 30) +Increase this when agnet timeout during estimating the compression ratio + +### LO_SNAPSHOT_HIGH_VOLUME_WARN_LOGICAL_GB + +Estimated logical (decompressed) large object volume, in GiB, at or above which the snapshot logs a +high-volume WARN at startup. Set to 0 to disable the warning (default: 5120) + ## Optional parameters ### METRICS_TASK_ENABLED From 5d2218a898d68ab5c67a5e04b11e2b970abf1b6e Mon Sep 17 00:00:00 2001 From: Yidian Sun Date: Thu, 27 Aug 2026 14:39:04 +0800 Subject: [PATCH 2/5] chore(ET-6463): calculate lo snapshot duration using the log timestamps --- .../migration/dms_agent_reference.mdx | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx index a946c63871..e51b24cf34 100644 --- a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx +++ b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx @@ -183,6 +183,21 @@ Large object agent setting: Reader mode agent parallelism: 24 Writer mode agent parallelism: 32 +How to calculate large object snapshot duration: +Search log in reader mode agent: +``` +T1: 2026-08-27 06:25:44,632 INFO [com.enterprisedb.transporter.cdcreader.largeobject.LargeObjectManager] (pool-11-thread-1) [LO snapshot] Start large object snapshot +T2: 2026-08-27 06:26:14,491 INFO [com.enterprisedb.transporter.cdcreader.largeobject.LargeObjectManager] (pool-11-thread-1) [LO snapshot] Published large object manifest completion for topic org_beaconator4Appli_prj_xGQSUwMKDJZCxvhx.67590cea-a1df-11f1-a5fa-f66ef5321c73.large-object-manifest (partitionCount=16) +``` +calculte the large object COPY OUT and upload time = T2 - T1 + +Search log in writer mode agent: +``` +T3: 2026-08-27 06:25:44,725 INFO [com.enterprisedb.transporter.cdcwriter.connector.Connector] (pool-9-thread-1) [LO snapshot] Launching large object manifest consumer for edb +T4: 2026-08-27 06:26:59,315 INFO [com.enterprisedb.transporter.cdcwriter.connector.Connector] (lo-manifest-consumer-edb) [LO snapshot] Large object snapshot finished for edb +``` +calculte the end to end time = T4 - T1 + Result: E2E duration | Speed(MB/s) | Reader Peak CPU | Reader Peak Mem | Writer Peak CPU | Writer Peak Mem | Source Peak CPU | Source Peak Mem | Target Peak CPU | Target Peak Mem 8min30s 401.5686275 47.90% 35.10% 49% 12.80% 38% 75.10% 39.50% 25% From f70d9f784a54d6e29090113777efe185e046520a Mon Sep 17 00:00:00 2001 From: Yidian Sun Date: Thu, 27 Aug 2026 14:42:04 +0800 Subject: [PATCH 3/5] chore(ET-6463): calculate lo snapshot duration using the log timestamps --- .../using_hybrid_manager/migration/dms_agent_reference.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx index e51b24cf34..692dd65234 100644 --- a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx +++ b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx @@ -200,6 +200,7 @@ calculte the end to end time = T4 - T1 Result: E2E duration | Speed(MB/s) | Reader Peak CPU | Reader Peak Mem | Writer Peak CPU | Writer Peak Mem | Source Peak CPU | Source Peak Mem | Target Peak CPU | Target Peak Mem + 8min30s 401.5686275 47.90% 35.10% 49% 12.80% 38% 75.10% 39.50% 25% ### LO_SNAPSHOT_PARTITION_TARGET_MB From d9daa1ea9099412dc0a05cac35346f3562089def Mon Sep 17 00:00:00 2001 From: Yidian Sun Date: Thu, 27 Aug 2026 14:42:56 +0800 Subject: [PATCH 4/5] chore(ET-6463): calculate lo snapshot duration using the log timestamps --- .../migration/dms_agent_reference.mdx | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx index 692dd65234..2a10555e3d 100644 --- a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx +++ b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx @@ -162,28 +162,43 @@ or published. Set to true to opt into the large object snapshot. ### LO_SNAPSHOT_PARALLELISM Number of parallel workers used for the large object snapshot. + For example the LO_SNAPSHOT_PARALLELISM = 20 + If the agent mode is reader, there will be 20 workers COPY OUT the large object from source database and upload to object storage in parallel. + If the agent mode is writer, there will be 20 workers download the large object and COPY IN to the target database. Recommandation: + Our performance benchmark test environment setting: + Deploy Source postgres database(8vCPU, 32G Mem) and Reader mode agent(16vCPU, 16G Mem) in one i4i.8xlarge EC2 + Deploy Target epas database(16vCPU, 32G Mem) and Writer mode agent(4vCPU, 16G Mem) in another one i4i.8xlarge EC2 Tables setting: + Large objects counts: 6400 + Large objects size: 32MB + Large objects expected partion size: 1024MB + Forign key paraent table size: 100000rows + Forign key child table size: 10000000rows + Integer Non-unique index size: 10000000rows Large object agent setting: + Reader mode agent parallelism: 24 + Writer mode agent parallelism: 32 How to calculate large object snapshot duration: + Search log in reader mode agent: ``` T1: 2026-08-27 06:25:44,632 INFO [com.enterprisedb.transporter.cdcreader.largeobject.LargeObjectManager] (pool-11-thread-1) [LO snapshot] Start large object snapshot From 87f231e54cfff3dbe4ae49997e05e9c41f25a593 Mon Sep 17 00:00:00 2001 From: Yidian Sun Date: Thu, 27 Aug 2026 14:47:44 +0800 Subject: [PATCH 5/5] chore(ET-6463): fmt --- .../migration/dms_agent_reference.mdx | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx index 2a10555e3d..85ca6794a7 100644 --- a/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx +++ b/product_docs/docs/edb-postgres-ai/innovation-release/hybrid-manager/using_hybrid_manager/migration/dms_agent_reference.mdx @@ -169,15 +169,15 @@ If the agent mode is reader, there will be 20 workers COPY OUT the large object If the agent mode is writer, there will be 20 workers download the large object and COPY IN to the target database. -Recommandation: +#### Recommandation: -Our performance benchmark test environment setting: +**Our performance benchmark test environment setting:** Deploy Source postgres database(8vCPU, 32G Mem) and Reader mode agent(16vCPU, 16G Mem) in one i4i.8xlarge EC2 Deploy Target epas database(16vCPU, 32G Mem) and Writer mode agent(4vCPU, 16G Mem) in another one i4i.8xlarge EC2 -Tables setting: +**Tables setting:** Large objects counts: 6400 @@ -191,13 +191,13 @@ Forign key child table size: 10000000rows Integer Non-unique index size: 10000000rows -Large object agent setting: +**Large object agent setting:** Reader mode agent parallelism: 24 Writer mode agent parallelism: 32 -How to calculate large object snapshot duration: +**How to calculate large object snapshot duration:** Search log in reader mode agent: ``` @@ -213,10 +213,11 @@ T4: 2026-08-27 06:26:59,315 INFO [com.enterprisedb.transporter.cdcwriter.connec ``` calculte the end to end time = T4 - T1 -Result: -E2E duration | Speed(MB/s) | Reader Peak CPU | Reader Peak Mem | Writer Peak CPU | Writer Peak Mem | Source Peak CPU | Source Peak Mem | Target Peak CPU | Target Peak Mem +**Result:** -8min30s 401.5686275 47.90% 35.10% 49% 12.80% 38% 75.10% 39.50% 25% +| E2E duration | Speed (MB/s) | Reader Peak CPU | Reader Peak Mem | Writer Peak CPU | Writer Peak Mem | Source Peak CPU | Source Peak Mem | Target Peak CPU | Target Peak Mem | +|--------------|--------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------| +| 8min30s | 401.57 | 47.90% | 35.10% | 49% | 12.80% | 38% | 75.10% | 39.50% | 25% | ### LO_SNAPSHOT_PARTITION_TARGET_MB