Skip to content

fix(connectors): fix postgres source CDC producing no messages#3640

Merged
mmodzelewski merged 10 commits into
apache:masterfrom
mattp5657:fix/connectors-postgres-builtin-cdc
Jul 23, 2026
Merged

fix(connectors): fix postgres source CDC producing no messages#3640
mmodzelewski merged 10 commits into
apache:masterfrom
mattp5657:fix/connectors-postgres-builtin-cdc

Conversation

@mattp5657

@mattp5657 mattp5657 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Relates to #3582

Rationale

Builtin CDC mode currently produces zero messages against a real
database - wrong plugin/read-path pairing, and a parser matching a
format no real Postgres output plugin emits. This PR is scoped to
getting CDC actually working and provably correct, nothing more.

I have reduce the scope of this specific PR for reviewability, in the future
would look to create issues and work on these issues:

  • pgoutput as the long-term plugin - real binary decoder + streaming
    connection is a much bigger lift; test_decoding unblocks correctness
    today. Target once cdc_backend = "pg_replicate" gets picked up.
  • At-least-once / crash safety (slot advances before send is confirmed,
    no LSN tracked) - needs its own review since it touches the
    state-persistence contract. Follow-up: #TODO.
  • Cosmetic items (UUID vs PK as message id, processing time vs WAL
    commit time, old_data unpopulated) - mechanical, separate PR.

What changed?

Before: wrong slot plugin, a parser matching a format no real Postgres
output emits, and invalid setup SQL - CDC produced zero messages, and
several related bugs failed silently too. After: parser rewritten
against real captured output, setup/poll SQL fixed and bounded, and
those failure modes now work correctly or fail loudly. Covered by
corpus-driven unit tests plus integration tests against a live
wal_level=logical container.

  • Parser rewritten against real test_decoding output
  • Slot/setup SQL fixed, including the upgrade path and table filtering
  • Batch handling hardened (bad rows, unbounded backlog)
  • Dead config fields removed

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

Claude was used to help generate and review this PR.

@mattp5657 mattp5657 changed the title Fix/connectors postgres builtin cdc fix(connectors): fix postgres source CDC producing no messages Jul 9, 2026
@mattp5657
mattp5657 marked this pull request as ready for review July 9, 2026 23:35
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 9, 2026
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.57364% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.98%. Comparing base (700221b) to head (2f77f92).

Files with missing lines Patch % Lines
core/connectors/sources/postgres_source/src/lib.rs 94.57% 27 Missing and 8 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3640       +/-   ##
=============================================
- Coverage     74.35%   40.98%   -33.37%     
  Complexity      950      950               
=============================================
  Files          1303     1301        -2     
  Lines        148712   129522    -19190     
  Branches     124221   105033    -19188     
=============================================
- Hits         110569    53089    -57480     
- Misses        34668    73663    +38995     
+ Partials       3475     2770      -705     
Components Coverage Δ
Rust Core 33.00% <94.57%> (-41.59%) ⬇️
Java SDK 62.64% <ø> (ø)
C# SDK 72.20% <ø> (-0.07%) ⬇️
Python SDK 92.27% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 92.82% <ø> (+0.21%) ⬆️
Go SDK 43.08% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sources/postgres_source/src/lib.rs 73.20% <94.57%> (+16.52%) ⬆️

... and 424 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hubcio

hubcio commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

i think this is somewhat related to #3313

@hubcio

hubcio commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@mmodzelewski could you please check this one?

@mmodzelewski mmodzelewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @mattp5657. The change looks solid, I've posted a few comments where the code could be improved.

Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/integration/tests/connectors/postgres/postgres_source_cdc.rs Outdated
Comment thread core/connectors/sources/postgres_source/README.md
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 15, 2026
@mattp5657

Copy link
Copy Markdown
Contributor Author

These should be fixed, let me know if you have any other feedback :)

@mattp5657

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jul 19, 2026

@mmodzelewski mmodzelewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @mattp5657, approved

@mmodzelewski
mmodzelewski merged commit 6513b2f into apache:master Jul 23, 2026
59 checks passed
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants