feat(python): expose remaining Topic/TopicDetails fields and partitions by mattp5657 · Pull Request #3623 · apache/iggy

mattp5657 · 2026-07-07T00:01:29Z

Which issue does this PR address?

Rationale

Continuation of #3572. That PR exposed Topic/TopicDetails basics but left several fields returned by the server unavailable in Python, so callers couldn't inspect a topic's expiry, size, max size, or its partitions without another SDK.

What changed?

Topic and TopicDetails previously exposed only id, name, partitions_count, compression_algorithm, and replication_factor, leaving created_at, size, message_expiry, max_topic_size, and per-partition data unavailable in Python.

Both classes now expose created_at, size, message_expiry (new IggyExpiry enum), and max_topic_size (new MaxTopicSize enum). TopicDetails also gains partitions, a list of new Partition objects. Partition (core/common) now derives Clone to support building that list.

Local Execution

Passed
Pre-commit hooks ran

Breaking Chnages

create_topic's message_expiry and max_topic_size parameters both
change type on this branch, and both changes are breaking relative to
the last released version (0.8.0):

Parameter	`0.8.0` (released)	This branch
`message_expiry`	`datetime.timedelta \| None`	`IggyExpiry \| None`
`max_topic_size`	`builtins.int \| None`	`MaxTopicSize \| None`

AI Usage

Claude was used to help generate and review this PR.

codecov · 2026-07-07T00:09:16Z

Codecov Report

❌ Patch coverage is 84.40860% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.49%. Comparing base (a5001d3) to head (320135a).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
foreign/python/src/topic.rs	79.68%	26 Missing ⚠️
foreign/python/src/consumer.rs	87.50%	2 Missing ⚠️
foreign/python/src/client.rs	97.43%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3623      +/-   ##
============================================
- Coverage     74.34%   72.49%   -1.86%     
  Complexity      950      950              
============================================
  Files          1303     1303              
  Lines        148712   143588    -5124     
  Branches     124225   119024    -5201     
============================================
- Hits         110562   104090    -6472     
- Misses        34673    35719    +1046     
- Partials       3477     3779     +302

Components	Coverage Δ
Rust Core	`72.37% <ø> (-2.20%)`	⬇️
Java SDK	`62.64% <ø> (ø)`
C# SDK	`71.13% <ø> (-1.16%)`	⬇️
Python SDK	`91.06% <84.40%> (-1.21%)`	⬇️
PHP SDK	`84.52% <ø> (ø)`
Node SDK	`92.70% <ø> (ø)`
Go SDK	`43.08% <ø> (ø)`

Files with missing lines	Coverage Δ
foreign/python/src/lib.rs	`100.00% <100.00%> (ø)`
foreign/python/src/client.rs	`99.15% <97.43%> (+0.01%)`	⬆️
foreign/python/src/consumer.rs	`81.11% <87.50%> (+0.57%)`	⬆️
foreign/python/src/topic.rs	`83.33% <79.68%> (-11.67%)`	⬇️

... and 144 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

slbotbm

You've defined MaxTopicSize and IggyExpiry as enums here, but the sdk utilizes normal python ints and timedeltas. Let's change those functions to accept your enums instead of the python ints and structs.

slbotbm

will review in more depth later when I get more time

slbotbm · 2026-07-08T06:15:06Z

+    /// Use the server's default message expiry.
+    ServerDefault(),
+    /// Messages expire after this duration.
+    ExpireDuration { duration: Py<PyDelta> },
+    /// Messages never expire.
+    NeverExpire(),


Please also write exactly what does server default and never expire mean. Also, range of values that expireduration accepts

slbotbm · 2026-07-08T06:16:16Z

+pub enum MaxTopicSize {
+    /// Use the server's default max size.
+    ServerDefault(),
+    /// The topic is limited to this many bytes.
+    Custom { bytes: u64 },
+    /// The topic has no maximum size.
+    Unlimited(),
+}


Same here. Make the docs explicit. We are doing this since the user cannot trace what exactly these mean without looking at rust sdk code.

slbotbm · 2026-07-09T14:12:17Z

/author

mattp5657 · 2026-07-10T12:31:04Z

You've defined MaxTopicSize and IggyExpiry as enums here, but the sdk utilizes normal python ints and timedeltas. Let's change those functions to accept your enums instead of the python ints and structs.

This was an oversight from me, thanks for the call out :)

…d update_topic

mattp5657 · 2026-07-10T23:38:40Z

I made the outlined changes, my only concern is if we should continue with backwards compatibility and incorporate it into testing. But wanted to get your thoughts before doing so. :)

mattp5657 · 2026-07-10T23:39:09Z

/ready

slbotbm

Some comments

slbotbm · 2026-07-12T07:32:49Z

+        match expiry {
+            IggyExpiry::ServerDefault() => RustIggyExpiry::ServerDefault,
+            IggyExpiry::ExpireDuration { duration } => {
+                RustIggyExpiry::ExpireDuration(py_delta_to_iggy_duration(duration))


This also accepts negative duration, which should be rejected as PyValueError

slbotbm · 2026-07-12T07:33:35Z

+    /// the server deletes the oldest sealed segments to make room for new
+    /// messages.
+    ///
+    /// `bytes` must be greater than zero and less than `u64::MAX`: those two


Python callers do not know what u64::MAX means.

mattp5657 · 2026-07-12T23:38:06Z

Fixed both boundary-collision issues flagged in review: negative timedelta values for IggyExpiry.ExpireDuration and 0/u64::MAX values for MaxTopicSize.Custom were silently accepted and misinterpreted on the wire (as ServerDefault/Unlimited) instead of being rejected. Both conversions now raise PyValueError for out-of-range input, propagated through create_topic/update_topic. Updated the affected doc comments/stubs and added a regression test for each invalid case.

mattp5657 · 2026-07-12T23:38:30Z

/ready

hubcio · 2026-07-15T09:34:46Z

    Python::attach(|py| {
        let delta = delta1.bind(py);
-        let seconds = (delta.get_days() * 60 * 60 * 24 + delta.get_seconds()) as u64;
+        let total_seconds = delta.get_days() * 60 * 60 * 24 + delta.get_seconds();


delta.get_days() * 60 * 60 * 24 + delta.get_seconds() is all i32 here (pyo3's get_days/get_seconds both return i32). for |days| >= 24856 (~68yr) the multiply overflows - release wheels wrap silently, debug builds panic. the nasty part is it defeats the negative-duration guard this PR just added right below: a large negative timedelta (easy to get from date arithmetic) wraps to a positive total_seconds, slips past the if total_seconds < 0 check, and a positive expiry gets persisted from a negative input. one-line fix: let total_seconds = i64::from(delta.get_days()) * 86_400 + i64::from(delta.get_seconds()); then range-check before the as u64.

hubcio · 2026-07-15T09:34:46Z

-        #[gen_stub(override_type(type_repr = "datetime.timedelta | None", imports=("datetime")))]
-        message_expiry: Option<Py<PyDelta>>,
-        #[gen_stub(override_type(type_repr = "builtins.int | None"))] max_topic_size: Option<u64>,
+        #[gen_stub(override_type(type_repr = "IggyExpiry | None"))] message_expiry: Option<


note this is a breaking change for create_topic.message_expiry - it went from timedelta | None to IggyExpiry | None, and that param shipped in the released 0.8.0, so existing callers passing a timedelta now hit a TypeError. the max_topic_size retype and all of update_topic are post-0.8.0 so those break nothing released. worth calling out the message_expiry break, at least in the PR descr.

Sorry, I left a comment but hadn't updated the PR description yet. I know it's sometimes easier to follow a PR when the description reflects the original push and comments show how the discussion evolved, but I should have been clearer here, apologies.

I see three options:

Keep the current change. I think it's the most robust approach, but I understand it's a breaking change for message_expiry.

Revert to the original types used to init the server settings (backwards compatible).

Accept both, support the old type alongside the new enum.

My main concern is whether we should preserve backwards compatibility here and add test coverage for it, but I wanted to get your thoughts before going down that path.

hubcio · 2026-07-15T09:34:46Z

+    /// such by the server. A negative `timedelta` raises `ValueError` when
+    /// this value is passed to `create_topic`/`update_topic`. The upper
+    /// bound is whatever a `datetime.timedelta` can represent that also fits
+    /// in a `u64` microsecond count (about 584,942 years); in practice the


this bound isn't accurate today - the timedelta conversion in py_delta_to_iggy_duration overflows i32 at ~68 years, so the real ceiling is far below what's documented. the figure is also a typo: u64 microseconds is ~584,542 years, not 584,942. once the overflow is widened to i64 the ~584,542y bound becomes real and this doc is correct.

hubcio · 2026-07-15T09:34:46Z

+        Ok(match max_size {
+            MaxTopicSize::ServerDefault() => RustMaxTopicSize::ServerDefault,
+            MaxTopicSize::Custom { bytes } => {
+                if *bytes == 0 || *bytes == u64::MAX {


small consistency thing: MaxTopicSize.Custom(0) (and u64::MAX) reject with ValueError here, but the sibling sentinel IggyExpiry.ExpireDuration(timedelta(0)) silently coerces to server-default instead of rejecting. both are documented so it's not a bug, just worth picking one policy across the two - either both reject the reserved value or both coerce it.

hubcio · 2026-07-15T09:34:46Z

+
+    /// The collection of partitions in the topic.
+    #[getter]
+    pub fn partitions(&self) -> Vec<Partition> {


this getter clones the whole partition vec and builds fresh python objects on every access, so topic.partitions looks like a cheap attribute but re-materializes each time (and the returned objects don't have stable identity). it's a cold metadata path so not a perf concern, but worth a doc note or exposing it as a method so callers don't read it repeatedly in a loop.

hubcio · 2026-07-15T09:34:46Z

@@ -169,7 +170,9 @@ impl IggyClient {
    }

    /// Creates a new topic with the given parameters.


minor: create_topic documents its return and raises as free prose while update_topic below uses the structured Args/Returns/Raises form. worth aligning the two.

hubcio · 2026-07-15T09:34:46Z

-            Some(delta) => IggyExpiry::ExpireDuration(py_delta_to_iggy_duration(&delta)),
-            None => IggyExpiry::ServerDefault,
-        };
+        let expiry = message_expiry


this expiry + max_size resolution block is duplicated byte-for-byte in update_topic. could pull it into a small helper (or two) to dedup, no behavior change.

hubcio · 2026-07-15T09:34:46Z

+        ("max_topic_size_bytes", "expected_exception"),
        [
            (-1, OverflowError),
            (2e64, TypeError),


the create_topic version of this test also covers (0, ValueError) and (2**64-1, ValueError) for the reserved sentinels, but they're missing here even though update goes through the same try_from. there's also no update_topic test for a negative ExpireDuration like create has. worth mirroring the create cases.

…ests

mattp5657 · 2026-07-20T18:04:04Z

I have addressed all the comments and added the following to the PR:

create_topic's message_expiry and max_topic_size parameters both
change type on this branch, and both changes are breaking relative to
the last released version (0.8.0):

Parameter	`0.8.0` (released)	This branch
`message_expiry`	`datetime.timedelta \| None`	`IggyExpiry \| None`
`max_topic_size`	`builtins.int \| None`	`MaxTopicSize \| None`

mattp5657 · 2026-07-20T18:04:08Z

/ready

ryerraguntla · 2026-07-21T15:49:20Z

+    }
+}
+
+fn iggy_duration_to_py_delta(duration: Duration) -> Py<PyDelta> {


topic.rs:94-106 — iggy_duration_to_py_delta uses .expect on PyDelta::new; get_topic/message_expiry read path can abort on corrupt wire duration. Fix: map PyErr, raise ValueError.

mattp5657 · 2026-07-22T17:39:23Z

I think this test might be flaky. I'll create an issue to look into it.

server::message_cleanup::message_cleanup::expiry_respects_consumer_offset_expects

Matthew Patton added 2 commits July 6, 2026 19:53

feat(python): expose remaining Topic/TopicDetails fields

f598148

test(python): assert new Topic/TopicDetails fields in topic tests

159c023

mattp5657 marked this pull request as ready for review July 7, 2026 00:01

Merge branch 'master' into feat/python-sdk-topic-details-fields

1c7445c

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 7, 2026

mattp5657 changed the title ~~Feat/python sdk topic details fields~~ feat(python): expose remaining Topic/TopicDetails fields and partitions Jul 7, 2026

slbotbm requested changes Jul 8, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 8, 2026

slbotbm requested changes Jul 8, 2026

View reviewed changes

mattp5657 closed this Jul 9, 2026

github-actions Bot removed the S-waiting-on-author PR is waiting on author response label Jul 9, 2026

mattp5657 reopened this Jul 9, 2026

github-actions Bot added the S-waiting-on-author PR is waiting on author response label Jul 9, 2026

Matthew Patton added 2 commits July 10, 2026 19:16

feat(python): accept IggyExpiry/MaxTopicSize enums in create_topic an…

3758607

…d update_topic

docs(python): spell out IggyExpiry/MaxTopicSize variant semantics

87aabd7

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jul 10, 2026

slbotbm requested changes Jul 12, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 12, 2026

Matthew Patton added 3 commits July 12, 2026 19:25

fix(python): reject negative timedelta durations

b55642f

docs(python): note ValueError on negative message expiry

78b6500

fix(python): reject max topic size at u64 boundary

a511223

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jul 12, 2026

hubcio requested changes Jul 15, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 15, 2026

Matthew Patton added 3 commits July 20, 2026 10:36

fix(python): widen timedelta-to-seconds conversion to i64

d44155e

fix(python): reject IggyExpiry sentinel-boundary durations

529411f

refactor(python): dedup topic param resolution, mirror update_topic t…

9f8445b

…ests

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jul 20, 2026

style(python): rustfmt py_delta_to_iggy_duration

9610f27

ryerraguntla reviewed Jul 21, 2026

View reviewed changes

Matthew Patton added 2 commits July 21, 2026 20:20

Merge branch 'master' into feat/python-sdk-topic-details-fields

4739950

fix(python): raise ValueError on unrepresentable message expiry duration

320135a

slbotbm approved these changes Jul 23, 2026

View reviewed changes

		@@ -169,7 +170,9 @@ impl IggyClient {
		}

		/// Creates a new topic with the given parameters.

Uh oh!

Conversation

mattp5657 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR address?

Rationale

What changed?

Local Execution

Breaking Chnages

AI Usage

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slbotbm commented Jul 9, 2026

Uh oh!

mattp5657 commented Jul 10, 2026

Uh oh!

mattp5657 commented Jul 10, 2026

Uh oh!

mattp5657 commented Jul 10, 2026

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattp5657 commented Jul 12, 2026

Uh oh!

mattp5657 commented Jul 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattp5657 Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattp5657 commented Jul 20, 2026

Uh oh!

mattp5657 commented Jul 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattp5657 commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mattp5657 commented Jul 7, 2026 •

edited

Loading

codecov Bot commented Jul 7, 2026 •

edited

Loading

mattp5657 Jul 15, 2026 •

edited

Loading