[bug] `table.inspect.partitions()` does not respect partition evolution

[kevinjqliu]

Apache Iceberg version

main (development)

Please describe the bug 🐞

Notebook to repro https://gist.github.com/kevinjqliu/e3f454d829d8711ecf9909e11a20add6

Expectation:
First, the table is partitioned by the timestamp field.
After evolving the partition spec by adding the category field, table.inspect.partitions() should still use the table's old partition spec.
Additionally, after adding data using the new partition spec, I should expect to see both old and new partition specs.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[jayceslesar]

Maybe I am interpreting this wrong but I did attempt a fix... it comes down to replacing
pa.field("partition", pa_record_struct, nullable=False), with pa.field("partition", pa.map_(pa.string(), pa.string()), nullable=False), here. This gets us how I believe I am interpreting your expectation, and also makes sense to me because now the operation respects the current manifest, instead of the schema we used to create which would result in the output linked in your notebook of {'timestamp': 17, 'category': None} when we really should just be seeing {'timestamp': 17}

I get pretty confused because when running an existing integration test (test_inspect_partitions_partitioned), we get an error of

    def check_pyiceberg_df_equals_spark_df(df: pa.Table, spark_df: DataFrame) -> None:
        lhs = df.to_pandas().sort_values("spec_id")
        rhs = spark_df.toPandas().sort_values("spec_id")
        for column in df.column_names:
            for left, right in zip(lhs[column].to_list(), rhs[column].to_list()):
>               assert left == right, f"Difference in column {column}: {left} != {right}"
E               AssertionError: Difference in column partition: [('dt_month', '612')] != {'dt_month': 612, 'dt_day': None}
E               assert [('dt_month', '612')] == {'dt_month': 612, 'dt_day': None}
E                 Full diff:
E                 - {'dt_day': None, 'dt_month': 612}
E                 + [('dt_month', '612')]

tests/integration/test_inspect_table.py:589: AssertionError

And either I am not understanding something or spark is also giving us this None back. It is more likely that I am not understanding so any info/help is appreciated! But it does seem that the output does respect partition evolution on the main branch?

I also recognize that the type of the partition column in the table is now different but one issue at a time 😅