Audit: Align PyIceberg Avro schemas and conversion with the Iceberg specification

[kevinjqliu]

Question

Description

An audit of PyIceberg's production Avro definitions and schema-conversion code against the current Iceberg specification found several conformance and reader-compatibility gaps.

This is an umbrella issue. Individual findings can be addressed in separate PRs or issues.

Scope

The audit covered:

• Manifest entry schemas for format versions 1–3
• data_file schemas for format versions 1–3
• Manifest-list (manifest_file) schemas
• Partition field summaries
• Manifest Avro key-value metadata
• Position-delete helper schema
• Iceberg-to-Avro and Avro-to-Iceberg schema conversion

PyIceberg currently defines TableVersion = Literal[1, 2, 3]; format v4 was not treated as an implemented version.

Findings

1. equality_ids is encoded as list<long> instead of list<int>

Tracked separately by apache#3840.

Locations:

• pyiceberg/manifest.py:298
• pyiceberg/manifest.py:393

Current definition:

ListType(
    element_id=136,
    element_type=LongType(),
    element_required=True,
)

The Iceberg specification requires:

135 equality_ids: list<136: int>

This affects the v2 and v3 schemas. V1 is not affected because it does not contain equality_ids.

The incorrect type is embedded in every v2 manifest written by PyIceberg, even when equality_ids is null. Avro permits int to long promotion but not long to int, so conforming readers may reject these manifests.

A fix must write int while retaining compatibility with historical PyIceberg manifests containing long. The read and write schemas may need to be separated.

• Change new manifest output to list<int>
• Continue reading legacy PyIceberg list<long> manifests
• Add tests for both conforming and legacy schemas
• Add interoperability coverage with Java or another conforming implementation

2. V2 manifest metadata omits required schema-id

Location:

• pyiceberg/manifest.py:1117

The V2 writer includes:

• schema
• partition-spec
• partition-spec-id
• format-version
• content

It does not include the separately required schema-id Avro metadata property.

Expected metadata:

"schema-id": str(self._schema.schema_id)

The schema ID embedded inside the schema JSON does not replace this property.

• Write schema-id for V2 manifests
• Consider writing it for V1 as an optional property
• Add metadata conformance tests

3. Manifest readers discard conforming V2 and V3 fields

Locations:

• pyiceberg/manifest.py:874
• pyiceberg/manifest.py:997

Both readers always use DEFAULT_READ_VERSION, currently V2:

DEFAULT_READ_VERSION = 2

Consequences:

1. DATA_FILE_TYPE[2] omits optional V2 field:

   143 referenced_data_file: string
   

   Omitting it from a writer schema is allowed, but the read projection should retain it when another implementation writes it.

2. V3 manifest entries are projected through the V2 schema, discarding:

   • 142 first_row_id
   • 143 referenced_data_file
   • 144 content_offset
   • 145 content_size_in_bytes

3. V3 manifest lists are projected through the V2 schema, discarding:

   • 520 first_row_id

These fields are needed for V3 row lineage and deletion-vector metadata.

• Define a permissive read schema containing all supported V1–V3 fields
• Preserve referenced_data_file when reading conforming V2 manifests
• Preserve V3 row-lineage and deletion-vector fields
• Add properties or accessors for the additional fields
• Add external V2/V3 manifest fixtures

4. Position-delete pos is defined as int instead of long

Also identified in apache#3618.

Location:

• pyiceberg/manifest.py:778

Current definition:

POSITIONAL_DELETE_SCHEMA = Schema(
    NestedField(2147483546, "file_path", StringType()),
    NestedField(2147483545, "pos", IntegerType()),
)

The specification requires field 2147483545 pos to be long.

This schema is currently used only for metrics evaluation and is not written as an Avro file. It should nevertheless be corrected before position-delete writing is implemented.

• Change pos to LongType
• Add a schema assertion test
• Ensure position-delete readers support positions greater than 2**31 - 1

5. Optional list elements are not represented as null unions

Location:

• pyiceberg/utils/schema_conversion.py:553

The Iceberg Avro requirements state that optional array elements must be wrapped in an Avro union with null.

For:

ListType(
    element_id=2,
    element_type=IntegerType(),
    element_required=False,
)

the converter currently emits:

{
  "type": "array",
  "element-id": 2,
  "items": "int"
}

It should emit:

{
  "type": "array",
  "element-id": 2,
  "items": ["null", "int"]
}

• Respect ListType.element_required
• Ensure the stored schema matches the binary writer behavior
• Add required and optional element round-trip tests

6. Optional map values are not represented as null unions

Location:

• pyiceberg/utils/schema_conversion.py:559

For both native Avro maps and logical-map array representations, optional map values must use a union with null.

The logical-map path currently emits the value field directly:

{"name": "value", "type": value_result, "field-id": value_id}

It does not account for MapType.value_required=False.

• Respect MapType.value_required
• Cover both string-key and non-string-key maps
• Ensure map keys remain required
• Add required and optional value round-trip tests

The built-in manifest maps currently have required values, so findings 5 and 6 do not explain the existing manifest interoperability problem.

7. Avro-to-Iceberg conversion does not recognize timestamp-nanos

Location:

• pyiceberg/utils/schema_conversion.py:373

The write converter emits the correct V3 annotations:

{
  "type": "long",
  "logicalType": "timestamp-nanos",
  "adjust-to-utc": false
}

and:

{
  "type": "long",
  "logicalType": "timestamp-nanos",
  "adjust-to-utc": true
}

However, _convert_logical_type only handles timestamp-micros. Parsing an Avro schema containing timestamp-nanos raises an unknown logical-type error.

• Map timestamp-nanos with adjust-to-utc=false to TimestampNanoType
• Map timestamp-nanos with adjust-to-utc=true to TimestamptzNanoType
• Add schema and data round-trip tests

8. Optional UnknownType produces an invalid duplicate union

Locations:

• pyiceberg/utils/schema_conversion.py:526
• pyiceberg/utils/schema_conversion.py:639

UnknownType converts to "null". An optional field then wraps it again:

["null", "null"]

Avro unions cannot contain duplicate branches. The Iceberg specification allows an unknown value to be represented as null or for the field to be omitted.

• Emit a single "null" schema for UnknownType
• Avoid creating duplicate union branches
• Add optional unknown-field conversion tests

Definitions that conform

The following fixed definitions matched the Iceberg V1–V3 field IDs, types, and requiredness rules:

• V1 data_file
• V1–V3 manifest_entry, except for the nested equality_ids issue
• V1–V3 manifest_file
• field_summary
• Partition field IDs copied into the manifest partition struct
• Avro primitive mappings for boolean, int, long, float, double, decimal, date, time, microsecond timestamps, string, UUID, fixed, binary, geometry, and geography
• Field-ID placement for struct fields, list elements, and logical-map key/value fields

Deprecated optional fields omitted by PyIceberg, such as file_ordinal, sort_columns, and distinct_counts, are allowed to be omitted and are not conformance failures.

Support gaps noted during the audit

These are broader feature gaps rather than incorrect existing definitions:

• Manifest and manifest-list writers reject format version 3 despite V3 schema constants being defined.
• Format version 4 is not represented by TableVersion.
• V3 Variant Avro encoding is not implemented.

These may be better tracked separately from the concrete conformance defects above.

References

• Iceberg manifest specification: https://iceberg.apache.org/spec/#manifests
• Iceberg manifest-list specification: https://iceberg.apache.org/spec/#manifest-lists
• Iceberg Avro requirements: https://iceberg.apache.org/spec/#avro
• Iceberg writer requirements: https://iceberg.apache.org/spec/#writer-requirements
• Position-delete specification: https://iceberg.apache.org/spec/#position-delete-files
• Existing equality_ids issue: equality_ids written as list<long>, but spec and all other implementations use list<int> apache/iceberg-python#3840
• Existing position-delete feature issue: Support merge-on-read deletes on v2 tables apache/iceberg-python#3618