Skip to content

Hive Catalog cannot create table with TimestamptzType field#583

Description

@HonahX

Apache Iceberg version

0.6.0 (latest release)

Please describe the bug 馃悶

This bug was reported by Jorge Arada on slack: https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1712309474600169

Reported example to reproduce:

catalog = load_hive("iceberg_catalog", {"uri": "some_url"})
 fields = [NestedField( field_id=0, name="teste_tz", field_type=TimestamptzType(), required=False)]
schema = Schema(*fields)
catalog.create_table(identifier="raw.test_table", schema=schema)

Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 315, in create_table
    sd=_construct_hive_storage_descriptor(schema, location),
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 155, in _construct_hive_storage_descriptor
    [FieldSchema(field.name, visit(field.field_type, SchemaToHiveConverter()), field.doc) for field in schema.fields],
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 155, in <listcomp>
    [FieldSchema(field.name, visit(field.field_type, SchemaToHiveConverter()), field.doc) for field in schema.fields],
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 909, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/schema.py", line 847, in _
    return visitor.primitive(obj)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 228, in primitive
    return HIVE_PRIMITIVE_TYPES[type(primitive)]
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: <class 'pyiceberg.types.TimestamptzType'>

It seems we miss the TimestampzTypein the Iceberg to Hive type conversion map.
A similar issue has been observed and fixed in glue: #366

It would be good to include the fix in 0.6.1

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions