diff --git a/CHANGELOG.md b/CHANGELOG.md index 65d674e4826..4bed55f95f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -208,6 +208,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti ### New Features - Added AWS KMS decryption-key access through the `decryptionKeys` catalog storage configuration property. - Added Kafka PolarisEventListener for publishing events to Kafka. +- Added a Table Metrics Reports REST API (beta) for querying persisted Iceberg scan and commit metrics reports. The OpenAPI spec and query SPI ship in the `polaris-extensions-metrics-reports*` extension modules; the HTTP endpoint returns HTTP 501 unless a durable query backend (e.g. the JDBC extension) is installed. Reads are gated by the new `TABLE_READ_METRICS` privilege. - Added GCS principal attribution for vended credentials (the GCP counterpart of AWS STS session tags). Set `GCS_PRINCIPAL_ATTRIBUTION_ENABLED=true` to activate; the feature flags `GCS_PRINCIPAL_ATTRIBUTION_WIF_AUDIENCE`, `GCS_PRINCIPAL_ATTRIBUTION_TOKEN_ISSUER`, and `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_FILE` are then required (a missing value is a fatal configuration error). Also requires a `gcpServiceAccount` on the catalog StorageConfiguration. When enabled, credential vending chains a catalog-signed JWT through a Workload Identity Federation token exchange and service-account impersonation, so the Polaris principal appears in GCS Data Access audit logs (`serviceAccountDelegationInfo.principalSubject`) for any client. `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_ID` sets the JWT `kid` for JWKS key rotation. Attribution is keyed per-principal in the credential cache; when disabled (default), GCP vending behaviour is unchanged. - Added the `DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED` feature flag (off by default). When enabled, a managed location generated for a table or view created without an explicit location is given a unique, unpredictable suffix, so that no two tables share a path prefix. - Added the `ALLOW_CLIENT_SPECIFIED_TABLE_LOCATION` feature flag (on by default). When set to false, a caller-specified location (the `location` field, a `SetLocation` update, or the `write.data.path` / `write.metadata.path` properties) on a create-table (including a staged create-table request), create-view, update-table, replace-view, or commit-transaction request is rejected, forcing Polaris to manage all locations. Federated catalogs, committing an already staged create, and `register table` / `register view` are unaffected. diff --git a/bom/build.gradle.kts b/bom/build.gradle.kts index 380c84651e9..3d6a2fbd25b 100644 --- a/bom/build.gradle.kts +++ b/bom/build.gradle.kts @@ -31,6 +31,7 @@ dependencies { api(project(":polaris-api-management-model")) api(project(":polaris-api-management-service")) api(project(":polaris-api-openlineage-service")) + api(project(":polaris-extensions-metrics-reports-api")) api(project(":polaris-container-spec-helper")) api(project(":polaris-floci-aws-testcontainer")) @@ -102,6 +103,7 @@ dependencies { api(project(":polaris-extensions-federation-bigquery")) api(project(":polaris-extensions-federation-hadoop")) api(project(":polaris-extensions-federation-hive")) + api(project(":polaris-extensions-metrics-reports-spi")) api(project(":polaris-extensions-metrics-reports")) api(project(":polaris-extensions-metrics-reports-jdbc")) api(project(":polaris-extensions-openlineage")) diff --git a/extensions/auth/ranger/README.md b/extensions/auth/ranger/README.md index 78b1bc480a7..126c3845176 100644 --- a/extensions/auth/ranger/README.md +++ b/extensions/auth/ranger/README.md @@ -38,4 +38,20 @@ polaris.authorization.ranger.authz.audit.destination.solr.urls=http://solr-servi ``` -3. Run or restart Polaris to see that all accesses are authorized by Ranger policies, with access audit records available in Apache Ranger console. +3. Register the Polaris service type with Ranger Admin using the service definition shipped at + `src/main/resources/polaris-ranger-servicedef.json` (packaged as `polaris-ranger-servicedef.json` + on the classpath of `polaris-extensions-auth-ranger`), for example: +``` +curl -u : -X POST \ + -H "Content-Type: application/json" \ + -d @polaris-ranger-servicedef.json \ + http://ranger-admin:6080/service/public/v2/api/servicedef +``` + This is the same `serviceDef` exercised by `RangerPolarisAuthorizerTest` + (`RangerServiceDefConsistencyTest` fails the build if the two drift apart), so any access type + available to grant through Ranger policies is guaranteed to be understood by the authorizer. + +4. Create a Ranger service instance of type `polaris` (matching the `service-name` configured + above), then define policies against it. + +5. Run or restart Polaris to see that all accesses are authorized by Ranger policies, with access audit records available in Apache Ranger console. diff --git a/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json b/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json index 701746f589b..629ddc17e53 100644 --- a/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json +++ b/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json @@ -96,6 +96,7 @@ "table-drop", "table-data-read", "table-data-write", + "table-metrics-read", "table-properties-read", "table-properties-write", "table-properties-set", @@ -338,6 +339,7 @@ "table-create", "table-drop", "table-list", + "table-metrics-read", "table-properties-read", "table-properties-write", "table-properties-set", @@ -423,6 +425,7 @@ "table-statistics-remove" ] }, + { "itemId": 70, "name": "table-metrics-read", "label": "Table Metrics Read", "category": "READ", "impliedGrants": [ "table-list" ] }, { "itemId": 56, "name": "view-create", "label": "View Create", "category": "CREATE", "impliedGrants": [ "view-list" ] }, { "itemId": 57, "name": "view-drop", "label": "View Drop", "category": "DELETE" }, @@ -586,7 +589,13 @@ { "type": "table-statistics-remove" }, { "type": "table-structure-manage" } ], - "users": [ "admin1" ] } + "users": [ "admin1" ] }, + { + "accesses": [ + { "type": "table-data-read" }, + { "type": "table-data-write" } + ], + "users": [ "dataonly1" ] } ] }, { diff --git a/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java b/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java index f8436d9024b..56eded02324 100644 --- a/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java +++ b/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java @@ -86,6 +86,7 @@ enum ResolvedPathRooting { private static final String TABLE_WRITE_PROPERTIES = "table-properties-write"; private static final String TABLE_READ_DATA = "table-data-read"; private static final String TABLE_WRITE_DATA = "table-data-write"; + private static final String TABLE_READ_METRICS = "table-metrics-read"; private static final String TABLE_ATTACH_POLICY = "table-policy-attach"; private static final String TABLE_DETACH_POLICY = "table-policy-detach"; private static final String TABLE_ASSIGN_UUID = "table-uuid-assign"; @@ -254,6 +255,10 @@ enum ResolvedPathRooting { PolarisAuthorizableOperation.REPORT_READ_METRICS, new RangerPolarisOperationSemantics( toSet(TABLE_READ_DATA), null, ResolvedPathRooting.ROOT)); + RBAC_SEMANTICS_BY_OPERATION.put( + PolarisAuthorizableOperation.LIST_TABLE_METRICS, + new RangerPolarisOperationSemantics( + toSet(TABLE_READ_METRICS), null, ResolvedPathRooting.ROOT)); RBAC_SEMANTICS_BY_OPERATION.put( PolarisAuthorizableOperation.REPORT_WRITE_METRICS, new RangerPolarisOperationSemantics( diff --git a/extensions/auth/ranger/src/main/resources/polaris-ranger-servicedef.json b/extensions/auth/ranger/src/main/resources/polaris-ranger-servicedef.json new file mode 100644 index 00000000000..fbf08b6a01e --- /dev/null +++ b/extensions/auth/ranger/src/main/resources/polaris-ranger-servicedef.json @@ -0,0 +1,444 @@ +{ + "name": "polaris", + "displayName": "Polaris (draft)", + "label": "Apache Polaris", + "description": "Apache Polaris", + "guid": "ca1b484b-e397-4ab4-b6e3-36a154662d7d", + "resources": [ + { + "itemId": 1, + "name": "root", + "label": "Root", + "description": "Root", + "parent": "", + "level": 10, + "isValidLeaf": true, + "accessTypeRestrictions": [ + "service-access-manage", + "catalog-create", + "catalog-list", + "principal-create", + "principal-list" + ] + }, + { + "itemId": 2, + "name": "catalog", + "label": "Catalog", + "description": "Catalog", + "parent": "root", + "level": 20, + "isValidLeaf": true, + "accessTypeRestrictions": [ + "catalog-drop", + "catalog-properties-read", + "catalog-properties-write", + "catalog-metadata-full", + "catalog-metadata-manage", + "catalog-content-manage", + "catalog-policy-attach", + "catalog-policy-detach" + ] + }, + { + "itemId": 3, + "name": "principal", + "label": "Principal", + "description": "Principal", + "parent": "root", + "level": 20, + "isValidLeaf": true, + "accessTypeRestrictions": [ + "principal-drop", + "principal-properties-read", + "principal-properties-write", + "principal-metadata-full", + "principal-credentials-rotate", + "principal-credentials-reset" + ] + }, + { + "itemId": 4, + "name": "namespace", + "label": "Namespace", + "description": "Namespace", + "parent": "catalog", + "level": 30, + "isValidLeaf": true, + "accessTypeRestrictions": [ + "namespace-create", + "namespace-drop", + "namespace-list", + "namespace-properties-read", + "namespace-properties-write", + "namespace-metadata-full", + "namespace-policy-attach", + "namespace-policy-detach", + "table-create", + "table-list", + "view-create", + "view-list", + "policy-create", + "policy-list" + ] + }, + { + "itemId": 5, + "name": "table", + "label": "Table", + "description": "Table", + "parent": "namespace", + "level": 40, + "isValidLeaf": true, + "accessTypeRestrictions": [ + "table-drop", + "table-data-read", + "table-data-write", + "table-metrics-read", + "table-properties-read", + "table-properties-write", + "table-properties-set", + "table-properties-remove", + "table-metadata-full", + "table-policy-attach", + "table-policy-detach", + "table-uuid-assign", + "table-format-version-upgrade", + "table-schema-add", + "table-schema-set-current", + "table-partition-spec-add", + "table-partition-specs-remove", + "table-sort-order-add", + "table-sort-order-set-default", + "table-snapshot-add", + "table-snapshots-remove", + "table-snapshot-ref-set", + "table-snapshot-ref-remove", + "table-location-set", + "table-statistics-set", + "table-statistics-remove", + "table-structure-manage", + "view-drop", + "view-properties-read", + "view-properties-write", + "view-metadata-full" + ] + }, + { + "itemId": 6, + "name": "policy", + "label": "Policy", + "description": "Policy", + "parent": "namespace", + "level": 40, + "isValidLeaf": true, + "accessTypeRestrictions": [ + "policy-read", + "policy-drop", + "policy-write", + "policy-metadata-full", + "policy-attach", + "policy-detach" + ] + } + ], + "accessTypes": [ + { "itemId": 1, "name": "service-access-manage", "label": "Service Manage Access", "category": "MANAGE", + "impliedGrants": [ + "catalog-create", + "catalog-drop", + "catalog-list", + "catalog-properties-read", + "catalog-properties-write", + "catalog-metadata-full", + "principal-create", + "principal-drop", + "principal-list", + "principal-properties-read", + "principal-properties-write", + "principal-metadata-full", + "principal-credentials-reset" + ] + }, + + { "itemId": 2, "name": "catalog-create", "label": "Catalog Create", "category": "CREATE", "impliedGrants": [ "catalog-list" ] }, + { "itemId": 3, "name": "catalog-drop", "label": "Catalog Drop", "category": "DELETE" }, + { "itemId": 4, "name": "catalog-list", "label": "Catalog List", "category": "READ" }, + { "itemId": 5, "name": "catalog-content-manage", "label": "Catalog Manage Content", "category": "MANAGE", + "impliedGrants": [ + "catalog-list", + "catalog-metadata-manage", + "catalog-properties-read", + "catalog-properties-write", + "catalog-policy-attach", + "catalog-policy-detach", + "namespace-create", + "namespace-drop", + "namespace-list", + "namespace-metadata-full", + "namespace-properties-read", + "namespace-properties-write", + "namespace-policy-attach", + "namespace-policy-detach", + "table-create", + "table-drop", + "table-list", + "table-data-read", + "table-data-write", + "table-metadata-full", + "table-properties-read", + "table-properties-write", + "table-properties-set", + "table-properties-remove", + "table-uuid-assign", + "table-format-version-upgrade", + "table-schema-add", + "table-schema-set-current", + "table-partition-spec-add", + "table-partition-specs-remove", + "table-policy-attach", + "table-policy-detach", + "table-sort-order-add", + "table-sort-order-set-default", + "table-snapshot-add", + "table-snapshots-remove", + "table-snapshot-ref-set", + "table-snapshot-ref-remove", + "table-location-set", + "table-statistics-set", + "table-statistics-remove", + "table-structure-manage", + "view-create", + "view-drop", + "view-list", + "view-metadata-full", + "view-properties-read", + "view-properties-write", + "policy-create", + "policy-drop", + "policy-list", + "policy-read", + "policy-write", + "policy-attach", + "policy-detach" + ] + }, + { "itemId": 6, "name": "catalog-metadata-full", "label": "Catalog Metadata Full", "category": "MANAGE", "impliedGrants": [ "catalog-create", "catalog-drop", "catalog-list", "catalog-properties-read", "catalog-properties-write" ] }, + { "itemId": 7, "name": "catalog-metadata-manage", "label": "Catalog Metadata Manage", "category": "MANAGE", + "impliedGrants": [ + "catalog-list", + "catalog-properties-read", + "catalog-properties-write", + "catalog-policy-attach", + "catalog-policy-detach", + "namespace-create", + "namespace-drop", + "namespace-list", + "namespace-properties-read", + "namespace-properties-write", + "namespace-metadata-full", + "namespace-policy-attach", + "namespace-policy-detach", + "table-create", + "table-drop", + "table-list", + "table-properties-read", + "table-properties-write", + "table-properties-set", + "table-properties-remove", + "table-metadata-full", + "table-uuid-assign", + "table-format-version-upgrade", + "table-schema-add", + "table-schema-set-current", + "table-partition-spec-add", + "table-partition-specs-remove", + "table-policy-attach", + "table-policy-detach", + "table-sort-order-add", + "table-sort-order-set-default", + "table-snapshot-add", + "table-snapshots-remove", + "table-snapshot-ref-set", + "table-snapshot-ref-remove", + "table-location-set", + "table-statistics-set", + "table-statistics-remove", + "table-structure-manage", + "view-create", + "view-drop", + "view-list", + "view-properties-read", + "view-properties-write", + "view-metadata-full", + "policy-create", + "policy-drop", + "policy-list", + "policy-read", + "policy-write", + "policy-attach", + "policy-detach" + ] + }, + { "itemId": 8, "name": "catalog-policy-attach", "label": "Catalog Policy Attach", "category": "MANAGE" }, + { "itemId": 9, "name": "catalog-policy-detach", "label": "Catalog Policy Detach", "category": "MANAGE" }, + { "itemId": 10, "name": "catalog-properties-read", "label": "Catalog Properties Read", "category": "READ", "impliedGrants": [ "catalog-list" ] }, + { "itemId": 11, "name": "catalog-properties-write", "label": "Catalog Properties Write", "category": "UPDATE", "impliedGrants": [ "catalog-list", "catalog-properties-read" ] }, + + { "itemId": 12, "name": "principal-create", "label": "Principal Create", "category": "CREATE", "impliedGrants": [ "principal-list" ] }, + { "itemId": 13, "name": "principal-drop", "label": "Principal Drop", "category": "DELETE" }, + { "itemId": 14, "name": "principal-list", "label": "Principal List", "category": "READ" }, + { "itemId": 15, "name": "principal-credentials-reset", "label": "Principal Credentials Reset", "category": "MANAGE" }, + { "itemId": 16, "name": "principal-credentials-rotate", "label": "Principal Credentials Rotate", "category": "MANAGE" }, + { "itemId": 17, "name": "principal-metadata-full", "label": "Principal Metadata Full", "category": "MANAGE", "impliedGrants": [ "principal-create", "principal-drop", "principal-list", "principal-properties-read", "principal-properties-write" ] }, + { "itemId": 18, "name": "principal-properties-read", "label": "Principal Properties Read", "category": "READ", "impliedGrants": [ "principal-list" ] }, + { "itemId": 19, "name": "principal-properties-write", "label": "Principal Properties Write", "category": "UPDATE", "impliedGrants": [ "principal-list", "principal-properties-read" ] }, + + { "itemId": 20, "name": "namespace-create", "label": "Namespace Create", "category": "CREATE", "impliedGrants": [ "namespace-list" ] }, + { "itemId": 21, "name": "namespace-drop", "label": "Namespace Drop", "category": "DELETE" }, + { "itemId": 22, "name": "namespace-list", "label": "Namespace List", "category": "READ" }, + { "itemId": 23, "name": "namespace-metadata-full", "label": "Namespace Metadata Full", "category": "MANAGE", "impliedGrants": [ "namespace-create", "namespace-drop", "namespace-list", "namespace-properties-read", "namespace-properties-write" ] }, + { "itemId": 24, "name": "namespace-policy-attach", "label": "Namespace Policy Attach", "category": "MANAGE" }, + { "itemId": 25, "name": "namespace-policy-detach", "label": "Namespace Policy Detach", "category": "MANAGE" }, + { "itemId": 26, "name": "namespace-properties-read", "label": "Namespace Properties Read", "category": "READ", "impliedGrants": [ "namespace-list" ] }, + { "itemId": 27, "name": "namespace-properties-write", "label": "Namespace Properties Write", "category": "UPDATE", "impliedGrants": [ "namespace-list", "namespace-properties-read" ] }, + + { "itemId": 28, "name": "table-create", "label": "Table Create", "category": "CREATE", "impliedGrants": [ "table-list" ] }, + { "itemId": 29, "name": "table-drop", "label": "Table Drop", "category": "DELETE" }, + { "itemId": 30, "name": "table-list", "label": "Table List", "category": "READ" }, + { "itemId": 31, "name": "table-data-read", "label": "Table Data Read", "category": "READ", "impliedGrants": [ "table-list", "table-properties-read" ] }, + { "itemId": 32, "name": "table-data-write", "label": "Table Data Write", "category": "UPDATE", + "impliedGrants": [ + "table-list", + "table-data-read", + "table-properties-read", + "table-properties-set", + "table-properties-remove", + "table-uuid-assign", + "table-format-version-upgrade", + "table-schema-add", + "table-schema-set-current", + "table-sort-order-add", + "table-sort-order-set-default", + "table-snapshot-add", + "table-snapshots-remove", + "table-snapshot-ref-set", + "table-snapshot-ref-remove", + "table-location-set", + "table-statistics-set", + "table-statistics-remove", + "table-partition-spec-add", + "table-partition-specs-remove", + "table-structure-manage" + ] + }, + { "itemId": 33, "name": "table-metadata-full", "label": "Table Metadata Full", "category": "MANAGE", + "impliedGrants": [ + "table-create", + "table-drop", + "table-list", + "table-metrics-read", + "table-properties-read", + "table-properties-write", + "table-properties-set", + "table-properties-remove", + "table-uuid-assign", + "table-format-version-upgrade", + "table-schema-add", + "table-schema-set-current", + "table-partition-spec-add", + "table-partition-specs-remove", + "table-sort-order-add", + "table-sort-order-set-default", + "table-snapshot-add", + "table-snapshots-remove", + "table-snapshot-ref-set", + "table-snapshot-ref-remove", + "table-location-set", + "table-statistics-set", + "table-statistics-remove", + "table-structure-manage" + ] + }, + { "itemId": 34, "name": "table-policy-attach", "label": "Table Policy Attach", "category": "MANAGE" }, + { "itemId": 35, "name": "table-policy-detach", "label": "Table Policy Detach", "category": "MANAGE" }, + { "itemId": 36, "name": "table-properties-read", "label": "Table Properties Read", "category": "READ", "impliedGrants": [ "table-list" ] }, + { "itemId": 37, "name": "table-properties-write", "label": "Table Properties Write", "category": "UPDATE", + "impliedGrants": [ + "table-list", + "table-properties-read", + "table-properties-set", + "table-properties-remove", + "table-uuid-assign", + "table-format-version-upgrade", + "table-schema-add", + "table-schema-set-current", + "table-partition-spec-add", + "table-partition-specs-remove", + "table-sort-order-add", + "table-sort-order-set-default", + "table-snapshot-add", + "table-snapshots-remove", + "table-snapshot-ref-set", + "table-snapshot-ref-remove", + "table-location-set", + "table-statistics-set", + "table-statistics-remove", + "table-structure-manage" + ] + }, + { "itemId": 38, "name": "table-properties-set", "label": "Table Properties Set", "category": "UPDATE" }, + { "itemId": 39, "name": "table-properties-remove", "label": "Table Properties Remove", "category": "UPDATE" }, + { "itemId": 40, "name": "table-uuid-assign", "label": "Table UUID Assign", "category": "UPDATE" }, + { "itemId": 41, "name": "table-format-version-upgrade", "label": "Table Format Version Upgrade", "category": "UPDATE" }, + { "itemId": 42, "name": "table-schema-add", "label": "Table Schema Add", "category": "UPDATE" }, + { "itemId": 43, "name": "table-schema-set-current", "label": "Table Schema Set Current", "category": "UPDATE" }, + { "itemId": 44, "name": "table-partition-spec-add", "label": "Table Partition Spec Add", "category": "UPDATE" }, + { "itemId": 45, "name": "table-partition-specs-remove", "label": "Table Partition Specs Remove", "category": "UPDATE" }, + { "itemId": 46, "name": "table-sort-order-add", "label": "Table Sort Order Add", "category": "UPDATE" }, + { "itemId": 47, "name": "table-sort-order-set-default", "label": "Table Sort Order Set Default", "category": "UPDATE" }, + { "itemId": 48, "name": "table-snapshot-add", "label": "Table Snapshot Add", "category": "UPDATE" }, + { "itemId": 49, "name": "table-snapshots-remove", "label": "Table Snapshots Remove", "category": "UPDATE" }, + { "itemId": 50, "name": "table-snapshot-ref-set", "label": "Table Snapshot-ref Set", "category": "UPDATE" }, + { "itemId": 51, "name": "table-snapshot-ref-remove", "label": "Table Snapshot-ref Remove", "category": "UPDATE" }, + { "itemId": 52, "name": "table-location-set", "label": "Table Location Set", "category": "UPDATE" }, + { "itemId": 53, "name": "table-statistics-set", "label": "Table Statistics Set", "category": "UPDATE" }, + { "itemId": 54, "name": "table-statistics-remove", "label": "Table Statistics Remove", "category": "UPDATE" }, + { "itemId": 55, "name": "table-structure-manage", "label": "Table Structure Manage", "category": "UPDATE", + "impliedGrants": [ + "table-properties-set", + "table-properties-remove", + "table-uuid-assign", + "table-format-version-upgrade", + "table-schema-add", + "table-schema-set-current", + "table-partition-spec-add", + "table-partition-specs-remove", + "table-sort-order-add", + "table-sort-order-set-default", + "table-snapshots-remove", + "table-snapshot-ref-remove", + "table-location-set", + "table-statistics-set", + "table-statistics-remove" + ] + }, + { "itemId": 70, "name": "table-metrics-read", "label": "Table Metrics Read", "category": "READ", "impliedGrants": [ "table-list" ] }, + + { "itemId": 56, "name": "view-create", "label": "View Create", "category": "CREATE", "impliedGrants": [ "view-list" ] }, + { "itemId": 57, "name": "view-drop", "label": "View Drop", "category": "DELETE" }, + { "itemId": 58, "name": "view-list", "label": "View List", "category": "READ" }, + { "itemId": 59, "name": "view-metadata-full", "label": "View Metadata Full", "category": "MANAGE", "impliedGrants": [ "view-create", "view-drop", "view-list", "view-properties-read", "view-properties-write" ] }, + { "itemId": 60, "name": "view-properties-read", "label": "View Properties Read", "category": "READ", "impliedGrants": [ "view-list" ] }, + { "itemId": 61, "name": "view-properties-write", "label": "View Properties Write", "category": "UPDATE", "impliedGrants": [ "view-list", "view-properties-read" ] }, + + { "itemId": 62, "name": "policy-create", "label": "Policy Create", "category": "CREATE", "impliedGrants": [ "policy-list" ] }, + { "itemId": 63, "name": "policy-drop", "label": "Policy Drop", "category": "DELETE" }, + { "itemId": 64, "name": "policy-list", "label": "Policy List", "category": "READ" }, + { "itemId": 65, "name": "policy-read", "label": "Policy Read", "category": "READ", "impliedGrants": [ "policy-list" ] }, + { "itemId": 66, "name": "policy-write", "label": "Policy Write", "category": "UPDATE", "impliedGrants": [ "policy-list", "policy-read" ] }, + { "itemId": 67, "name": "policy-attach", "label": "Policy Attach", "category": "MANAGE" }, + { "itemId": 68, "name": "policy-detach", "label": "Policy Detach", "category": "MANAGE" }, + { "itemId": 69, "name": "policy-metadata-full", "label": "Policy Metadata Full", "category": "MANAGE", "impliedGrants": [ "policy-create", "policy-drop", "policy-list", "policy-read", "policy-write" ] } + ] +} diff --git a/extensions/auth/ranger/src/test/java/org/apache/polaris/extension/auth/ranger/RangerServiceDefConsistencyTest.java b/extensions/auth/ranger/src/test/java/org/apache/polaris/extension/auth/ranger/RangerServiceDefConsistencyTest.java new file mode 100644 index 00000000000..c65fbdffe45 --- /dev/null +++ b/extensions/auth/ranger/src/test/java/org/apache/polaris/extension/auth/ranger/RangerServiceDefConsistencyTest.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.polaris.extension.auth.ranger; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; + +import java.io.InputStream; +import org.junit.jupiter.api.Test; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.json.JsonMapper; + +/** + * Guards against the operator-facing {@code polaris-ranger-servicedef.json} drifting from the + * {@code serviceDef} that the authorization tests actually exercise. Every operation covered by + * {@link RangerPolarisAuthorizerTest} runs against {@code /authz_tests/dev_polaris.json}, so + * keeping the two identical is what makes those tests representative of the artifact operators + * register with Ranger Admin. + */ +public class RangerServiceDefConsistencyTest { + + @Test + public void shippedServiceDefMatchesTestFixture() throws Exception { + JsonMapper mapper = JsonMapper.builder().build(); + + JsonNode shippedServiceDef = readJson(mapper, "/polaris-ranger-servicedef.json"); + JsonNode testFixture = readJson(mapper, "/authz_tests/dev_polaris.json"); + JsonNode testServiceDef = testFixture.get("serviceDef"); + + assertNotNull(testServiceDef, "test fixture is missing a serviceDef"); + assertEquals( + shippedServiceDef, + testServiceDef, + "extensions/auth/ranger/src/main/resources/polaris-ranger-servicedef.json must match " + + "the serviceDef in authz_tests/dev_polaris.json, otherwise the authorization tests " + + "no longer exercise the artifact operators register with Ranger Admin"); + } + + private static JsonNode readJson(JsonMapper mapper, String resourcePath) throws Exception { + try (InputStream in = RangerServiceDefConsistencyTest.class.getResourceAsStream(resourcePath)) { + assertNotNull(in, resourcePath + " not found on classpath"); + return mapper.readTree(in); + } + } +} diff --git a/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json b/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json index f55a3286595..37fa6a74e1a 100644 --- a/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json +++ b/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json @@ -96,6 +96,7 @@ "table-drop", "table-data-read", "table-data-write", + "table-metrics-read", "table-properties-read", "table-properties-write", "table-properties-set", @@ -338,6 +339,7 @@ "table-create", "table-drop", "table-list", + "table-metrics-read", "table-properties-read", "table-properties-write", "table-properties-set", @@ -423,6 +425,7 @@ "table-statistics-remove" ] }, + { "itemId": 70, "name": "table-metrics-read", "label": "Table Metrics Read", "category": "READ", "impliedGrants": [ "table-list" ] }, { "itemId": 56, "name": "view-create", "label": "View Create", "category": "CREATE", "impliedGrants": [ "view-list" ] }, { "itemId": 57, "name": "view-drop", "label": "View Drop", "category": "DELETE" }, @@ -576,7 +579,13 @@ { "type": "table-statistics-remove" }, { "type": "table-structure-manage" } ], - "users": [ "admin1" ] } + "users": [ "admin1" ] }, + { + "accesses": [ + { "type": "table-data-read" }, + { "type": "table-data-write" } + ], + "users": [ "dataonly1" ] } ] }, { diff --git a/extensions/auth/ranger/src/test/resources/authz_tests/tests_authz_table.json b/extensions/auth/ranger/src/test/resources/authz_tests/tests_authz_table.json index 90ebaa2b0b6..16dba5eb8fc 100644 --- a/extensions/auth/ranger/src/test/resources/authz_tests/tests_authz_table.json +++ b/extensions/auth/ranger/src/test/resources/authz_tests/tests_authz_table.json @@ -36,6 +36,14 @@ "request": { "authzOp": "REPORT_WRITE_METRICS", "principal": { "name": "admin1" }, "target": "TABLE_LIKE:POLARIS/catalog1/namespace1/table1" }, "result": { "isAllowed": true } }, + { + "request": { "authzOp": "LIST_TABLE_METRICS", "principal": { "name": "admin1" }, "target": "TABLE_LIKE:POLARIS/catalog1/namespace1/table1" }, + "result": { "isAllowed": true } + }, + { + "request": { "authzOp": "LIST_TABLE_METRICS", "principal": { "name": "dataonly1" }, "target": "TABLE_LIKE:POLARIS/catalog1/namespace1/table1" }, + "result": { "isAllowed": false } + }, { "request": { "authzOp": "ATTACH_POLICY_TO_TABLE", "principal": { "name": "admin1" }, "target": "POLICY:POLARIS/catalog1/namespace1/policy1", "secondary": "TABLE_LIKE:POLARIS/catalog1/namespace1/table1" }, "result": { "isAllowed": true } diff --git a/extensions/metrics-reports/api/build.gradle.kts b/extensions/metrics-reports/api/build.gradle.kts new file mode 100644 index 00000000000..ce80aceb238 --- /dev/null +++ b/extensions/metrics-reports/api/build.gradle.kts @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.openapitools.generator.gradle.plugin.tasks.GenerateTask + +plugins { + alias(libs.plugins.openapi.generator) + id("polaris-client") + id("org.kordamp.gradle.jandex") +} + +dependencies { + implementation(project(":polaris-core")) + + compileOnly(platform(libs.jackson.bom)) + compileOnly("com.fasterxml.jackson.core:jackson-annotations") + + compileOnly(libs.jakarta.annotation.api) + compileOnly(libs.jakarta.inject.api) + compileOnly(libs.jakarta.validation.api) + compileOnly(libs.microprofile.fault.tolerance.api) + compileOnly(libs.swagger.annotations) + + implementation(libs.jakarta.servlet.api) + implementation(libs.jakarta.ws.rs.api) + + compileOnly(platform(libs.micrometer.bom)) + compileOnly("io.micrometer:micrometer-core") + + implementation(libs.slf4j.api) +} + +val rootDir = rootProject.layout.projectDirectory +val specsDir = rootDir.dir("spec") +val templatesDir = rootDir.dir("server-templates") +val generatedDir = project.layout.buildDirectory.dir("generated-openapi") +val generatedOpenApiSrcDir = project.layout.buildDirectory.dir("generated-openapi/src/main/java") + +openApiGenerate { + inputSpec = provider { specsDir.file("metrics-reports-service.yml").asFile.absolutePath } + generatorName = "jaxrs-resteasy" + outputDir = provider { generatedDir.get().asFile.absolutePath } + apiPackage = "org.apache.polaris.service.metrics.api" + modelPackage = "org.apache.polaris.core.metrics.api.model" + ignoreFileOverride.set(provider { rootDir.file(".openapi-generator-ignore").asFile.absolutePath }) + removeOperationIdPrefix.set(true) + templateDir.set(provider { templatesDir.asFile.absolutePath }) + globalProperties.put("apis", "") + globalProperties.put("models", "") + globalProperties.put("apiDocs", "false") + globalProperties.put("modelTests", "false") + configOptions.put("openApiNullable", "false") + configOptions.put("useBeanValidation", "true") + configOptions.put("sourceFolder", "src/main/java") + configOptions.put("useJakartaEe", "true") + configOptions.put("generateBuilders", "true") + configOptions.put("generateConstructorWithAllArgs", "true") + configOptions.put("hideGenerationTimestamp", "true") + additionalProperties.put("apiNamePrefix", "Polaris") + additionalProperties.put("apiNameSuffix", "Api") + additionalProperties.put("metricsPrefix", "polaris") + serverVariables.put("basePath", "api/metrics-reports/v1") +} + +listOf("sourcesJar", "compileJava", "processResources").forEach { task -> + tasks.named(task) { dependsOn("openApiGenerate") } +} + +sourceSets { main { java { srcDir(generatedOpenApiSrcDir) } } } + +tasks.named("openApiGenerate") { + inputs.dir(templatesDir) + inputs.dir(specsDir) + actions.addFirst { delete { delete(generatedDir) } } +} + +tasks.named("javadoc") { dependsOn("jandex") } diff --git a/extensions/metrics-reports/base/build.gradle.kts b/extensions/metrics-reports/base/build.gradle.kts index f2e22cde0e1..210f30a8819 100644 --- a/extensions/metrics-reports/base/build.gradle.kts +++ b/extensions/metrics-reports/base/build.gradle.kts @@ -25,6 +25,7 @@ plugins { dependencies { implementation(project(":polaris-core")) implementation(project(":polaris-runtime-service")) + implementation(project(":polaris-extensions-metrics-reports-spi")) implementation(platform(libs.iceberg.bom)) implementation("org.apache.iceberg:iceberg-api") @@ -35,6 +36,8 @@ dependencies { implementation(libs.slf4j.api) compileOnly(libs.jspecify) + compileOnly(platform(libs.quarkus.bom)) + compileOnly("io.quarkus.arc:arc") testImplementation(platform(libs.junit.bom)) testImplementation("org.junit.jupiter:junit-jupiter") diff --git a/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/NoOpMetricsQuery.java b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/NoOpMetricsQuery.java new file mode 100644 index 00000000000..24b43e26c6f --- /dev/null +++ b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/NoOpMetricsQuery.java @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.extension.metrics.reports; + +import io.quarkus.arc.DefaultBean; +import jakarta.enterprise.context.ApplicationScoped; +import java.util.List; +import org.apache.polaris.core.persistence.pagination.Page; +import org.apache.polaris.core.persistence.pagination.PageToken; +import org.apache.polaris.extension.metrics.spi.MetricsQuerySpi; +import org.jspecify.annotations.NonNull; +import org.jspecify.annotations.Nullable; + +/** + * No-op default implementation of {@link MetricsQuerySpi} that returns empty pages. + * + *

Annotated {@link DefaultBean} so it is only active when no durable query backend (e.g. the + * {@code polaris-extensions-metrics-reports-jdbc} extension) contributes a {@link MetricsQuerySpi}. + * With this default present the read API always resolves a provider: it returns an empty result set + * until a durable backend is installed. + */ +@ApplicationScoped +@DefaultBean +public class NoOpMetricsQuery implements MetricsQuerySpi { + + @Override + public QueryResult listReports( + @NonNull MetricType metricType, + long catalogId, + @NonNull List tableIds, + @Nullable Long snapshotId, + @Nullable Long timestampFrom, + @Nullable Long timestampTo, + @NonNull PageToken pageToken) { + return switch (metricType) { + case SCAN -> new ScanResult(Page.fromItems(List.of())); + case COMMIT -> new CommitResult(Page.fromItems(List.of())); + }; + } +} diff --git a/extensions/metrics-reports/spi/build.gradle.kts b/extensions/metrics-reports/spi/build.gradle.kts new file mode 100644 index 00000000000..6f8df4f89c4 --- /dev/null +++ b/extensions/metrics-reports/spi/build.gradle.kts @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +plugins { + id("polaris-server") + id("org.kordamp.gradle.jandex") +} + +dependencies { + implementation(project(":polaris-core")) + + implementation(libs.guava) + + compileOnly(libs.jspecify) +} diff --git a/extensions/metrics-reports/spi/src/main/java/org/apache/polaris/extension/metrics/spi/MetricsQuerySpi.java b/extensions/metrics-reports/spi/src/main/java/org/apache/polaris/extension/metrics/spi/MetricsQuerySpi.java new file mode 100644 index 00000000000..801968b7eb9 --- /dev/null +++ b/extensions/metrics-reports/spi/src/main/java/org/apache/polaris/extension/metrics/spi/MetricsQuerySpi.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.extension.metrics.spi; + +import com.google.common.annotations.Beta; +import java.util.List; +import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; +import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; +import org.apache.polaris.core.persistence.pagination.Page; +import org.apache.polaris.core.persistence.pagination.PageToken; +import org.jspecify.annotations.NonNull; +import org.jspecify.annotations.Nullable; + +/** + * SPI for querying persisted Iceberg metrics reports. + * + *

Implementations are provided by persistence-backend extension modules (e.g. {@code + * polaris-extensions-metrics-reports-jdbc}). When no implementation is on the classpath, the read + * path returns HTTP 501 Not Implemented. + * + * @see org.apache.polaris.core.persistence.metrics.MetricsPersistence for the corresponding write + * SPI + */ +@Beta +public interface MetricsQuerySpi { + + /** Discriminates which kind of metrics report {@link #listReports} should query. */ + enum MetricType { + SCAN, + COMMIT + } + + /** + * Result of {@link #listReports}, pairing the requested {@link MetricType} with the page of + * records of the matching record type so callers can switch exhaustively without casting. + */ + sealed interface QueryResult permits ScanResult, CommitResult { + MetricType metricType(); + } + + record ScanResult(Page reports) implements QueryResult { + @Override + public MetricType metricType() { + return MetricType.SCAN; + } + } + + record CommitResult(Page reports) implements QueryResult { + @Override + public MetricType metricType() { + return MetricType.COMMIT; + } + } + + /** + * Lists persisted metrics reports of the given {@link MetricType} for the given tables, applying + * the supplied filters and returning at most one page of results merged across all requested + * tables. + * + *

The returned {@link QueryResult#metricType()} must equal {@code metricType}: {@code SCAN} + * must yield a {@link ScanResult} and {@code COMMIT} must yield a {@link CommitResult}. + * + * @param tableIds internal table entity IDs to query, all belonging to {@code catalogId} + */ + QueryResult listReports( + @NonNull MetricType metricType, + long catalogId, + @NonNull List tableIds, + @Nullable Long snapshotId, + @Nullable Long timestampFrom, + @Nullable Long timestampTo, + @NonNull PageToken pageToken); +} diff --git a/gradle/projects.main.properties b/gradle/projects.main.properties index e1f02fb9be1..8e6c7df7b7f 100644 --- a/gradle/projects.main.properties +++ b/gradle/projects.main.properties @@ -25,6 +25,7 @@ polaris-api-management-model=api/management-model polaris-api-management-service=api/management-service polaris-api-catalog-service=api/polaris-catalog-service polaris-api-openlineage-service=api/openlineage-service +polaris-extensions-metrics-reports-api=extensions/metrics-reports/api polaris-runtime-defaults=runtime/defaults polaris-runtime-service=runtime/service polaris-server=runtime/server @@ -54,6 +55,7 @@ polaris-extensions-auth-ranger=extensions/auth/ranger polaris-extensions-events-kafka=extensions/events/kafka polaris-extensions-semantic-models=extensions/semantic-models polaris-extensions-openlineage=extensions/openlineage +polaris-extensions-metrics-reports-spi=extensions/metrics-reports/spi polaris-extensions-metrics-reports=extensions/metrics-reports/base polaris-extensions-metrics-reports-jdbc=extensions/metrics-reports/persistence/relational-jdbc diff --git a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java index 2fc3b6157da..fb0d18d2133 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java @@ -56,6 +56,7 @@ public enum PolarisAuthorizableOperation { VIEW_EXISTS, RENAME_VIEW, REPORT_READ_METRICS, + LIST_TABLE_METRICS, REPORT_WRITE_METRICS, SEND_NOTIFICATIONS, LIST_CATALOGS, diff --git a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java index cdd81ecf242..08a4b8366bf 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java @@ -98,6 +98,7 @@ import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_MANAGE_GRANTS_ON_SECURABLE; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_MANAGE_STRUCTURE; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_DATA; +import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_METRICS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_PROPERTIES; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PARTITION_SPECS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PROPERTIES; @@ -459,6 +460,9 @@ public class PolarisAuthorizerImpl implements PolarisAuthorizer { SUPER_PRIVILEGES.putAll( TABLE_READ_DATA, List.of(CATALOG_MANAGE_CONTENT, TABLE_READ_DATA, TABLE_WRITE_DATA)); SUPER_PRIVILEGES.putAll(TABLE_WRITE_DATA, List.of(CATALOG_MANAGE_CONTENT, TABLE_WRITE_DATA)); + SUPER_PRIVILEGES.putAll( + TABLE_READ_METRICS, + List.of(CATALOG_MANAGE_CONTENT, TABLE_FULL_METADATA, TABLE_READ_METRICS)); SUPER_PRIVILEGES.putAll( NAMESPACE_FULL_METADATA, List.of(CATALOG_MANAGE_CONTENT, CATALOG_MANAGE_METADATA, NAMESPACE_FULL_METADATA)); diff --git a/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java b/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java index e0628805f33..76e522032b6 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java @@ -80,6 +80,7 @@ import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_PRINCIPAL_ROLES_ASSIGNED; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_SEMANTIC_MODEL; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_TABLES; +import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_TABLE_METRICS; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_VIEWS; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LOAD_NAMESPACE_METADATA; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LOAD_POLICY; @@ -199,6 +200,7 @@ import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_LIST; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_MANAGE_GRANTS_ON_SECURABLE; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_DATA; +import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_METRICS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_PROPERTIES; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PARTITION_SPECS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PROPERTIES; @@ -331,6 +333,7 @@ private static void register( // Metrics and notifications register(REPORT_READ_METRICS, TABLE_READ_DATA); + register(LIST_TABLE_METRICS, TABLE_READ_METRICS); register(REPORT_WRITE_METRICS, TABLE_WRITE_DATA); register( SEND_NOTIFICATIONS, diff --git a/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java b/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java index ff7e029ba54..f618696d7de 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java @@ -256,6 +256,19 @@ public enum PolarisPrivilege { PolarisEntityType.TABLE_LIKE, List.of(PolarisEntitySubType.ICEBERG_TABLE, PolarisEntitySubType.GENERIC_TABLE), PolarisEntityType.CATALOG_ROLE), + /** + * Read-only access to table scan and commit metrics reports. Does not grant access to table data. + * Implied by TABLE_FULL_METADATA. + * + *

Restricted to {@link PolarisEntitySubType#ICEBERG_TABLE}: the metrics ingestion and query + * paths only cover Iceberg tables today, and granting this on a generic table would make the + * privilege authorize successfully while returning no reports. + */ + TABLE_READ_METRICS( + 103, + PolarisEntityType.TABLE_LIKE, + List.of(PolarisEntitySubType.ICEBERG_TABLE), + PolarisEntityType.CATALOG_ROLE), ; /** diff --git a/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java b/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java index 14596911fd7..048f9ce8893 100644 --- a/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java +++ b/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java @@ -131,7 +131,8 @@ static Stream polarisPrivileges() { Arguments.of(100, PolarisPrivilege.TABLE_REMOVE_STATISTICS), Arguments.of(101, PolarisPrivilege.TABLE_REMOVE_PARTITION_SPECS), Arguments.of(102, PolarisPrivilege.TABLE_MANAGE_STRUCTURE), - Arguments.of(103, null)); + Arguments.of(103, PolarisPrivilege.TABLE_READ_METRICS), + Arguments.of(104, null)); } @ParameterizedTest diff --git a/runtime/service/build.gradle.kts b/runtime/service/build.gradle.kts index f9d588399e3..8a4e53de58b 100644 --- a/runtime/service/build.gradle.kts +++ b/runtime/service/build.gradle.kts @@ -30,6 +30,8 @@ dependencies { implementation(project(":polaris-api-management-service")) implementation(project(":polaris-api-iceberg-service")) implementation(project(":polaris-api-catalog-service")) + implementation(project(":polaris-extensions-metrics-reports-api")) + implementation(project(":polaris-extensions-metrics-reports-spi")) runtimeOnly(project(":polaris-relational-jdbc")) diff --git a/runtime/service/src/main/java/org/apache/polaris/service/metrics/MetricsReportsService.java b/runtime/service/src/main/java/org/apache/polaris/service/metrics/MetricsReportsService.java new file mode 100644 index 00000000000..2c74430bb1f --- /dev/null +++ b/runtime/service/src/main/java/org/apache/polaris/service/metrics/MetricsReportsService.java @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.metrics; + +import com.google.common.annotations.Beta; +import com.google.common.base.Preconditions; +import jakarta.enterprise.context.RequestScoped; +import jakarta.enterprise.inject.Any; +import jakarta.enterprise.inject.Instance; +import jakarta.inject.Inject; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.exceptions.NotFoundException; +import org.apache.polaris.core.auth.AuthorizationIntent; +import org.apache.polaris.core.auth.AuthorizationRequest; +import org.apache.polaris.core.auth.AuthorizationState; +import org.apache.polaris.core.auth.PolarisAuthorizableOperation; +import org.apache.polaris.core.auth.PolarisAuthorizer; +import org.apache.polaris.core.auth.PolarisPrincipal; +import org.apache.polaris.core.auth.SingleTargetAuthorizationIntent; +import org.apache.polaris.core.catalog.PolarisCatalogHelpers; +import org.apache.polaris.core.context.RealmContext; +import org.apache.polaris.core.entity.CatalogEntity; +import org.apache.polaris.core.entity.PolarisEntitySubType; +import org.apache.polaris.core.entity.PolarisEntityType; +import org.apache.polaris.core.metrics.api.model.CommitMetricsObject; +import org.apache.polaris.core.metrics.api.model.CommitMetricsReport; +import org.apache.polaris.core.metrics.api.model.CommitPayload; +import org.apache.polaris.core.metrics.api.model.CommitPayloadData; +import org.apache.polaris.core.metrics.api.model.ListCommitMetricsResponse; +import org.apache.polaris.core.metrics.api.model.ListScanMetricsResponse; +import org.apache.polaris.core.metrics.api.model.MetricsActor; +import org.apache.polaris.core.metrics.api.model.MetricsRequest; +import org.apache.polaris.core.metrics.api.model.QueryMetricsRequest; +import org.apache.polaris.core.metrics.api.model.ScanMetricsObject; +import org.apache.polaris.core.metrics.api.model.ScanMetricsReport; +import org.apache.polaris.core.metrics.api.model.ScanPayload; +import org.apache.polaris.core.metrics.api.model.ScanPayloadData; +import org.apache.polaris.core.metrics.api.model.TableRef; +import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper; +import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; +import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; +import org.apache.polaris.core.persistence.pagination.PageToken; +import org.apache.polaris.core.persistence.resolver.PolarisResolutionManifest; +import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory; +import org.apache.polaris.core.persistence.resolver.ResolvedPathKey; +import org.apache.polaris.core.persistence.resolver.ResolverPath; +import org.apache.polaris.core.persistence.resolver.ResolverStatus; +import org.apache.polaris.extension.metrics.spi.MetricsQuerySpi; +import org.apache.polaris.service.catalog.common.PolarisSecurableMapper; +import org.apache.polaris.service.metrics.api.PolarisCatalogsApiService; +import org.jspecify.annotations.NonNull; + +/** + * Service implementation for the Metrics Reports API. + * + *

Resolves catalog/namespace/table names to internal IDs, performs authorization, and delegates + * durable reads to {@link MetricsQuerySpi} when an implementation is available. + */ +@Beta +@RequestScoped +public class MetricsReportsService implements PolarisCatalogsApiService { + + private final PolarisAuthorizer authorizer; + private final PolarisPrincipal polarisPrincipal; + private final ResolutionManifestFactory resolutionManifestFactory; + private final Instance queryProvider; + + @Inject + public MetricsReportsService( + @NonNull PolarisAuthorizer authorizer, + @NonNull PolarisPrincipal polarisPrincipal, + @NonNull ResolutionManifestFactory resolutionManifestFactory, + @Any Instance queryProvider) { + this.authorizer = authorizer; + this.polarisPrincipal = polarisPrincipal; + this.resolutionManifestFactory = resolutionManifestFactory; + this.queryProvider = queryProvider; + } + + @Override + public Response queryTableMetrics( + String catalogName, + QueryMetricsRequest request, + RealmContext realmContext, + SecurityContext securityContext) { + + List tableRefs = request.getTables(); + if (tableRefs == null || tableRefs.isEmpty()) { + throw new IllegalArgumentException("tables must not be empty"); + } + + List identifiers = + tableRefs.stream() + .map( + ref -> + TableIdentifier.of( + Namespace.of(ref.getNamespace().toArray(new String[0])), ref.getName())) + .toList(); + + PolarisResolutionManifest manifest = resolveAndAuthorizeTableMetrics(catalogName, identifiers); + + CatalogEntity catalogEntity = manifest.getResolvedCatalogEntity(); + Preconditions.checkNotNull(catalogEntity, "No catalog available"); + long catalogId = catalogEntity.getId(); + + List tableIds = new ArrayList<>(identifiers.size()); + Map tableIdToIdentifier = new LinkedHashMap<>(); + for (TableIdentifier identifier : identifiers) { + PolarisResolvedPathWrapper tableWrapper = + manifest.getResolvedPath( + ResolvedPathKey.ofTableLike(identifier), PolarisEntitySubType.ICEBERG_TABLE, true); + long tableId = tableWrapper.getRawLeafEntity().getId(); + tableIds.add(tableId); + tableIdToIdentifier.put(tableId, identifier); + } + + MetricsQuerySpi.MetricType type = parseMetricType(request.getMetricType().toString()); + PageToken pt = PageToken.build(request.getPageToken(), request.getPageSize(), () -> true); + MetricsQuerySpi provider = queryProvider.get(); + + MetricsQuerySpi.QueryResult result = + provider.listReports( + type, + catalogId, + tableIds, + request.getSnapshotId(), + request.getTimestampFrom(), + request.getTimestampTo(), + pt); + + Preconditions.checkState( + result.metricType() == type, "Provider returned %s for %s", result.metricType(), type); + + return switch (result) { + case MetricsQuerySpi.ScanResult scan -> toScanResponse(scan, tableIdToIdentifier); + case MetricsQuerySpi.CommitResult commit -> toCommitResponse(commit, tableIdToIdentifier); + }; + } + + private static Response toScanResponse( + MetricsQuerySpi.ScanResult scan, Map tableIdToIdentifier) { + List reports = + scan.reports().items().stream() + .map(r -> toScanReport(r, tableIdToIdentifier.get(r.tableId()))) + .toList(); + return Response.ok( + new ListScanMetricsResponse( + scan.reports().encodedResponseToken(), + ListScanMetricsResponse.MetricTypeEnum.SCAN, + reports)) + .build(); + } + + private static Response toCommitResponse( + MetricsQuerySpi.CommitResult commit, Map tableIdToIdentifier) { + List reports = + commit.reports().items().stream() + .map(r -> toCommitReport(r, tableIdToIdentifier.get(r.tableId()))) + .toList(); + return Response.ok( + new ListCommitMetricsResponse( + commit.reports().encodedResponseToken(), + ListCommitMetricsResponse.MetricTypeEnum.COMMIT, + reports)) + .build(); + } + + private static MetricsQuerySpi.MetricType parseMetricType(String metricType) { + if ("scan".equalsIgnoreCase(metricType)) { + return MetricsQuerySpi.MetricType.SCAN; + } + if ("commit".equalsIgnoreCase(metricType)) { + return MetricsQuerySpi.MetricType.COMMIT; + } + throw new IllegalArgumentException( + "metricType must be one of [scan, commit], got: " + metricType); + } + + private PolarisResolutionManifest resolveAndAuthorizeTableMetrics( + String catalogName, List identifiers) { + PolarisResolutionManifest manifest = + resolutionManifestFactory.createResolutionManifest(polarisPrincipal, catalogName); + for (TableIdentifier identifier : identifiers) { + manifest.addPassthroughPath( + new ResolverPath( + Arrays.asList(identifier.namespace().levels()), PolarisEntityType.NAMESPACE)); + manifest.addPassthroughPath( + new ResolverPath( + PolarisCatalogHelpers.tableIdentifierToList(identifier), + PolarisEntityType.TABLE_LIKE)); + } + ResolverStatus status = manifest.resolveAll(); + + if (status.getStatus() == ResolverStatus.StatusEnum.ENTITY_COULD_NOT_BE_RESOLVED) { + throw new NotFoundException( + "TopLevelEntity of type %s does not exist: %s", + status.getFailedToResolvedEntityType(), status.getFailedToResolvedEntityName()); + } + if (status.getStatus() == ResolverStatus.StatusEnum.PATH_COULD_NOT_BE_FULLY_RESOLVED) { + throw new NotFoundException("Table not found"); + } + + AuthorizationRequest authorizationRequest = + new AuthorizationRequest( + polarisPrincipal, + identifiers.stream() + .map( + identifier -> + new SingleTargetAuthorizationIntent( + PolarisAuthorizableOperation.LIST_TABLE_METRICS, + PolarisSecurableMapper.tableLike(catalogName, identifier))) + .toList()); + AuthorizationState authorizationState = new AuthorizationState(manifest); + authorizer.resolveAuthorizationInputs(authorizationState, authorizationRequest); + + for (TableIdentifier identifier : identifiers) { + PolarisResolvedPathWrapper tableWrapper = + manifest.getResolvedPath( + ResolvedPathKey.ofTableLike(identifier), PolarisEntitySubType.ICEBERG_TABLE, true); + + if (tableWrapper == null) { + throw new NotFoundException("Table not found: %s", identifier); + } + } + + authorizer.authorize(authorizationState, authorizationRequest).throwIfDenied(); + + return manifest; + } + + private static ScanMetricsReport toScanReport(ScanMetricsRecord r, TableIdentifier identifier) { + MetricsActor actor = r.principalName() != null ? new MetricsActor(r.principalName()) : null; + MetricsRequest request = + (r.requestId() != null || r.otelTraceId() != null || r.otelSpanId() != null) + ? new MetricsRequest(r.requestId(), r.otelTraceId(), r.otelSpanId()) + : null; + ScanMetricsObject object = + new ScanMetricsObject(toTableRef(identifier), r.snapshotId().orElse(null)); + ScanPayloadData data = + ScanPayloadData.builder() + .setSchemaId(r.schemaId().orElse(null)) + .setFilterExpression(r.filterExpression().orElse(null)) + .setProjectedFieldIds(r.projectedFieldIds()) + .setProjectedFieldNames(r.projectedFieldNames()) + .setResultDataFiles(r.resultDataFiles()) + .setResultDeleteFiles(r.resultDeleteFiles()) + .setTotalFileSizeBytes(r.totalFileSizeBytes()) + .setTotalDataManifests(r.totalDataManifests()) + .setTotalDeleteManifests(r.totalDeleteManifests()) + .setScannedDataManifests(r.scannedDataManifests()) + .setScannedDeleteManifests(r.scannedDeleteManifests()) + .setSkippedDataManifests(r.skippedDataManifests()) + .setSkippedDeleteManifests(r.skippedDeleteManifests()) + .setSkippedDataFiles(r.skippedDataFiles()) + .setSkippedDeleteFiles(r.skippedDeleteFiles()) + .setTotalPlanningDurationMs(r.totalPlanningDurationMs()) + .setEqualityDeleteFiles(r.equalityDeleteFiles()) + .setPositionalDeleteFiles(r.positionalDeleteFiles()) + .setIndexedDeleteFiles(r.indexedDeleteFiles()) + .setTotalDeleteFileSizeBytes(r.totalDeleteFileSizeBytes()) + .build(); + ScanPayload payload = + new ScanPayload( + ScanPayload.TypeEnum.ICEBERG_METRICS_SCAN, ScanPayload.VersionEnum.NUMBER_1, data); + return new ScanMetricsReport( + r.reportId(), r.timestamp().toEpochMilli(), actor, request, object, payload); + } + + private static CommitMetricsReport toCommitReport( + CommitMetricsRecord r, TableIdentifier identifier) { + MetricsActor actor = r.principalName() != null ? new MetricsActor(r.principalName()) : null; + MetricsRequest request = + (r.requestId() != null || r.otelTraceId() != null || r.otelSpanId() != null) + ? new MetricsRequest(r.requestId(), r.otelTraceId(), r.otelSpanId()) + : null; + CommitMetricsObject object = new CommitMetricsObject(toTableRef(identifier), r.snapshotId()); + CommitPayloadData data = + CommitPayloadData.builder() + .setSequenceNumber(r.sequenceNumber().orElse(null)) + .setOperation(r.operation()) + .setAddedDataFiles(r.addedDataFiles()) + .setRemovedDataFiles(r.removedDataFiles()) + .setTotalDataFiles(r.totalDataFiles()) + .setAddedDeleteFiles(r.addedDeleteFiles()) + .setRemovedDeleteFiles(r.removedDeleteFiles()) + .setTotalDeleteFiles(r.totalDeleteFiles()) + .setAddedEqualityDeleteFiles(r.addedEqualityDeleteFiles()) + .setRemovedEqualityDeleteFiles(r.removedEqualityDeleteFiles()) + .setAddedPositionalDeleteFiles(r.addedPositionalDeleteFiles()) + .setRemovedPositionalDeleteFiles(r.removedPositionalDeleteFiles()) + .setAddedRecords(r.addedRecords()) + .setRemovedRecords(r.removedRecords()) + .setTotalRecords(r.totalRecords()) + .setAddedFileSizeBytes(r.addedFileSizeBytes()) + .setRemovedFileSizeBytes(r.removedFileSizeBytes()) + .setTotalFileSizeBytes(r.totalFileSizeBytes()) + .setTotalDurationMs(r.totalDurationMs().orElse(null)) + .setAttempts(r.attempts()) + .build(); + CommitPayload payload = + new CommitPayload( + CommitPayload.TypeEnum.ICEBERG_METRICS_COMMIT, + CommitPayload.VersionEnum.NUMBER_1, + data); + return new CommitMetricsReport( + r.reportId(), r.timestamp().toEpochMilli(), actor, request, object, payload); + } + + private static TableRef toTableRef(TableIdentifier identifier) { + return new TableRef(Arrays.asList(identifier.namespace().levels()), identifier.name()); + } +} diff --git a/runtime/service/src/test/java/org/apache/polaris/service/metrics/MetricsReportsServiceTest.java b/runtime/service/src/test/java/org/apache/polaris/service/metrics/MetricsReportsServiceTest.java new file mode 100644 index 00000000000..431012014ae --- /dev/null +++ b/runtime/service/src/test/java/org/apache/polaris/service/metrics/MetricsReportsServiceTest.java @@ -0,0 +1,242 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.metrics; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import jakarta.enterprise.inject.Instance; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import java.util.List; +import java.util.Set; +import org.apache.iceberg.exceptions.ForbiddenException; +import org.apache.iceberg.exceptions.NotFoundException; +import org.apache.polaris.core.auth.AuthorizationDecision; +import org.apache.polaris.core.auth.PolarisAuthorizer; +import org.apache.polaris.core.auth.PolarisPrincipal; +import org.apache.polaris.core.context.RealmContext; +import org.apache.polaris.core.entity.CatalogEntity; +import org.apache.polaris.core.entity.PolarisEntity; +import org.apache.polaris.core.entity.PolarisEntitySubType; +import org.apache.polaris.core.entity.PolarisEntityType; +import org.apache.polaris.core.metrics.api.model.QueryMetricsRequest; +import org.apache.polaris.core.metrics.api.model.TableRef; +import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper; +import org.apache.polaris.core.persistence.pagination.Page; +import org.apache.polaris.core.persistence.pagination.PageToken; +import org.apache.polaris.core.persistence.resolver.PolarisResolutionManifest; +import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory; +import org.apache.polaris.core.persistence.resolver.ResolvedPathKey; +import org.apache.polaris.core.persistence.resolver.ResolverPath; +import org.apache.polaris.core.persistence.resolver.ResolverStatus; +import org.apache.polaris.extension.metrics.spi.MetricsQuerySpi; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +/** + * Unit tests for {@link MetricsReportsService}. + * + *

Without a durable query backend the no-op default provider returns empty pages; the durable + * JDBC implementation (#4756) returns real data. These tests cover authorization, resolution error + * paths, and input validation. + */ +class MetricsReportsServiceTest { + + private static final String CATALOG = "test-catalog"; + private static final List NAMESPACE = List.of("db", "schema"); + private static final String TABLE = "events"; + + private PolarisAuthorizer authorizer; + private PolarisResolutionManifest manifest; + private PolarisPrincipal principal; + private ResolutionManifestFactory factory; + private MetricsReportsService service; + private RealmContext realmContext; + private SecurityContext securityContext; + private Instance queryProvider; + + @BeforeEach + void setUp() { + authorizer = mock(PolarisAuthorizer.class); + principal = mock(PolarisPrincipal.class); + + PolarisResolvedPathWrapper tableWrapper = mock(PolarisResolvedPathWrapper.class); + PolarisEntity leafEntity = mock(PolarisEntity.class); + when(leafEntity.getId()).thenReturn(42L); + when(tableWrapper.getRawLeafEntity()).thenReturn(leafEntity); + manifest = mock(PolarisResolutionManifest.class); + factory = mock(ResolutionManifestFactory.class); + realmContext = mock(RealmContext.class); + securityContext = mock(SecurityContext.class); + + CatalogEntity catalogEntity = mock(CatalogEntity.class); + when(catalogEntity.getId()).thenReturn(7L); + + when(manifest.resolveAll()).thenReturn(new ResolverStatus(ResolverStatus.StatusEnum.SUCCESS)); + when(manifest.getResolvedCatalogEntity()).thenReturn(catalogEntity); + when(manifest.getResolvedPath( + any(ResolvedPathKey.class), eq(PolarisEntitySubType.ICEBERG_TABLE), eq(true))) + .thenReturn(tableWrapper); + when(manifest.getAllActivatedCatalogRoleAndPrincipalRoles()).thenReturn(Set.of()); + when(factory.createResolutionManifest(eq(principal), eq(CATALOG))).thenReturn(manifest); + when(authorizer.authorize(any(), any())).thenReturn(AuthorizationDecision.allow()); + + // By default the no-op query provider is active (durable backend absent) and returns + // empty pages, mirroring the @DefaultBean NoOpMetricsQuery in + // polaris-extensions-metrics-reports. + MetricsQuerySpi noOp = mock(MetricsQuerySpi.class); + when(noOp.listReports( + eq(MetricsQuerySpi.MetricType.SCAN), + anyLong(), + any(), + any(), + any(), + any(), + any(PageToken.class))) + .thenReturn(new MetricsQuerySpi.ScanResult(Page.fromItems(List.of()))); + when(noOp.listReports( + eq(MetricsQuerySpi.MetricType.COMMIT), + anyLong(), + any(), + any(), + any(), + any(), + any(PageToken.class))) + .thenReturn(new MetricsQuerySpi.CommitResult(Page.fromItems(List.of()))); + @SuppressWarnings("unchecked") + Instance noOpProvider = mock(Instance.class); + when(noOpProvider.get()).thenReturn(noOp); + queryProvider = noOpProvider; + + service = new MetricsReportsService(authorizer, principal, factory, queryProvider); + realmContext = mock(RealmContext.class); + securityContext = mock(SecurityContext.class); + } + + private static QueryMetricsRequest requestFor( + String metricType, List namespace, String table) { + return new QueryMetricsRequest( + "scan".equals(metricType) + ? QueryMetricsRequest.MetricTypeEnum.SCAN + : QueryMetricsRequest.MetricTypeEnum.COMMIT, + List.of(new TableRef(namespace, table))); + } + + @Test + void authorizedRequestWithNoBackendReturnsEmptyPage() { + Response response = + service.queryTableMetrics( + CATALOG, requestFor("scan", NAMESPACE, TABLE), realmContext, securityContext); + + // With the no-op default query provider, the read path succeeds with an empty result set. + assertThat(response.getStatus()).isEqualTo(Response.Status.OK.getStatusCode()); + } + + @Test + void unauthorizedRequestThrowsForbiddenException() { + when(authorizer.authorize(any(), any())).thenReturn(AuthorizationDecision.deny("denied")); + + assertThatThrownBy( + () -> + service.queryTableMetrics( + CATALOG, requestFor("scan", NAMESPACE, TABLE), realmContext, securityContext)) + .isInstanceOf(ForbiddenException.class); + } + + @Test + void tableNotFoundThrowsNotFoundException() { + when(manifest.getResolvedPath( + any(ResolvedPathKey.class), eq(PolarisEntitySubType.ICEBERG_TABLE), eq(true))) + .thenReturn(null); + + assertThatThrownBy( + () -> + service.queryTableMetrics( + CATALOG, requestFor("scan", NAMESPACE, TABLE), realmContext, securityContext)) + .isInstanceOf(NotFoundException.class) + .hasMessageContaining(TABLE); + } + + @Test + void catalogNotFoundPropagatesNotFoundException() { + when(manifest.resolveAll()).thenReturn(new ResolverStatus(PolarisEntityType.CATALOG, CATALOG)); + + assertThatThrownBy( + () -> + service.queryTableMetrics( + CATALOG, requestFor("scan", NAMESPACE, TABLE), realmContext, securityContext)) + .isInstanceOf(NotFoundException.class) + .hasMessageContaining(CATALOG); + } + + @Test + void pathNotFoundPropagatesNotFoundException() { + ResolverPath failedPath = new ResolverPath(NAMESPACE, PolarisEntityType.NAMESPACE); + when(manifest.resolveAll()).thenReturn(new ResolverStatus(failedPath, 0)); + + assertThatThrownBy( + () -> + service.queryTableMetrics( + CATALOG, requestFor("scan", NAMESPACE, TABLE), realmContext, securityContext)) + .isInstanceOf(NotFoundException.class); + } + + @Test + void emptyTablesThrowsIllegalArgumentException() { + QueryMetricsRequest request = + new QueryMetricsRequest(QueryMetricsRequest.MetricTypeEnum.SCAN, List.of()); + + assertThatThrownBy( + () -> service.queryTableMetrics(CATALOG, request, realmContext, securityContext)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("tables"); + } + + @Test + void multiTableRequestQueriesAllResolvedTableIds() { + Response response = + service.queryTableMetrics( + CATALOG, + new QueryMetricsRequest( + QueryMetricsRequest.MetricTypeEnum.SCAN, + List.of(new TableRef(NAMESPACE, TABLE), new TableRef(NAMESPACE, "other-table"))), + realmContext, + securityContext); + + assertThat(response.getStatus()).isEqualTo(Response.Status.OK.getStatusCode()); + } + + @Test + void multiLevelNamespaceIsPassedThrough() { + Response response = + service.queryTableMetrics( + CATALOG, + requestFor("scan", List.of("db", "schema"), TABLE), + realmContext, + securityContext); + + assertThat(response.getStatus()).isEqualTo(Response.Status.OK.getStatusCode()); + } +} diff --git a/site/content/in-dev/unreleased/managing-security/access-control.md b/site/content/in-dev/unreleased/managing-security/access-control.md index 40752651b30..6bb45888649 100644 --- a/site/content/in-dev/unreleased/managing-security/access-control.md +++ b/site/content/in-dev/unreleased/managing-security/access-control.md @@ -115,6 +115,7 @@ To grant the full set of privileges (drop, list, read, write, etc.) on an object | TABLE_WRITE_PROPERTIES | Enables configuring properties for the table. | | TABLE_READ_DATA | Enables reading data from the table by receiving short-lived read-only storage credentials from the catalog. | | TABLE_WRITE_DATA | Enables writing data to the table by receiving short-lived read+write storage credentials from the catalog. | +| TABLE_READ_METRICS | Enables reading persisted Iceberg scan and commit metrics reports for the table via the Metrics Reports API. | | TABLE_FULL_METADATA | Grants all table privileges, except TABLE_READ_DATA and TABLE_WRITE_DATA, which need to be granted individually. | | TABLE_ATTACH_POLICY | Enables attaching policy to a table. | | TABLE_DETACH_POLICY | Enables detaching policy from a table. | diff --git a/spec/README.md b/spec/README.md index 21e7288727b..4bb75d4fef0 100644 --- a/spec/README.md +++ b/spec/README.md @@ -45,6 +45,9 @@ Apache Polaris provides the following OpenAPI specifications: - [oauth-tokens-api.yaml](polaris-catalog-apis/oauth-tokens-api.yaml) - Contains the specification for the internal OAuth Token endpoint, extracted from the Apache Iceberg REST Catalog API. +- [metrics-reports-service.yml](metrics-reports-service.yml) - Defines the experimental, read-only API for querying + persisted Iceberg table metrics (scan and commit reports). + ## Generated Specification Files The specification files in the generated folder are automatically created using OpenAPI bundling tools such as diff --git a/spec/metrics-reports-service.yml b/spec/metrics-reports-service.yml new file mode 100644 index 00000000000..f73b9272fa5 --- /dev/null +++ b/spec/metrics-reports-service.yml @@ -0,0 +1,501 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +openapi: 3.0.3 +info: + title: Apache Polaris Metrics Reports API + description: > + **Experimental / Beta**: Read-only API for querying Iceberg table metrics (scan and commit + reports) from Apache Polaris. + + + **This API is experimental and subject to change, including breaking changes, in any future + release without prior notice. It should not be used in production environments. The "beta" + label indicates early-access status, not stability.** + + + Requires TABLE_READ_METRICS privilege on every requested table. Durable persistence backing for + this API is provided by the polaris-extensions-metrics-reports-jdbc extension; without it the + endpoint returns HTTP 501. + version: 0.1.0 + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0.html + +servers: + - url: "{scheme}://{host}/api/metrics-reports/v1" + variables: + scheme: + default: https + host: + default: localhost + +paths: + /catalogs/{catalogName}/metrics/query: + parameters: + - $ref: '#/components/parameters/catalogName' + post: + operationId: queryTableMetrics + summary: Query metrics reports for one or more tables + description: > + Returns persisted metrics reports for the tables listed in the request body. All requested + tables must belong to the catalog identified by `catalogName`. The required `metricType` + selects between scan reports (produced during table reads) and commit reports (produced + during table writes). Results from all requested tables are merged into a single page, + ordered by timestamp descending, and disambiguated by the `table` field on each report's + `object`. Requires TABLE_READ_METRICS privilege on every requested table. + tags: + - Metrics + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/QueryMetricsRequest' + responses: + '200': + description: Paginated list of metrics reports + content: + application/json: + schema: + $ref: '#/components/schemas/ListMetricsResponse' + '400': + description: Bad request (missing or invalid parameters) + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + '403': + description: Insufficient privileges (TABLE_READ_METRICS required) + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + '404': + description: Catalog, namespace, or table not found + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + '501': + description: Durable metrics query backing is not available in this deployment + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + +components: + parameters: + catalogName: + name: catalogName + in: path + required: true + schema: + type: string + + schemas: + TableRef: + type: object + description: Identifies a table by its namespace levels and name + required: + - namespace + - name + properties: + namespace: + type: array + items: + type: string + description: Namespace levels, e.g. ["a", "b"] for namespace a.b + name: + type: string + + QueryMetricsRequest: + type: object + required: + - metricType + - tables + properties: + metricType: + type: string + enum: [scan, commit] + description: Type of metrics to retrieve + tables: + type: array + minItems: 1 + description: > + Tables to query. All tables must belong to the catalog identified by catalogName in + the path. + items: + $ref: '#/components/schemas/TableRef' + pageToken: + type: string + description: Opaque cursor from a previous response's nextPageToken field + pageSize: + type: integer + minimum: 1 + default: 100 + description: Maximum number of results to return per page + snapshotId: + type: integer + format: int64 + description: Filter results to a specific snapshot ID + timestampFrom: + type: integer + format: int64 + description: Inclusive lower bound on report timestamp (Unix epoch milliseconds) + timestampTo: + type: integer + format: int64 + description: Exclusive upper bound on report timestamp (Unix epoch milliseconds) + + ListMetricsResponse: + description: > + Polymorphic response for metrics queries. The concrete type is determined by the + metricType discriminator field, which echoes the requested metricType query parameter. + oneOf: + - $ref: '#/components/schemas/ListScanMetricsResponse' + - $ref: '#/components/schemas/ListCommitMetricsResponse' + discriminator: + propertyName: metricType + mapping: + scan: '#/components/schemas/ListScanMetricsResponse' + commit: '#/components/schemas/ListCommitMetricsResponse' + + ListScanMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + nullable: true + description: > + Opaque cursor for fetching the next page. Null or absent when no further pages exist. + metricType: + type: string + enum: [scan] + description: Discriminator — always "scan" for this response type + reports: + type: array + items: + $ref: '#/components/schemas/ScanMetricsReport' + + ListCommitMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + nullable: true + description: > + Opaque cursor for fetching the next page. Null or absent when no further pages exist. + metricType: + type: string + enum: [commit] + description: Discriminator — always "commit" for this response type + reports: + type: array + items: + $ref: '#/components/schemas/CommitMetricsReport' + + MetricsActor: + type: object + description: Identity of the principal who triggered the operation + properties: + principalName: + type: string + nullable: true + + MetricsRequest: + type: object + description: Request context for correlation with logs and traces + properties: + requestId: + type: string + nullable: true + otelTraceId: + type: string + nullable: true + description: OpenTelemetry trace ID + otelSpanId: + type: string + nullable: true + description: OpenTelemetry span ID + + ScanMetricsObject: + type: object + description: Resource context for the scanned table operation + required: + - table + properties: + table: + $ref: '#/components/schemas/TableRef' + snapshotId: + type: integer + format: int64 + nullable: true + + CommitMetricsObject: + type: object + description: Resource context for the committed table operation + required: + - table + - snapshotId + properties: + table: + $ref: '#/components/schemas/TableRef' + snapshotId: + type: integer + format: int64 + + ScanPayloadData: + type: object + description: Iceberg scan metrics data + properties: + schemaId: + type: integer + nullable: true + filterExpression: + type: string + nullable: true + projectedFieldIds: + type: array + nullable: true + items: + type: integer + description: Projected field IDs + projectedFieldNames: + type: array + nullable: true + items: + type: string + description: Projected field names + resultDataFiles: + type: integer + format: int64 + resultDeleteFiles: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDataManifests: + type: integer + format: int64 + totalDeleteManifests: + type: integer + format: int64 + scannedDataManifests: + type: integer + format: int64 + scannedDeleteManifests: + type: integer + format: int64 + skippedDataManifests: + type: integer + format: int64 + skippedDeleteManifests: + type: integer + format: int64 + skippedDataFiles: + type: integer + format: int64 + skippedDeleteFiles: + type: integer + format: int64 + totalPlanningDurationMs: + type: integer + format: int64 + equalityDeleteFiles: + type: integer + format: int64 + positionalDeleteFiles: + type: integer + format: int64 + indexedDeleteFiles: + type: integer + format: int64 + totalDeleteFileSizeBytes: + type: integer + format: int64 + + ScanPayload: + type: object + required: + - type + - version + - data + properties: + type: + type: string + enum: [iceberg.metrics.scan] + version: + type: integer + enum: [1] + data: + $ref: '#/components/schemas/ScanPayloadData' + + CommitPayloadData: + type: object + description: Iceberg commit metrics data + properties: + sequenceNumber: + type: integer + format: int64 + nullable: true + operation: + type: string + description: Commit operation (append, overwrite, delete, replace) + addedDataFiles: + type: integer + format: int64 + removedDataFiles: + type: integer + format: int64 + totalDataFiles: + type: integer + format: int64 + addedDeleteFiles: + type: integer + format: int64 + removedDeleteFiles: + type: integer + format: int64 + totalDeleteFiles: + type: integer + format: int64 + addedEqualityDeleteFiles: + type: integer + format: int64 + removedEqualityDeleteFiles: + type: integer + format: int64 + addedPositionalDeleteFiles: + type: integer + format: int64 + removedPositionalDeleteFiles: + type: integer + format: int64 + addedRecords: + type: integer + format: int64 + removedRecords: + type: integer + format: int64 + totalRecords: + type: integer + format: int64 + addedFileSizeBytes: + type: integer + format: int64 + removedFileSizeBytes: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDurationMs: + type: integer + format: int64 + nullable: true + attempts: + type: integer + + CommitPayload: + type: object + required: + - type + - version + - data + properties: + type: + type: string + enum: [iceberg.metrics.commit] + version: + type: integer + enum: [1] + data: + $ref: '#/components/schemas/CommitPayloadData' + + ScanMetricsReport: + type: object + description: Stable envelope for a persisted Iceberg scan metrics report + required: + - id + - timestampMs + - object + - payload + properties: + id: + type: string + description: Unique identifier for this report + timestampMs: + type: integer + format: int64 + description: Server-side timestamp when the report was received (Unix epoch milliseconds) + actor: + $ref: '#/components/schemas/MetricsActor' + nullable: true + request: + $ref: '#/components/schemas/MetricsRequest' + nullable: true + object: + $ref: '#/components/schemas/ScanMetricsObject' + payload: + $ref: '#/components/schemas/ScanPayload' + + CommitMetricsReport: + type: object + description: Stable envelope for a persisted Iceberg commit metrics report + required: + - id + - timestampMs + - object + - payload + properties: + id: + type: string + description: Unique identifier for this report + timestampMs: + type: integer + format: int64 + description: Server-side timestamp when the report was received (Unix epoch milliseconds) + actor: + $ref: '#/components/schemas/MetricsActor' + nullable: true + request: + $ref: '#/components/schemas/MetricsRequest' + nullable: true + object: + $ref: '#/components/schemas/CommitMetricsObject' + payload: + $ref: '#/components/schemas/CommitPayload' + + ErrorResponse: + type: object + properties: + message: + type: string + type: + type: string + code: + type: integer