Polaris fails to create a Table because of an STS policy size limit

[pingtimeout]

Describe the bug

While running benchmarks with much longer namespace names, I got the following error while creating a table:

Failed to get subscoped credentials: Packed policy consumes 118% of allotted space, please use smaller policy.

The path that caused this issue is composed by 14 nested namespaces, each with a 32 characters name. The problem is located in

polaris/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java

Line 175 in 31c7784

private IamPolicy policyString(

. It results in the following STS policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-very-long-s3-bucket/catalog/f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-very-long-s3-bucket",
      "Condition": {
        "StringLike": {
          "s3:prefix": "catalog/f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351/*"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetBucketLocation",
      "Resource": "arn:aws:s3:::my-very-long-s3-bucket"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::my-very-long-s3-bucket/catalog/f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351/*"
    }
  ]
}

I don't think we can eliminate the STS limitation at all. But I think Polaris could be improved to better deal with this error. For example:

• The returned HTTP code (400) is correct (Bad Request) but the associated error message should be clearer for the user. The current one may not be understandable for everyone.
• During namespace creation, no check is performed on the total length of the path. It might be worth adding so that the failure happens earlier.
• Parts of the STS policy could be merged together (AFAICT): the s3:PutObject and s3:DeleteObject statements could be merged with the s3:GetObject and s3:GetObjectVersion statements. This would allow Polaris to support longer paths, while not removing the maximum path limitation entirely though.

To Reproduce

• Create a deeply nested namespace structure with long length (e.g. f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351)
• Create a table in the last namespace

Actual Behavior

No response

Expected Behavior

No response

Additional context

No response

System information

No response