Make it easier to use AWS classes as a custom credentials provider #22007
Conversation
The `trino.s3.credentials-provider` configuration property allows specifying a class used as the credentials provider, but it requires the class to have a constructor that accepts a URI and a Configuration object. The AWS SDK classes implementing `AWSCredentialsProvider` have constructors without any arguments, so to use them, a wrapper class had to be created, and an extra JAR had to be added to the deployment. Trying to instantiate the custom credentials provider class without any arguments makes it easier to pin to exactly one provider without using the default providers chain.
|
@electrum should we allow pinning to a specific provider in the new FS and Glue implementations? If yes, should we add a similar config option for a class name? Maybe also in here: https://github.com/trinodb/trino/blob/master/plugin/trino-exchange-filesystem/src/main/java/io/trino/plugin/exchange/filesystem/s3/S3FileSystemExchangeStorage.java |
| interface and provide a two-argument constructor that takes a | ||
| `java.net.URI` and a Hadoop `org.apache.hadoop.conf.Configuration` | ||
| as arguments. A custom credentials provider can be used to provide | ||
| temporary credentials from STS (using `STSSessionCredentialsProvider`), |
There was a problem hiding this comment.
trino-hdfs is legacy & to be removed rather sooner than later. if this was broken and didn't work, maybe no need to fix it? Just remove incorrect stuff from the docs?
There was a problem hiding this comment.
It worked, just was hard to use. This feature is missing in new native fs library, so I'd also like to discuss if we want to add it there. Being able to choose a single credentials provider instead of relying on the (default) chain solves some issues.
How soon are we going to remove this, weeks or months?
There was a problem hiding this comment.
I'd hope it's sooner than later (so weeks rather than months).
I am not against having this feature (done correctly) in the new native fs library, if it solves problems. Would it be possible to have an issue describing those problems?
There was a problem hiding this comment.
See the issue linked in the PR description
There was a problem hiding this comment.
Thanks. The linked issue is sufficient to me to justify adding the configuration to the new s3 filesystem. @nineinchnick do you plan on working on this?
Description
The
trino.s3.credentials-providerconfiguration property allows specifying a class used as the credentials provider, but it requires the class to have a constructor that accepts a URI and a Configuration object. The AWS SDK classes implementingAWSCredentialsProviderhave constructors without any arguments, so to use them, a wrapper class had to be created, and an extra JAR had to be added to the deployment.Trying to instantiate the custom credentials provider class without any arguments makes it easier to pin to exactly one provider without using the default providers chain.
This actually makes the docs correct, because it already mentioned using providers from the AWS SDK, which was not possible without a wrapper class.
This at least makes the workaround for #15267 easier, I'm not sure if it will allow closing that issue.
Additional context and related issues
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: