Logging errors in a structured format PoC, log-when-encountered approach

[vladem]

Description of change

With this approach of error reporting we closely follow the existing logging approach, with which we log S3 request errors as soon as we encounter them in the mountpoint-s3-client.

The perk of this approach is the simplicity of implementation as it does not require passing error type encoded by s3_crt_client::S3RequestError enum to the fs layer.

For instance, information about an error type S3RequestError::Forbidden is inaccessible from the
SuperblockInner::remote_lookup as the generic implementation of SuperblockInner only knows that the value returned by head_object is ClientError (as a part of ObjectClientResult<HeadObjectResult, HeadObjectError, Self::ClientError>) which is type ClientError: std::error::Error + Send + Sync + 'static. Next in the S3Filesystem::open example, ClientError is converted to InodeError::ClientError(#[source] anyhow::Error). Enum InodeError will also require changes to propagate information with the required granularity to the S3FileSystem::open.

The critical downside of this approach is that using it we will report errors even if they were retried by the fs layer and didn't make it to the user application.

Sample output (note transient ClientError(IncorrectRegion(..)) being reported:

{"timestamp":"2024-05-03T20:13:13.043530Z","level":"ERROR","fields":{"event_type":"errors.client.internal","message":"ClientError(IncorrectRegion(\"eu-west-1\"))"},"target":"mountpoint_s3_client::s3_crt_client","span":{"bucket":"my_bucket","continued":false,"delimiter":"","id":0,"max_keys":"0","prefix":"","name":"list_objects"},"spans":[{"bucket":"my_bucket","continued":false,"delimiter":"","id":0,"max_keys":"0","prefix":"","name":"list_objects"}]}
{"timestamp":"2024-05-03T20:13:17.207594Z","level":"ERROR","fields":{"event_type":"errors.client.forbidden","message":"ClientError(Forbidden(\"<no message>\"))"},"target":"mountpoint_s3_client::s3_crt_client","span":{"bucket":"my_bucket","id":1,"key":"1.txt","name":"head_object"},"spans":[{"ino":1,"name":"\"1.txt\"","req":6,"name":"lookup"},{"bucket":"my_bucket","id":1,"key":"1.txt","name":"head_object"}]}
{"timestamp":"2024-05-03T20:13:17.208709Z","level":"ERROR","fields":{"event_type":"errors.client.internal","message":"ClientError(CrtError(Error(14347, \"aws-c-s3: AWS_ERROR_S3_CANCELED, Request successfully cancelled\")))"},"target":"mountpoint_s3_client::s3_crt_client","span":{"bucket":"my_bucket","continued":false,"delimiter":"/","id":2,"max_keys":"1","prefix":"1.txt/","name":"list_objects"},"spans":[{"ino":1,"name":"\"1.txt\"","req":6,"name":"lookup"},{"bucket":"my_bucket","continued":false,"delimiter":"/","id":2,"max_keys":"1","prefix":"1.txt/","name":"list_objects"}]}

PoC with reporting Forbidden from the S3Filesystem follows in the next alternative PR.

Relevant issues: #721

Does this change impact existing behavior?

The change in this form does not impact any of the existing behaviour.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

[vladem]

Closing this in favour of: #894