-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS S3 sync operations do not work with S3 directory buckets (S3 Express One Zone) #8470
Comments
Hi @jeffgardnerdev, thanks for reaching out. Could you provide debug logs of this behavior? You can get debug logs by adding |
Here are the two debug outputs, one without the |
@jeffgardnerdev, thanks for your patience. I was able to reproduce this behavior, and your theory that this is related to If other people are experiencing this issue, providing details and any impact in this issue would be appreciated. Ticket # for internal use : P114641353 |
@RyanFitzSimmonsAK Could you share some more information about the decision to disable the sync command for directory buckets entirely? This is useful functionality that would be nice to keep, provided the ordering issue could be fixed. |
Hello, Thank you for reporting this issue. We have released AWS CLI v1.32.25 and v2.15.13 and strongly recommend that you upgrade to address this issue. If you are unable to upgrade, do not run the |
@jeffgardnerdev I did see your question about further information, which I will post here if I am able. |
Here are some additional technical details on this issue if you are interested. The reason we’ve removed the In the versions of the CLI I noted above, v1.32.25 and v2.15.13 and later, we removed 1: Specifically, "Sorting order of returned objects" on the linked documentation page. |
This issue is now closed. Comments on closed issues are hard for our team to see. |
Isn't it a bit weird to close this issue, @kellertk? I would assume that people would want |
Describe the bug
If you run
aws s3 sync
with the source from a standard bucket and the destination from a directory bucket, the comparison does not work properly. Some files that do exist in the source are not recognized and some files that do exist in the destination are also not recognized.Expected Behavior
Only the source objects that changed since the last sync are copied to the destination.
Current Behavior
Some objects that did not change since the last sync are copied to the destination. Those objects are also deleted from the destination before copying from source if the
--delete
flag is supplied, indicating that it actually does see the object at the destination, but doesn't consider it the same object as what is in the source.Reproduction Steps
Test scenario: a source prefix in a standard bucket is successfully synced to the same prefix in a directory bucket. The prefix has 5 objects. A subsequent sync performed immediately after results in copying one of those objects again, even though nothing has changed. The debug messages contain a message that "file does not exist at destination" even though it does exist at the destination. Interestingly, if the
--delete
flag is supplied, that file is deleted from the destination and then recopied.Possible Solution
I believe this bug stems from an inconsistency in the response order in the
list-objects-v2
API. For whatever reason the file that was considered missing in the test scenario is listed last in the response for the directory bucket, while it is listed first in the response for the standard bucket.Additional Information/Context
No response
CLI version used
2.14.4
Environment details (OS name and version, etc.)
macOS Ventura 13.4.1
The text was updated successfully, but these errors were encountered: