S3 Move job keeping folder structure

[Luc23]

I have used your program for years and am now attempting to use a new install to move folders from S3 to a local folder on my server. I've got the move working fine, but it keeps the entire folder structure without the files on the source.

This is not the case on a previous install I have running. Besides the program version, the only difference I see in the two profiles is that the one that works as expected uses FTP while this is S3. Checking "Remove Folders That Were Emptied" does nothing.

Any help on this? I need it to actually MOVE everything, including the folder structure.

[Luc23]

I just recreated the same profile on the old version using S3 and it also did not work as I would expect. I realized that the current profile that does clean up the folder structure on the source is actually not using FTP. Its just connecting to another share on a different server over my network.

Any thoughts on how to get an S3 job to clean up empty folders?

[Luc23]

I am still hitting my head against the wall trying to resolve this weird issue. I'm pulling from an S3 bucket to w Windows share. The pull to destination is working just fine, but the folders persist in the source.

Running the same profile but using the source as a Windows share instead of S3 DOES remove the folder structure on the source.

This seems to be S3 craziness and I don't know how to fix it. Any help? Please!

[Luc23]

Yes, I am talking to myself, but for anyone that might come across this issue... I think the issue is that S3 essentially doesn't have "folders". It has buckets and objects. Ehh, I'll just paste the snippet from the AWS guy in this post:
https://github.com/aws/aws-cli/issues/912

"kyleknap commented on Sep 12, 2014
This behavior is known. The reason why the sync command behaves this way is that s3 does not physically use directories. There are only buckets and objects. Objects have prefixes that act like directories, but s3 does not designate a specific physical object to be a directory.

Therefore, when the syncing occurs, only files are transferred to s3 because s3 does not have physical directories. So when you try to sync up empty directories, nothing is uploaded because there are no files in them. Once you put items in the directory, then the file (with the prefix representing the directory) will be uploaded."

So although it helps understand why Syncovery is behaving the way it is, it gets me no closer to figuring out how to get it to clean up empty directory structures left behind in S3....

[superflexible]

Hello,
sorry for the late reply. Yes S3 does not have native folders, but they do emulate them somehow. Syncovery only sees folders on S3 if there are files in them. So Syncovery does not support empty folders on S3.

Which tool do you use to see that folders remain on S3?

How were the folders created in the first place (with what tool)?

[Luc23]

We resolved this in our code. We modified the Lambda job that copies from one S3 bucket to another. For anyone that happens across this (or for the Syncovery devs that might be able to code this into the product), here is what our AWS support guy has said:

I know that you have already resolved this case, but I wanted to provide you with an update as the CLI team has gotten back to my inquiry. The short and sweet of it is that yes they are aware of this and are tracking it via our GitHub page here: https://github.com/aws/aws-cli/issues/912

There is no ETA on when an update may come down stream that assists with this, but they did offer up an alternative suggestion. To get around this they recommend not using the mv command, but instead a cp --recursive and then rm --recursive if you want to remove all of the placeholders. The rm --recursive command will always remove the placeholders.

I hope that this helps you with your en devours in AWS. Should you have any further questions feel free to re-open this case.

Best regards,

Matthew B.
Amazon Web Services

[superflexible]

Many thanks for the info!

[aligntech]

I know this is an old post, but I'm experiencing the same exact behavior still as of this morning.

Was there ever any type of Syncovery built-in resolution to the situation where files\"folders" are being moved from S3 (or in my case S3-compatible - Wasabi) storage to another location? I'm constantly having to go back in and manually clean out the skeleton folder structure after I complete a scheduled move from Wasabi to Local (Windows) with Syncovery and it seems like there should be a way for this to self-clean in the profile.

Windows Server 2008 R2 SP1 64-bit
Syncovery 8.18e 64-bit

[superflexible]

Hello,
again sorry for the late reply. Syncovery does not support empty folders on S3 or Wasabi, hence it does not see them and cannot remove them.

I hope I can add it soon.