How can I reduce Internet traffic overhead for syncs?

[rossmcm]

I have a both-ways sync job that runs twice a day on both a home and a work PC to keep them both synchronized. The file set is around 100,000 files. I've been watching the traffic and in order to read the file set from S3 each time generates about 30 Mb incoming traffic. This is many times the traffic generated by the actual sync of any files that have changed.

If there any way I can reduce this overhead? I thought about filtering by file age but I figure SFFS will still need to read the entire file set from S3 each time.

Maybe you could use compressed HTTP to get the file listing from S3?

[superflexible]

I am not aware of any way to reduce this overhead for two-way syncs. If you were only uploading, the program could remember the data on S3 in a database.

If you were using some web space with FTP access, the overhead would be much lower and the file listing could be retrieved much faster if the server supports the recursive listing via LIST -alR.

[rossmcm]

OK how about this for an idea for syncing 2 computers with SFFS without the overhead of reading a large chunk of data from the web each time . I'm in the process of trying it, but it looks pretty encouraging.

Create a folder on the desktop of computer 1 called "Sync"
Create a profile for SFFS to sync between your folders of interest and Desktop\Sync.

Create a folder on the desktop of computer 2 called "Sync"
Create a profile for SFFS to sync between your folders of interest and Desktop\Sync.

Get a free account from SugarSync (5Gb)
Sync the 2 folders using SugarSync.

Advantages:

Apart from the initial syncs, web traffic is only for the files that are changed.
Deletes, renames etc should be propagated.
You get to use SFFS exclusion/compression/encryption etc (this is a big plus because there is no way to specify your own encryption keys in SugarSync, and you can't have file exclusions)
Syncs are faster because SFFS is reading the "remote" file set from a local folder, and copying to and from that local folder. Once SFFS finishes, SugarSync takes over and syncs its folder to the web (and therefore the distant computer).

Disadvantages:

Need additional storage on both machines for the "Sync" folders

The same scheme should work with DropBox (but you only get 2Gb in a free account).

Any comments?

[superflexible]

Sounds like a good idea!

[rossmcm]

Tried SugarSync and although 5 Mb free was tempting, it just seems too klunky, and I seemed to get into a situation where it says it's doing nothing but I can see furious internet traffic heading outwards.

Dropbox just works. It's biggest drawback is the lack of file exclusions and ability to (easily) sync arbitrary folders, but SFFS largely gets around those drawbacks.

But will I open my wallet when the trial period runs out? .....