Can anyone explain SFFS Internet Traffic

[rossmcm]

SFFS.png (17.74 KiB) Viewed 1125 times

If I look at the Internet traffic for a first-time sync to S3 I am running I notice 3 things:

there is just as much incoming traffic as outgoing (maybe SFFS is doing a read-after-write file verification - I don't remember anything in the options for that).

the traffic varies a bit - most times it is around 6Mb/hour, sometimes it spikes (I have seen rates of 50Mb/hour).

SFFS seems quite slow compared to other file uploads (e.g. FTP)

Can anyone comment?

[superflexible]

I think the varying traffic is caused by varying file sizes. Apparently there are a lot of small files. To use more traffic, you need to copy several files in parallel. The number is specified on the Files tab sheet.

The verification settings are on the tab sheet Comparison->More, so you can take a look there. You don't need any of them for Amazon S3 because all uploads are always checked by MD5 checksum.

If you have small files, the HTTP protocol may produce some significant incoming traffic.

The Amazon S3 file listings are quite big. So if the program retrieves the complete file listing often, that can be a lot of traffic. To avoid it, see below. But this should not be relevant if the program has been running just the one job all day.

Obviously your Internet connection as well as Amazon server availability will cause varying transfer speeds too.

SFFS seems quite slow compared to other file uploads (e.g. FTP).

SFFS can do FTP too, so I really do not know what you mean by this sentence. It's like saying your car is much slower than the highway.

If you are only uploading and repeating the same job in the future, you can save time by using the "Cache Destination File List" feature on the Special tab sheet.

[rossmcm]

Hey, I am REALLY impressed by the support response. Well done.

Your explanations make sense. My comment about FTP was based on uploads of big files, which will be faster.

I'll look at the options.

[rossmcm]

I set the number of concurrent file transfers to 10 (it was 1) and the speed increased markedly - like ten times faster:

SFFS.png (25.95 KiB) Viewed 1107 times

Between 0000 and 1000 it was one thread
Between 1000 and 1500 it was 10 threads
Between 1500 and 2000 it was 5 threads

so it looks as though 5 threads is sufficient to get maximum throughput on my (slow -1 Mbit max) connection.

[superdb]

Can you tell me how to enable "concurrent file transfers"? my speeds are really slow using sftp and i am assuming because its only doing a single file transfer and not multiple. i dont see where in the options i change this setting. please help!

[superflexible]

Hello,

when you edit a job in Advanced Mode, the setting is on the "Files" category in the lower half of the dialog. At the bottom of the Files tab sheet, you can specify the number of files to copy in parallel.