Building File List takes FOREVER

[chadmarkley]

My source server has a volume I am syncing that is about 1.5TB with close to 350K files. I sync every 4 hours so there aren't many changes to sync over, but with each sync the "building file list" takes FOREVER. I see some options under "Special" that I am curious about. Is there anything I can do to "cache" the file listing to speed the process up?

Great product BTW!

[superflexible]

Yes, there are 3-4 ways to speed this up. The best summary of them is on our upcoming new web site:

http://www.syncovery.com/documentation/faq/building-the-file-list-takes-too-long-how-to-make-it-faster/

The information applies to Superflexible version 5 in the same way. Just that the remote service is called "ExtremeSync Remote Service" and not "Syncovery Remote Service".

The old documentation is in our FAQ: "Building the file list takes too long" on
http://www.superflexible.com/docs.htm

[chadmarkley]

Yes, there are 3-4 ways to speed this up. The best summary of them is on our upcoming new web site:

http://www.syncovery.com/documentation/faq/building-the-file-list-takes-too-long-how-to-make-it-faster/

The information applies to Superflexible version 5 in the same way. Just that the remote service is called "ExtremeSync Remote Service" and not "Syncovery Remote Service".

The old documentation is in our FAQ: "Building the file list takes too long" on
http://www.superflexible.com/docs.htm

WOW, the post is even named perfectly! So I read through the instructions on using the Remote Service. I installed it on the remote/left (source) server and I changed the option to use the Remote Service under "Special" tab sheet. I think I did this correctly but when I run the sync I get a message that states "waiting for remote file listing" and it looks like its just hanging there. The seconds tick off on the message pane but nothing seems to happen. WHen I check the source server, there is no additinal CPU utilization or network traffic. It's as though the process isn't doing anything.

I am wondering if I didn't set up the remote service correctly on my source server? I have it running as a service. That is about all I can see to do besides the MD5 and ZIP option tabs.

Thx again for the quick replies

[chadmarkley]

I think I figured it out. On the "Configure Checksummer and remote listener" tab I need to put in the folders I am wanting to use the remote service with, right? Seems like its doing something now.

[superflexible]

Yes that is correct. Is it working now?

[gkr]

Hello,

I've been using SFFS for a few months now, mostly doing full 2-way syncs, but for moves and mirrors as well. It generally works very well.

Presently, I'm trying to 2-way-sync about 12,000 files from from WindowsXP(left) to a TonidoPlug2(linux, SFTP)(right). Regardless of whether the TonidoPlug is on the same network or remote, when it does the full sync, building the file list, it goes fast(a few seconds) for about 1500 files & folders, and then incredibly slow (hours) for the rest.

I have read done what it says on http://www.syncovery.com/documentation/faq/building-the-file-list-takes-too-long-how-to-make-it-faster/

I just did a Skip Rest, and it came back with:
Access to these paths was not possible:
sftp://192.168.1.101/media/disk2part1/My Documents/Center Stuff/EPC-OtherPublicity/Leaflets/ZIP Disk Backup/Denverbrochure: An unexpected network error occurred. Connection timed out (error code is 10060)
sftp://192.168.1.101/media/disk2part1/My Documents/Center Stuff/EPC-Retreats/pingreeretreat05/Letters: An unexpected network error occurred. Connection timed out (error code is 10060)

Any suggestions?

Thank you for a nice program.

[superflexible]

That's normal. When you click "Skip Rest", the file list is not built completely and error messages can happen. You can ignore them because these are simply folders which are skipped along with the rest.

[gkr]

OK, thank you for clarifying that the error messages are just associated with Skip Rest.

Would you have any suggestions for improving the main issue, which is that it often can't build the file list during an 8 hour work day even on the same lan? It seems really odd that it's almost always around 1486 or something that it starts going really slow. I've tried deleting the particular folder it's cataloging when it starts to go slow, and that changes things slightly, perhaps to 1732 or something. Would it be some issue with SFTP? Or connectivity with the TonidoPlug? Thanks again.

[superflexible]

I don't know. It must be a problem with the Linux box. However you can try the "Recursive" checkmark on the second tab sheet of the FTP/Internet dialog. It should work if you have SSH shell access to the Linux box.

You can also generate a detailed log file with timings. On the Program Settings dialog, choose "With Timing Info" as well as "Detailed", and possibly "File List Building Details".

[gkr]

I don't know why, but just setting all the log settings you describe seems to have fixed that problem, so thank you very much!