Speed up backups

[solspin]

How do I get past the "Building List ... Entries"? Every time I run a local backup it takes FOREVER and all I see is this status message. Why does it take soooooo long to build a list? More to the point - why does it take so long to re-run a backup that has 2MB of changes?

[superflexible]

Normally it is very fast. Something must be wrong. There are some settings that can make it slower, such as Binary Comparison of Existing Files, and comparing the security settings for all existing files. Maybe you should create a new profile from scratch and make sure you don't select any settings unless you really need them.

You can choose the same source and destination folders in the new profile and it will not re-copy the files you have already backed up.

If problems remain, maybe you can give some more info about how many folders and files you have and how long this step really takes. Some people think 10 second is a long time, others think 1 hour is long.

[wizardict]

I'm having the same problem. Building the file list is not too bad, it takes about 30 minutes to build a list of 110GB of files. I can solve this by caching the destination as it never changes, but the actual backup takes forever.

I'm using SFTP and backing up 1.5GB is estimating at over 1 day! (I cancelled it after 30 minutes with 1% backed up). The same backup using rsync over SSH takes just over 1 hour so my file transfer speeds are ok.

[wizardict]

Just to add, I don't have Binary comparison selected nor do I have Process Security and Shares selected.

[superflexible]

Maybe you need to set it to copy several files in parallel. The number of files can be specified on the Files tab sheet in the profile. Between 3-5 seems reasonable. Maybe up to 10 if you want to max it out. But that may not be faster than 5.

In general, performance may vary depending on how well client and server go together. Please make sure you have the latest version 5.46.

You can also configure the Logs to be "With Timing Info" on the Program Settings Dialog (F8) and send a log to me by email.

[wizardict]

Ok setting to 5 improves things a bit, I'll try setting it higher and see where the improvement stops. It also seems to be backing up the complete changed file, which I guess is why it's so much slower than rsync, does "use partial file updating" not work without the remote service installed on the server?

I also have another problem. If I select destination caching to improve the file list speed it tells me I need to backup all 107GB! If I don't then it just backs up the 1GB or so that's changed. It seems to be checking the server to build the cache so I don't understand why this would be?

[superflexible]

does "use partial file updating" not work without the remote service installed on the server?

It works, but first it needs to upload the complete file and build the checksums. Then the next time, it can do a partial update. With the Remote Service on the other end, it can do the partial update even the first time.

If I select destination caching to improve the file list speed it tells me I need to backup all 107GB!

Apparently the cache database is incomplete due to canceled runs. You should delete the database and make sure you let it run thru and not cancel it, so that the database can be filled. The database folder is shown on the Program Settings dialog (tab sheet Misc). The database file name is also shown in the log file.

When you delete the database, the program will re-build it by scanning all folders again.

[wizardict]

The backup was previously cancelled so that fits.

Just to clarify

It works, but first it needs to upload the complete file and build the checksums. Then the next time, it can do a partial update. With the Remote Service on the other end, it can do the partial update even the first time.

The files were previously seed loaded as many of them are too big to upload in a reasonable time. Are you saying that even though the file is on the server it still needs to do a full backup of a file first before it can do partial updating?

[superflexible]

Are you saying that even though the file is on the server it still needs to do a full backup of a file first before it can do partial updating?

It depends. If the file has not changed since the seeding, then the program recognizes that it's identical to the local copy and just builds the MD5 checksums from the local copy and stores them in its database.

But if the local file has changed since the seeding, then the program has to re-upload it completely, and at the same time it builds the checksums to enable future partial updates.

To avoid the dependency on MD5 checksums stored in the database, you can install the Remote Service on the destination machine, if it is a Windows or Macintosh system.

On the other hand, if it is a Linux SSH server, you can try the recursive file listing option, which will speed up building the file listing dramatically so you may not need to use the destination cache feature. The recursive option is on the second tab sheet of the Internet/FTP dialog. This only works with Unix type SSH servers because it uses the SSH shell to run a ls -alR command line.

[wizardict]

Ok that makes sense, thanks for all your help.

It's actually a QNAP NAS box at the other side, which I believe is essential a linux kernel, so the Remote Service isn't an option but I'll give the recursive option a try.

Setting the parallel files to 10 has improved things hugely! The original estimate of over 1 day is now down to around 2 hours. This is still a little slower than I hoped for but it looks like building the checksum files is taking a lot of time. From what you say if it only needs to do this first time I think we might just have a speed to rival rsync after a bit of tinkering