Creating checksum file

[glercara]

I'm trying the software to see if it meets our needs. I'm having an issue where partial file updating is constantly asking the remote service to generate the checksum file. This happens after repeated syncs from left->right. The source server is in one location and the target is in another. The files are very large (> 120GB). But the partial file updating, when it works, significantly reduces network traffic. My question is: Why is the program not storing the checksum file in the database? It seems to me that there is no reason to re-generate the file on every run; especially if the previous run successfully creates it. The creation of the checksum file takes a very long time and it has to do it 2 times since I have 2 very large files.
1) I have verified that the database exists
2) I'm using Copy Left to right
3) Sync mode is Exact Mirror
4) under Special - Cache destination File List is selected, Use Partial File Updatding is selected, Right side uses Remote service is selected, No Full copying if service unavailable is selected, Right side listing uses Remote Service is selected.
Also, the right-side never changes.

I have been able to get the partial file updating to work correctly only when I run in Attended mode. But, when I schedule a job, it insists on asking for the checksum file to be created.

I'm obviously doing something wrong.

Thanks for your help!

[superflexible]

Please uncheck "Right side uses Remote service". Then the checksums are stored in the database.

The only drawback is that if the database does not contain the latest checksums yet, then the entire file needs to be copied. There is no logic in the program yet that would use the remote service only in such cases. It's either always or never.

To make sure that the database contains the latest checksums, you would ideally run the profile once when source and destination file are identical. Then the program sees that they are identical (based on file size and timestamp) and generates the checksums locally.

There should not be any difference whether you run it in attended mode or not.

[glercara]

The job failed because it took too long to generate the checksums. I'm running it now in interactive mode and it's actually working since, I'm guessing, the checksum files were finally completed on the remote side. If I make the changes you are suggesting, what is the point of using the remote service at all?

Whether the checksums are created locally or remotely, wouldn't it make sense for the program to store them into the database?

Once the interactive job completes, I'm going to try your suggestion and post back. Thanks for your help.

[superflexible]

The Remote Service was originally added for copying in the opposite direction, i.e. from remote to local.

Yes combining the remote service feature with the database is a good idea. I will consider it.

[glercara]

To make sure that the database contains the latest checksums, you would ideally run the profile once when source and destination file are identical. Then the program sees that they are identical (based on file size and timestamp) and generates the checksums locally.
.

I tried running the profile while the source and destination files are identical. but, it quickly said "No files copied or folders created". I hope I don't have to send every file to the remote location as it will take days to complete.

[superflexible]

Sounds good. If you unchecked the Remote Service for partial file updating prior to letting it run thru, then if it does nothing, it means that the checksums are already in the database.

Try it with some not so big file and you will see if the partial updating works. The precise statistics on the transfer amounts saved is shown at the end of the log file.

[glercara]

Ok, I'm happy to report that the job completed successfully in about 3 hours which is really not bad at all.
I guess you have another sale

Here's the log:

Super Flexible File Synchronizer v5.62 Report


Remaining days for evaluation: 23

Running on : CHROM-BAK-SRVR.chromusa.com


Profile : Veeam Backups

Start time: 1/17/2012 7:30:07 PM

Completed : 1/17/2012 10:31:50 PM


Short Results: 10 copied (277.3GB), 2 deleted


Detailed Results
-----------------------

Copied To Right Side: 10 (277.3GB)
Deleted On Right Side: 2 (1.4GB)
Files updated on right side : 6

Transfer amount saved due to partial file updating : 269.8GB <------
Remaining actual transfer amount for eligible files: 4.0GB <-------

Average copying time per file : 1086.0 s
Copying time for fastest file : 0.3 s
Copying time for slowest file : 2662.4 s

Average transfer rate : 26778.6 kB/sec
Transfer rate for slowest file : 54.9 kB/sec
Transfer rate for fastest file : 74335.4 kB/sec
For the fastest/slowest rates, only files at least 1 kilobyte in size were counted.



Actions Performed
-----------------------
DELETE R \\chrom-offsite\e$\backup\Veeam\Backup Chrom_VCx64\Backup Chrom_VCx642012-01-04T204639.vrb
DELETE R \\chrom-offsite\e$\backup\Veeam\Replicas\vm-3588\2012-01-09T220047.vbk
COPY L->R E:\backup\Veeam\Backup Chrom cluster\Backup Chrom cluster.vbk (135.6GB)
COPY L->R E:\backup\Veeam\Backup Chrom cluster\Backup Chrom cluster.vbm (206.2kB)
COPY L->R E:\backup\Veeam\Backup Chrom cluster\Backup Chrom cluster2012-01-16T180050.vrb (1.2GB)
COPY L->R E:\backup\Veeam\Backup Chrom_DBS\Backup Chrom_DBS.vbk (122.0GB)
COPY L->R E:\backup\Veeam\Backup Chrom_DBS\Backup Chrom_DBS.vbm (56.2kB)
COPY L->R E:\backup\Veeam\Backup Chrom_DBS\Backup Chrom_DBS2012-01-16T190035.vrb (1.2GB)
COPY L->R E:\backup\Veeam\Backup Chrom_VCx64\Backup Chrom_VCx64.vbk (16.2GB)
COPY L->R E:\backup\Veeam\Backup Chrom_VCx64\Backup Chrom_VCx64.vbm (41.6kB)
COPY L->R E:\backup\Veeam\Backup Chrom_VCx64\Backup Chrom_VCx642012-01-16T160034.vrb (1.1GB)
COPY L->R E:\backup\Veeam\Replicas\vm-3588\2012-01-17T220048.vbk (8.0MB)


Log File Attached.


Profile Description
-----------------------
Name: Veeam Backups
L -> R
Run Daily At 7:30 PM (next: 1/18/2012)

LEFT SIDE : E:\backup\Veeam
RIGHT SIDE: \\chrom-offsite\e$\backup\Veeam




The profile is executed by the scheduler in a separate thread.