How to synchronize log files which size turns to be larger?

[Hayframe]

Hello,

I have a scenario where I need to backup IRC logs stored inside a Linux server to a Windows desktop (via SFTP).

I'm hitting an issue: Because the IRC logs are being modified between the time the manifest is calculated and the time the files are downloaded, the file sizes do not match after the transfer and therefore the downloaded files are discarded.

10/01/2016 20:23:59 Copied file too big on dest (317543342>317538459) - deleting:
U:\Sync-Duplicate\Servers\MYSERVER Root\home\username\irclogs\stream\#infomarker.log.d20160110-u184543.s317538459.$syncovery
10/01/2016 20:24:02 Transfer error, dest size=317543342, src size=317538459
U:\Sync-Duplicate\Servers\MYSERVER Root\home\username\irclogs\stream\#infomarker.log.d20160110-u184543.s317538459.$syncovery
Exception Transfer error, dest size=317543342, src size=317538459 at 0000000001A71B8A.
Filename: sftp://MYSERVER_REDACTED.xyz/home/usern ... marker.log

How can I tell Syncovery to synchronize files in such a way that a discrepancy in the file size between the source and the destination folder after transfer is marked as a valid file anyways?

Please note the log files weigh around 300MB.
Also the sync type I need is One-direction (L->R) Exact Mirror

Thank you,

[superflexible]

Hello,
on the second tab sheet of the FTP dialog, please remove the checkmark "Verify destination file size after transfer".

In addition, you may be able to save bandwidth if you have to download the same file again and only want to download the new data at the end. To do this, you can use the following two checkmarks in the profile under Files->More:
X Always Append Smaller Destination Files
X Always Consider Larger Files Newer

[Hayframe]

I've unchecked that box and tried again, and it still did not work.

My log file confirms the option was activated, I think:

[Internet]
LeftFTPSettings=SFTP:Port=22,AbsolutePath=Y,Flags=UTF8+SkipSizeCheck+UTC+DetectTimezone+AvoidIPV6,"Proxy=no",TimZoneOfs=0/0,"Cert=none",SSHAuth=[pass],RecursLst=N
AdvancedSSHSettings=9/YY/2013261823/15/255/16383/8192/32768/131072/32/SFTP0123456

10/01/2016 23:26:14 Retrying because Transfer error, dest size=317581605, src size=317576710
10/01/2016 23:26:14 Attempt #2 downloading sftp://MYSERVER_REDACTED.xyz/home/usern ... marker.log from U:\Sync-Duplicate\Servers\MYSERVER Root\home\username\irclogs\stream\#infomarker.log.
10/01/2016 23:36:57 TGGetOrPut/GetDestFileSize: U:\Sync-Duplicate\Servers\MYSERVER Root\home\username\irclogs\stream\#infomarker.log.d20160110-u220839.s317576710.$syncovery, Result=317585699
10/01/2016 23:36:57 Copied file too big on dest (317585699>317576710) - deleting:
U:\Sync-Duplicate\Servers\MYSERVER Root\home\username\irclogs\stream\#infomarker.log.d20160110-u220839.s317576710.$syncovery
10/01/2016 23:37:00 Transfer error, dest size=317585699, src size=317576710
U:\Sync-Duplicate\Servers\MYSERVER Root\home\username\irclogs\stream\#infomarker.log.d20160110-u220839.s317576710.$syncovery

Sorry I can't post the full log file, it contains information that I don't want showing up on google search.

I'm not sure what to do from there. I know I can solve this server-side by setting up a cronjob that will duplicate the files daily for Syncovery to mirror instead, but first I'd like to see if Syncovery is capable of solving this without any server-side trick.

*edit: unchecked

[superflexible]

Hello,
I think you also need to remove the checkmark "Automatically resume" from the Files tab sheet in the profile, because this implies a size check too.

If problems remain, can you send a full log file to support@syncovery.com ?

Thanks.

[Hayframe]

It looks like it worked!

I ran it twice to test partial updates.
Even though it seems "Append Smaller Destination Files" didn't seem to work, it seems it transferred the entire file.

My suspicion is that "Append Smaller Destination Files" caused a check on the destination file size, which turned out discrepant (bigger than the manifest) so it deleted it and tried again.

I couldn't check "Always Consider Larger Files Newer" since it was grayed out. According to http://www.helpdoc-online.com/SCS0001EN ... s-More.htm it wouldn't have made a difference in this scenario I think..?


I guess I'll just disable this feature for now, no matter, at least I now have a working copy of the IRC logs on the right-hand side!

11/01/2016 00:47:34 Resuming from 0 Bytes (sftp://MYSERVER_RECADTED.xyz/home/main/ ... center.log)
11/01/2016 00:47:34 Copied file too big on dest (94299017>94298702) - deleting:
U:\Sync-Duplicate\Servers\MYSERVER_RECADTED Root\home\main\irclogs\net\#center.log
11/01/2016 00:47:37 Retrying because Transfer error, dest size=94299017, src size=94298702
11/01/2016 00:47:37 Attempt #2 downloading sftp://MYSERVER_RECADTED.xyz/home/main/ ... center.log from U:\Sync-Duplicate\Servers\MYSERVER_RECADTED Root\home\main\irclogs\net\#center.log.
Copy L->R sftp://MYSERVER_RECADTED.xyz/home/main/ ... center.log (89,9MB)
11/01/2016 00:47:40 Resuming from 0 Bytes (sftp://MYSERVER_RECADTED.xyz/home/usern ... marker.log)
11/01/2016 00:47:40 Copied file too big on dest (317624941>317623206) - deleting:
U:\Sync-Duplicate\Servers\MYSERVER_RECADTED Root\home\username\irclogs\stream\#infomarker.log
11/01/2016 00:47:44 Retrying because Transfer error, dest size=317624941, src size=317623206
11/01/2016 00:47:44 Attempt #2 downloading sftp://MYSERVER_RECADTED.xyz/home/usern ... marker.log from U:\Sync-Duplicate\Servers\MYSERVER_RECADTED Root\home\username\irclogs\stream\#infomarker.log.
Copy L->R sftp://MYSERVER_RECADTED.xyz/home/usern ... marker.log (302,9MB)

Thank you,

[superflexible]

Hello,
in this log excerpt, it checked the file size because the transfer was resumed. "Resuming from 0 Bytes" is a little strange though. I can't see in these few lines why it resumed from 0 Bytes. I will change it so that it doesn't do that.

[Hayframe]

For your information, the 2nd file synchronization I made to test partial file update involved 9 log files, 7 of which succeeded successfully on the first attempt. They also had the "Resuming from 0 Bytes" message but they didn't restart: those log files were all affecting very slow-moving IRC channels = there were probably no new messages between the manifest, the start of transfer and the end of transfer.

Here is the full redacted log file in case you are interested.

Syncovery v7.25a, build 211 (64-bit)

Remaining days for evaluation: 24
Running on : XXX
Running as : XXX\YYY(not Admin)

Log for profile XXX Servers: MY_SERVER HOME (Only logs) (NETWORK, NO ZIP), started at 00:43:12 on 11/01/2016

LEFT: sftp://MY_SERVER.xyz/home
RIGHT: U:\Sync-Duplicate\Servers\MY_SERVER Root\home

The profile is executed in attended mode.

Previous run was at 11/01/2016 00:25:36: 3 copied (427,5MB), 57 deleted

Right Side Volume Name: XXX

SFTP Version: 3, SSH Encryption Algorithms used:16/16/13/13
SSH DownloadBlockSize:8192 PipelineLength:32 AutoAdjustTransferBlock:TRUE
SSH CompressionLevel:9
11/01/2016 00:43:15 Timezone: listing offset 0, timestamp setting offset 0
Copying Direction : Left To Right
Exact Mirror Mode is being used.

11/01/2016 00:43:18 Timezone: listing offset 0, timestamp setting offset 0
11/01/2016 00:43:21 Timezone: listing offset 0, timestamp setting offset 0

Free Space on left side : unknown, needed: 0 Bytes
Free Space on Right side: 378 GB, needed: 2 832 Bytes
11/01/2016 00:47:37 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... m/#aaa.log)
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... m/#aaa.log (5,7MB)
11/01/2016 00:47:40 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... m/#bbb.log)
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... m/#bbb.log (34,7MB)
11/01/2016 00:47:40 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... m/#ccc.log)
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... m/#ccc.log (7,3MB)
11/01/2016 00:47:41 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... m/#ddd.log)
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... m/#ddd.log (6,5MB)
11/01/2016 00:47:41 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... m/#eee.log)
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... m/#eee.log (13,0MB)
11/01/2016 00:47:41 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... m/#fff.log)
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... m/#fff.log (5,0MB)
Created folder: U:\Sync-Duplicate\Servers\MY_SERVER Root\home\website\https\repos\my_repo\.git\objects\37
Created folder: U:\Sync-Duplicate\Servers\MY_SERVER Root\home\website\https\repos\my_repo\.git\objects\93
Created folder: U:\Sync-Duplicate\Servers\MY_SERVER Root\home\website\https\repos\my_repo\.git\objects\bc
Created folder: U:\Sync-Duplicate\Servers\MY_SERVER Root\home\website\https\repos\my_repo\.git\objects\ed
11/01/2016 00:47:42 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... m/#ggg.log)
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... m/#ggg.log (9,8MB)
11/01/2016 00:47:34 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/main/irclogs/net/#center.log)
11/01/2016 00:47:34 Copied file too big on dest (94299017>94298702) - deleting:
U:\Sync-Duplicate\Servers\MY_SERVER Root\home\main\irclogs\net\#center.log
11/01/2016 00:47:37 Retrying because Transfer error, dest size=94299017, src size=94298702
11/01/2016 00:47:37 Attempt #2 downloading sftp://MY_SERVER.xyz/home/main/irclogs/net/#center.log from U:\Sync-Duplicate\Servers\MY_SERVER Root\home\main\irclogs\net\#center.log.
Copy L->R sftp://MY_SERVER.xyz/home/main/irclogs/net/#center.log (89,9MB)
11/01/2016 00:47:40 Resuming from 0 Bytes (sftp://MY_SERVER.xyz/home/username/ircl ... marker.log)
11/01/2016 00:47:40 Copied file too big on dest (317624941>317623206) - deleting:
U:\Sync-Duplicate\Servers\MY_SERVER Root\home\username\irclogs\stream\#infomarker.log
11/01/2016 00:47:44 Retrying because Transfer error, dest size=317624941, src size=317623206
11/01/2016 00:47:44 Attempt #2 downloading sftp://MY_SERVER.xyz/home/username/ircl ... marker.log from U:\Sync-Duplicate\Servers\MY_SERVER Root\home\username\irclogs\stream\#infomarker.log.
Copy L->R sftp://MY_SERVER.xyz/home/username/ircl ... marker.log (302,9MB)

SUMMARY
------------------------------------------------------------------
Short Results: 9 copied (474,7MB)
Operation completed at 00:56:54 on 11/01/2016
Total duration: 00:13:41

Copied To Right Side: 9 (474,7MB)
Folders Created On Right Side: 4
Files updated on Right side : 9

Average copying time per file : 89,5 s
Copying time for fastest file : 0,3 s
Copying time for slowest file : 553,6 s

Average transfer rate : 603,3 kB/sec
Transfer rate for slowest file : 368,8 kB/sec
Transfer rate for fastest file : 72172,7 kB/sec
For the fastest/slowest rates, only files at least 1 kilobyte in size were counted.


------------------------------------------------------------------

Profile Settings From C:\ProgramData\Syncovery\Syncovery.ini

[General]
Name=XXX Servers: MY_SERVER HOME (Only logs) (NETWORK, NO ZIP)
LastModified=11/01/2016 00:25:30
LeftPath=sftp://MY_SERVER.xyz/home
RightPath=U:\Sync-Duplicate\Servers\MY_SERVER Root\home
LeftToRight=Yes

[Exact Mirror Config]
ExactMirrorDeletes=Yes
ExactMirrorReplacesNewerFiles=Yes

[Files->More]
AlwaysAppend=Yes

[Internet]
LeftFTPSettings=SFTP:Port=22,AbsolutePath=Y,Flags=UTF8+SkipSizeCheck+UTC+DetectTimezone+AvoidIPV6,"Proxy=no",TimZoneOfs=0/0,"Cert=none",SSHAuth=[pass],RecursLst=N
AdvancedSSHSettings=9/YY/2013261823/15/255/16383/8192/32768/131072/32/SFTP0123456

[Inclusion Masks]
FileMasks=*.log

[Exclusion Masks]
ExcludeMasksIgnoreTotally=Yes

[Safety->Unattended]
UnattendedDeleteMaxPercent=10


------------------------------------------------------------------


Email Notification Settings:
Email Notification is OFF.


------------------------------------------------------------------

Otherwise, this is a pretty solid software and fits absolutely all my needs. The ability to check a manifest before committing (like in case some photos/files on the left-hand side went missing or corrupted) is an invaluable feature that I absolutely needed compared to other file synchronization software. The fact that this software aggressively strives to preserve last modified timestamps is also an absolute requirement, since I often sort my files by last modified date as part of my workflow.

I looked at the recovery options and they look pretty good. Other than a full recovery I tried deleting one file from the left-hand side, and I was able to recover it from the manifest window by clicking "Copy Right to Left", it's pretty great.

I'm absolutely happy to purchase it once my trial period ends, it solves all my backups from/to external devices needs.

[superflexible]

Hello,
many thanks! I think I should still make an improvement or bug fix here, I will make some tests and let you know.

[superflexible]

Version 7.38 avoids "Resuming from 0" and therefore also prevents the size check in these cases.

[Hayframe]

I tried that version on the same setup (with Append Smaller activated) and it worked without errors nor restarts. Sync of a 300MB log file took a few seconds with a relatively slow connection, so I guess it worked!

Thank you,