rsync support?

[PaulC]

first off, happy user of Super Flexible File Sync for several years now. This tool, saved me untold hours of tedious work.

One thing that I'd love to see added is rsync support. It's great to see the partial updates and ExtremeSync features evolve, but it's more practical for me to have an rsync daemon at the other end (for a variety of reasons). I'm sure ExtremeSync is very cool, but if I have a linux machine on the other side (and more often than not, a non-x86 linux machine) it doesn't help me.

I see a comment on the 'Partial File Updating' page of the Super Flexible web site that says:

"Support for an FTP connection will be added in a later release."

Supporting rsync as a destination would be way, way more useful to me than partial updates over FTP. Given that rsync daemons are commonplace on the small/home office NAS devices too, I'd argue that it's pretty relevant to folks that don't even realize they're running a linux box disguised as a NAS product

-- Paul

[superflexible]

Thanks for the suggestion. Partial File Updating via FTP is now available in the form of Synthetic Backup in version 5. So I have updated the page that you quoted from.

Rsync is not a standardized protocol. It's just a tool that some people wrote. For this reason, there seem to be no libraries available for software developers to use on Windows. We have looked into support for rsync and so far it does not seem to be feasible. I hope that this situation changes sometime in the future.

[kwantum]

It's possible to rsync your back-up with SuperFlexible, though it's not tested thoroughly it should work.

Because we can only run script after backup, it's required to make a local backup first. Don't use ZIP compression or Encryption because this won't work correctly within Rsync. ZIP seems to create files that change a lot every time you re-zip it, Rsync has the upload almost the whole file everytime...

Gzip has a rsyncable option which makes the zipped file more efficient for rsync, my script uses gzip and rsyncrypto to secure your files.

It keeps a history of 10 days on the rsync server because this was the easiest to build at this time.

See my next post for the code.

[kwantum]

Code: Select all

@echo off
color 1f
title Rsync script

REM // Script is WINDOWS ONLY!! save it e.g. in c:\rsync\rsyncbackup.bat
REM //
REM // Backup the files in the keys folder (after first run), you must have these to decrypt files!!!
REM //
REM // Download cwRsync_4.1.0_Installer.exe from http://sourceforge.net/projects/sereds/files/cwRsync/4.1.0/ and unzip bin directory to c:\rsync
REM // Download rsynccrypto from http://rsyncrypto.lingnu.com/index.php/Home_Page and install it in c:\rsync\rsyncrypto
REM // Download openssl from http://www.slproweb.com/products/Win32OpenSSL.html and install it to C:\Rsync\OpenSSL
REM //
REM // Set the variables below and start this batch script to create RSA keys
REM // Put the public RSA key from keys\id_rsa.pub in the authorized_keys file on your rsync-ssh server
REM //
REM // Make a job in SuperFlexible, backup files to local folder c:\rsync\backup
REM // Don't use ZIP, Encryption, Versioning > just Standard Copying or rsync won't do it's job correctly
REM // Zipping creates files that put data all over over the file causing rsync to upload it completely
REM // Set this batch in Job > Execute command or script ... > After: !C:\Rsync\rsyncbackup.bat (! = unlimited wait time)

REM // Set your variables here
set HOME=c:\Rsync
set HOMECYGWIN=/cygdrive/C/Rsync
set SSHUSER=rsyncuser
set SSHHOST=ssh-rsync-server.domain.tld
set SSHPORT=22
set SSHKEYNAME=id_rsa
set RSYNCRYPTOKEY=key.file
set RSYNCRYPTOPATH=/cygdrive/C/Rsync/Rsyncrypto
set RSYNCRYPTOPATHWIN=c:\Rsync\Rsyncrypto
set OPENSSLPATH=c:\Rsync\OpenSSL\bin
set SCRIPTERROR=0

REM // Make sure directory exists on backup destination
set BACKUPDESTINATION=/home/rommebackup/backup
set BACKUPSOURCE=/cygdrive/C/Rsync/backup
set BACKUPSOURCEWIN=c:\Rsync\backup

REM // Choose to restore or backup
choice /t 10 /c BR /d B /m "Press R to start restore (auto-continue backup after 10 seconds)"
IF %ERRORLEVEL% EQU 2 GOTO RESTORE
cls

title Rsync backup script

if not exist %HOME%\keys\%SSHKEYNAME% (
   echo.
   echo.=== RSA key generation ===
   %HOME%\ssh-keygen.exe -t rsa -b 2048 -f %HOMECYGWIN%/keys/%SSHKEYNAME% -v
   echo.
   echo.Put %HOME%\keys\%SSHKEYNAME%.pub in authorized_keys on your server
   echo.
   set SCRIPTERROR=1
   pause
   GOTO END
)

echo.
echo.=== Current day of backup ===
set /a DAYOFBACKUP=%DATE:~4,1%-0
echo.Day of backup=%DAYOFBACKUP%

FOR /R %BACKUPSOURCEWIN%encrypted\ %%F IN (*.*) DO (
   echo.
   echo.=== Unfinished transfer, continue to rsync backup ===
   if exist %%F GOTO BACKUP
)

echo.
echo.=== File encryption, delete original afterwards ===
FOR /R %BACKUPSOURCEWIN%\ %%F IN (*.*) DO (
   %RSYNCRYPTOPATHWIN%\rsyncrypto.exe "%%F" "%%F.encrypted" %HOME%\keys\%RSYNCRYPTOKEY% %HOME%\keys\%SSHKEYNAME% --gzip=%RSYNCRYPTOPATHWIN%\gzip.exe --risky-writes --keybits=256
   IF %ERRORLEVEL% GTR 0 (
      set SCRIPTERROR=1
      echo.=== Error in rsyncrypto command ===
   )
   del /f /q "%%F"
   echo "%%F"
)

if exist %BACKUPSOURCEWIN%encrypted rmdir /s /q %BACKUPSOURCEWIN%encrypted
if exist %BACKUPSOURCEWIN% rename %BACKUPSOURCEWIN% backupencrypted
IF %ERRORLEVEL% GTR 0 (
   set SCRIPTERROR=1
   echo.=== Error in rename encrypted directory command ===
)
if not exist %BACKUPSOURCEWIN% mkdir %BACKUPSOURCEWIN%

:BACKUP
echo.
echo.=== Rsync backup ===
REM // delete-excluded will cleanup everything if no files in your backup folder exist, this is risky for now
REM %HOME%\rsync.exe -a --delete-excluded --no-perms --no-owner --no-group -H --chmod=u+w,a+rX --partial-dir=.rsync-partial --filter="P .rsync-partial/" --link-dest="%BACKUPDESTINATION%/1" --link-dest="%BACKUPDESTINATION%/2" --link-dest="%BACKUPDESTINATION%/3" --link-dest="%BACKUPDESTINATION%/4" --link-dest="%BACKUPDESTINATION%/5" --link-dest="%BACKUPDESTINATION%/6" --link-dest="%BACKUPDESTINATION%/7" --link-dest="%BACKUPDESTINATION%/8" --link-dest="%BACKUPDESTINATION%/9" --link-dest="%BACKUPDESTINATION%/10" -v -v --stats --progress --rsh="'%HOMECYGWIN%/ssh.exe' -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no -o UserKnownHostsFile='\"%HOMECYGWIN%/.ssh/known_hosts\"' -i '%HOMECYGWIN%/keys/%SSHKEYNAME%' -p %SSHPORT%" "%BACKUPSOURCE%encrypted" "%SSHUSER%@%SSHHOST%:%BACKUPDESTINATION%/%DAYOFBACKUP%"
%HOME%\rsync.exe -a --no-perms --no-owner --no-group -H --chmod=u+w,a+rX --partial-dir=.rsync-partial --filter="P .rsync-partial/" --link-dest="%BACKUPDESTINATION%/1" --link-dest="%BACKUPDESTINATION%/2" --link-dest="%BACKUPDESTINATION%/3" --link-dest="%BACKUPDESTINATION%/4" --link-dest="%BACKUPDESTINATION%/5" --link-dest="%BACKUPDESTINATION%/6" --link-dest="%BACKUPDESTINATION%/7" --link-dest="%BACKUPDESTINATION%/8" --link-dest="%BACKUPDESTINATION%/9" --link-dest="%BACKUPDESTINATION%/10" -v -v --stats --progress --rsh="'%HOMECYGWIN%/ssh.exe' -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no -o UserKnownHostsFile='\"%HOMECYGWIN%/.ssh/known_hosts\"' -i '%HOMECYGWIN%/keys/%SSHKEYNAME%' -p %SSHPORT%" "%BACKUPSOURCE%encrypted" "%SSHUSER%@%SSHHOST%:%BACKUPDESTINATION%/%DAYOFBACKUP%"
IF %ERRORLEVEL% GTR 0 (
   set SCRIPTERROR=1
   echo.=== Error in rsync command ===
)

echo.
echo.=== Backup folder cleanup ===
IF %ERRORLEVEL% EQU 0 rmdir /s /q %BACKUPSOURCEWIN%encrypted\

GOTO WAIT

:RESTORE
title Rsync restore script
set BACKUPSOURCE=%HOMECYGWIN%/restore
set BACKUPSOURCEWIN=%HOME%\restore

echo.
set /p DAYOFBACKUP=Day of backup to restore:

echo.
echo.=== Rsync restore ===
%HOME%\rsync.exe -a --no-perms --no-owner --no-group -H --chmod=u+w,a+rX --partial-dir=.rsync-partial --filter="P .rsync-partial/" --link-dest="%BACKUPDESTINATION%/1" --link-dest="%BACKUPDESTINATION%/2" --link-dest="%BACKUPDESTINATION%/3" --link-dest="%BACKUPDESTINATION%/4" --link-dest="%BACKUPDESTINATION%/5" --link-dest="%BACKUPDESTINATION%/6" --link-dest="%BACKUPDESTINATION%/7" --link-dest="%BACKUPDESTINATION%/8" --link-dest="%BACKUPDESTINATION%/9" --link-dest="%BACKUPDESTINATION%/10" -v -v --stats --progress --rsh="'%HOMECYGWIN%/ssh.exe' -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no -o UserKnownHostsFile='\"%HOMECYGWIN%/.ssh/known_hosts\"' -i '%HOMECYGWIN%/keys/%SSHKEYNAME%' -p %SSHPORT%" "%SSHUSER%@%SSHHOST%:%BACKUPDESTINATION%/%DAYOFBACKUP%" "%BACKUPSOURCE%"
IF %ERRORLEVEL% GTR 0 set SCRIPTERROR=1

echo.
echo.=== File decryption, delete original ===
FOR /R %BACKUPSOURCEWIN%\ %%F IN (*.encrypted) DO (
   %RSYNCRYPTOPATHWIN%\rsyncrypto.exe "%%F" "%%F.restored" %HOME%\keys\%RSYNCRYPTOKEY% %HOME%\keys\%SSHKEYNAME% -d
   IF %ERRORLEVEL% GTR 0 set SCRIPTERROR=1
   del /f /q "%%F"
)

echo.
echo.=== Rename decrypted files ===
FOR /R %BACKUPSOURCEWIN%\ %%F IN (*.restored) DO (
   RENAME "%%F" "%%~nF"
)
FOR /R %BACKUPSOURCEWIN%\ %%F IN (*.encrypted) DO (
   RENAME "%%F" "%%~nF"
)

:WAIT
echo.
echo.Scripterror=%SCRIPTERROR%
echo.
choice /t 10 /c CP /d C /m "Press C to close script, press P to pause (autoclose after 10 seconds)"
IF %ERRORLEVEL% EQU 2 GOTO PAUSE

:END
exit /B %SCRIPTERROR%

:PAUSE
pause
exit /B %SCRIPTERROR%


[superflexible]

Many thanks indeed! I am working on direct rsync integration now (using rsync.exe) and your script is a very useful source.

[superflexible]

The first beta version with integrated rsync support is now on our web site. Rsync can be selected as an Internet Protocol. However this is no replacement for the script posted in this thread, because our program can't do "rsyncable" compression or encryption yet. Therefore, rsync has to re-upload most zip files completely even if only a small part of the original file has changed. But for uncompressed files it is great.