Multiple machines, same profile?

[patrickm]

I'm testing Syncovery as a solution for redundancy in syncing two directories. An application creates an XML file in the source (left) and another application removes the XML file from the destination (right). Doesn't matter which machine copies or deletes the XML, as long as the file is not copied to the destination after it's been removed (that is, it needs to be deleted from the source ASAP). All machines have the exact same profile.

I've noticed that if one machine isn't aware that a file previously existed in the destination and was removed, it will copy the file again. If I wait long enough for each machine to run the profile at least once and 'see' the files, then I don't have any re-copy issues when I delete the file. This leads me to believe that each profile writes (overwrites) the directory states to each path (Advanced Settings > Special > Database), and that all profiles must 'see' the files first before any are removed because all profiles use the same database.

Currently, I've got the run interval set to 1s. I've not noticed any performance issues.

My questions:
1) Is my understanding of the database use/purpose correct?
2) Any performance concerns at 1s interval?
3) A better way?

[superflexible]

Hello,

I need to understand your situation better. How many storage locations are you syncing? A central one plus two "machine" ones? Totaling three? Is that right?

Cheers,
Tobias

[antonm]

Hello Tobias,

I work with patrikm, so I can provide more details on the configuration:

- We have one "source" folder on a remote server accessed via SFTP. This is where new files show up.
- The files that we transfer are usually very small, within 10-20K each
- We have two or more Windows machines running Syncovery, all of them sync files to the same network folder \\UNC\share. Each of those machines may go down and up at any time
- Other services/applications on each machine scan that network share periodically (3-5 second interval) to pick up new files, pricess them and then remove from the folder.
- After that one of the Syncovery instances detects that file was deleted and also removes it form the remote SFTP location.

What we need to ensure is that no other Syncovery instance is going to bring back the file that has just been deleted. So the queesitons are:

- Can different Synconvery instances from different machines use the same sync database located on a network share?

- It takes a few seconds to establish SSH/SFT connection. Is there a way to make such conneciton persystent to avoid such delay? We could not find any keep-alive options for remote folders.

- Will it be an issue if the sunc interval is kept at 1 second and Syncovery is kept running as a service for a long period of time (weeks, months etc)? Any memory issues?

Thanks!

Regards,
Anton

[superflexible]

Hi,

I think it will be better to use the operating mode "Move Files to Destination", meaning that Syncovery will download a file and then immediately delete it from the SFTP location. That way, you don't need any database. If two instances try to do the same download simultaneously, one of them should just fail because the destination file is locked (do not use "Automatically resume" because then destination files are not locked because a temporary download filename is used).

Unfortunately the SFTP connection cannot be kept up because that has not been implemented. Every 1 second should be fine and I don't think there are any memory issues, a) because we fight memory leaks diligently and b) because the process that executes jobs will only run 50 jobs, after which a new process is launched.

[antonm]

Thanks for the reply! We will try that configuration and see how that works for us. I couple of more quesitons:

- What happens if new source file is still being written and Syncovery copies but fails to delete it, then next instance or the same instance on the next scheduled run sees the same file? Will it try to copy it again or will just delete it? Or will it leave there in the source? Or does Syncovery try to lock file exclusively before even copying?

What we are trying to achive is that the same file is never copied multiple times from source to target. Regardless of how many instances there are up and running.

- Are there remote connections that implement keep alive? We can configure other services on the source if they would allow for such option.

Thanks!

[superflexible]

What happens if new source file is still being written and Syncovery copies but fails to delete it, then next instance or the same instance on the next scheduled run sees the same file? Will it try to copy it again or will just delete it? Or will it leave there in the source? Or does Syncovery try to lock file exclusively before even copying?

Whether Syncovery can even start to download files still being written depends on the server and the original program that creates the file. It is very unlikely to happen, because first Syncovery gets the file listing, then thinks about it, then starts downloading. During that time, small files like you mention are normally complete before Syncovery downloads them.

If it does happen, it all depends on the timestamps. If the timestamp on the source is newer, and maybe even the file size too, it will be downloaded again. But the "Move Files" mode has a Configure button with some options, and there are more options in the "Files" category (never replace etc).

Are there remote connections that implement keep alive?

No, it is not possible.

[antonm]

Thanks you!

[patrickm]

Yes, thanks for the replies, we'll give that a try!

[Dmitry]

Hello Tobias!
I have the same question -
How to setup redundant data transfer?
I have several tasks to copy large media files from several file-servers to other file servers.
Source media files moved to subfolders when transfers are done.
(Move files to destination + move deleted files into specified folder).
How to setup second server with Syncovery tasks to continue transfer tasks in case of main Syncovery server fault? (on-line redundancy)

[superflexible]

I will reply in the separate thread.