Problem with large files and realtime monitoring

[kiddailey]

Still loving this amazingly awesome software. I use realtime monitoring to mirror a few GBs of large files from one master server to another server and Amazon S3 and it works great.

I've noticed a weird issue though with the realtime folder monitoring and large files. Particularly when copying files into the monitored folders over a network. The problem relates to the upload being started before the file has finished copying into the folder that is being monitored. It appears that the software is creating a shadow copy to upload before the file copy is complete.

What eventually happens is that the software uploads a corrupt/partial file. When the partial upload is complete, it then uploads the complete file, replacing the partially uploaded file with the full file.

The real trouble here is bandwidth (especially if you're paying for usage like with AmazonS3). SFFS is essentially using double the bandwidth for large files!

It would be great to have the software not start the upload until the file copy is done. Is there any way that it could be made to wait, or is this a limitation of the OS's directory changed notification?

In the meantime, there are a couple of (annoying) workarounds:

• * If the file is under 100MB or so, copy the file to a folder outside the monitored folder first, and then move it into the monitored folder so that the move is performed quickly within the same filesystem.
• 
  
• * Pause the scheduler before the copy, and then restart it after the files are completely copied to the monitored folder.

[superflexible]

On the File Access tab sheet, please turn off the use of Volume Shadowing entirely. Also choose "Database-safe copying". Then files will be uploaded when they are complete only.

[kiddailey]

Cool!! And wow. I guess I should have looked a little harder in the app. Is there anything this app can't already do?

[kiddailey]

I have two schedules monitoring the same folder (they mirror the files to different servers) and although it resolves the original issue, it causes a new issue when they both try to read/upload the file at the same time.

Is there a particular configuration setting that will work around this particular setup while still resolving the original issue?

[superflexible]

It depends on which app copies the files there. You may be able to do it without the "Database-safe mode". Otherwise the two profiles will need to copy the file one after another.

[kiddailey]

It's just a simple filesystem copy from a Unix machine onto a Windows SMB share.

Unfortunately, with the "Database-safe mode" turned off, I'm back to square one. The upload starts before the file copy is complete.

I do like having them upload at the same time, but is there a way to setup the schedules so that one executes after the other is complete? Other than the delay between retries setting, I don't see anything that would allow me to setup this kind of dependency.

Sounds like I just may have to live with copying the files to a different folder on the share first and then move them into the monitored folder.

[superflexible]

With database-safe mode, one of the profiles will get the lock and the other will just retry until it can copy it too. That is, as long as new real-time events come in or you specify a regular schedule.

However - maybe you can set only one of the jobs to use real-time and the other can be triggered as an "Execute after" job in the first profile. This can be one on the Job tab sheet. The "Execute before/after" dialog explains how to chain profiles.

[kiddailey]

Thanks! That works as long as there is only a single file that is being copied -- though it obviously won't upload to both mirrors at the same time.

I think the only (mostly) foolproof method is to move the files to the same filesystem and then move them from there into the monitored folder.

SFFS really lives up to its name! If I had more free time, I'd make a set of use-case tutorials

[superflexible]

Thanks. I guess at some point in the future I should implement a database-safe mode that tolerates multiple jobs copying the same file. But that is a bit difficult.

[kiddailey]

It definitely does sound challenging

I've been running a chained setup as described above for a few days now and thought I'd mention a couple of points as a follow-up note for anyone that stumbles by here:

• • Depending on your particular case, it may be wise to use the remote server that is the most expensive bandwidth-wise as your primary profile doing the realtime monitoring. The reason is that the folder monitoring profile will use less bandwidth because it only uploads the files as they are detected. The chained profiles that follow however, will often do a complete remote folder scan and then upload the files, using a bit more bandwidth for the scan.
  
  In the case of AmazonS3, you pay for directory listing requests in addition to bandwidth for transferring files, so I imagine this could add up pretty quick if your synchronization activity is significant.
  
  
• • You may still have issues with multiple files being moved into the monitored folder at once, particularly when transferring over network shares. As I mentioned above, copying the files somewhere else on the same filesystem and moving them into the monitored folder from there seems to be the best way to deal with it.