Folder monitoring speed on folder with many objects

[habajaba]

I am running beta 28 and have setup folder monitoring on a folder with many objects (5,970,000 files, 100,550 folders, 80GB). I added the remote service on the other end to speed things up. Unfortunately, what I’m seeing is a constant load on my server of 25% CPU and 30 – 60MB memory used.

The program cycles from building list, to copying files, then back again. It is never idle. Recently, I timed a cycle. It took 6m15s to run and only copied 10 files, 150.8kb total. Just to reiterate, that’s over 6 minutes, eating 25% CPU and 60MB of memory to copy 150kb.

My question is, what can we do to improve this process? Building list seems to take the majority of time, which I expected to go away with the remote service in place. I’m fearful of putting this out on production where that CPU load could have a performance impact. Any help would be appreciated.

[superflexible]

These figures are actually extremely good for so many files.

Please clarify what you mean by "folder monitoring". In the profile, you can either specify a regular schedule or use Real-Time Synchronization. Real-Time Mode will not scan all the folders, so it should be very fast. It only depends on how many files change per second.

With Real-Time, the new version 6 offers the option to use an event-based mode or a folder-based mode. You need to try both and find out which one is faster for you. Maybe you can post or send a log created "With Timing Details" so I can get an idea of what you are doing.

[habajaba]

While I understand having that many objects is an issue, I need to find a solution that works well in this scenario. This file store is only going to grow, so the speed issues will only become greater over time. Also, this is all local right now, but the other server will be remote over a VPN soon. I would assume that will slow it down even more.

I am running beta 6 and using "process complete folders" per your suggestion previously. I just emailed you a log with timing details. Hopefully it gives you more info on the issue.

Thanks!