Slowness

[Jack]

Hi, I am trying out Superflexible for a solution I need. I have emailed back and forth with Tobias previously on another matter.

Anyway, the problem I'm having is regarding the speed of a profile when it reaches a certian point. The profile is over a network share between two local machins both running W2K3 server (sp2). There are millions of files being syncronized so maybe that has something to do with the problem. I am doing a mirror. Superflexible is running on the destination computer. The source computer receives files from a variety of other sources and puts them into the folders being mirrored. Everything works fine until a large group of files start coming in from multiple sources. At this point Superflexible may work fine for a time but at some point it will go into some sort of slow mode where it seems to scan for new files. Then, when it starts to copy it will just sit and take forever. Right now it has been running for about an hour or longer and has only copied 74 bytes of 32,676k.

It is not using a lot of CPU time, just constant hard drive activity on drive c: which is where the log files and database are stored. If I look into the folders containg the logs and database I can see the time stamps updated often so it is doing something. The drive where the data is being mirrored to shows only the rare flash of the activiy lights.

I don't know what's going on but I need to find a way to speed this up. Should I break the profile into smaller pieces? That would make a lot more work and be much more proned to error on my part.

[superflexible]

Hello,

please make sure you have the latest version 4.89. If problems remain, please send a few log files to support@superflexible.com. Some activity should be listed there that we can analyze. It's also possible to include Timing Info in the logs (see the Program Settings dialog - F8).

Best wishes,
Tobias Giesen

[superflexible]

In fact if your jobs use the database, you might want to try the latest preview version 5. It uses a new database engine that will handle millions of files easily. However maybe you don't even need a database. It sounds like you may not need any of the following settings, which would cause a database to be used:

- SmartTracking
- Partial File Updating
- Destination Cache for the file list

None of these seem to be of any use in your situation.

The only option that might justify using a database would be Exact Mirror with delayed deletions.

[Jack]

I'm already using version 5. None of the settings are in use that you mentioned except the delayed deletions.

I am using exact mirror and delayed deletions set to 14 days.

I saw a thread after I posted this that is similar to my situation. I am now going with the scheduled profile with 1 hour between runs. I would like to use the real time monitor but not if it takes a long time to sort it out. I had this profile running for arounde 36 hours once and it never finished. I finally had to kill it and start over. It seems like about every three days or so it does this.

I'll see how the schdule works out.

If you want to try to figure out the problem for future changes I would be willing to send you the logs. Let me know what settings you want me to use and I'll do it.

[Jack]

Is there a way to increase the amount of RAM available to the database engine? That may be what it needs.

I don't care how much RAM it uses since the main purpose of this computer is to have a ready backup of the files on the other server just in case.

[superflexible]

Hi,

I think you may still have logs that show the problem. Maybe you can look in the logs folder if there are any from around the time this happened.

In theory, real-time can be an advantage if you have many files because the whole directory tree does not need to be scanned each time. Real-time should be very fast actually, so maybe there is a problem in the preview release which needs to be fixed.

Before going to regular scheduling, you should try the job manually in attended mode to see how it performs. You probably need to install the ExtremeSync Remote Service on the other computer to speed up building the file list. See also http://www.superflexible.com/remoteservice.htm.

[superflexible]

The database cannot be the problem in V5. It is ultra fast regardless of the situtation. It uses as much RAM as it needs.

[superflexible]

By the way, it is usually better to "push" data, i.e. install the software on the source machine. It saves all those real-time notifications from having to go over the LAN before the copying can actually start.

And if you use regular scheduling, having it on the source machine is much better, too. To speed up building the file listing, you could then simply use the Destination Cache option rather than having to install the ExtremeSync Remote Service on the other end.

[Jack]

I don't install anything on my production server unless I know it will stay there.

I don't know exactly what it is but I know it's something with Superflexible since it's the only thing running on this machine right now. It's a fresh out of the box install with all the latest patches and only Superflexible is installed and running. When I stop the extremsync service the disk activity stops.

There is still plenty of RAM available according to Task manager and in PerfMon the RAM page is almost nothing yet the HDD activity light is buring brightly and only flashes off once in a while. I ram PerfMon after reading the other thread with the similar issue and the disk queue length was at or near the top of the scale constantly.

[superflexible]

Of course there will be activity, since you have configured a huge job. You need to check the logs. For example, the program might be running a full comparison of the two servers. This is configured on the Real-Time Settings dialog.

To understand what it does, you need to go to the Scheduler tab sheet and watch the progress reporting.

To understand even more, stop the service and/or background scheduler and run the "Foregound Scheduler" and watch what it does.

If you don't want to test it on the main server, you need two additional machines where you can simulate the real situation. The key is pushing the data, not pulling it.