Ditto

Bernard - try changing the Java VM options to enable releasing memory back to the operating system. See "Additional JVM arguments" here:
http://www.visokio.com/kb/installconfig .
Update your installconfig.properties file to have the following line and restart the Scheduler:
ADDITIONAL_JVM_ARGS=-XX:+UseSerialGC
This may cause very short periods of unresponsiveness in the user interface, in exchange for being able to release freed memory back to the operating system.

Use Omniscope_debug.exe in the install folder to verify that the new -XX option is being passed correctly to Java. You will see a series of dialogs and should see this option listed in one of them.

Also schedule the new "Diagnostics" action. You may need to upgrade to a newer version of 2.6. "Garbage Collect First" is ticked and schedule the task every 5 minutes.

This will allow you to monitor whether the Java process releases memory back to the operating system when sufficient memory becomes free. As well as observing Task Manager, check the logs for the Diagnostics action.

Please report back here with the results. If necessary we can follow this with a memory dump to allow us to look for memory leaks, but this would need to be justified by unnecessarily high "used" memory figures in the Diagnostics action output.

Neil - we have already given you this option but I haven't heard back if you have been able to observe Omniscope releasing memory to the operating system (i.e. Task Manager memory value going significantly down as well as up) since you made this change.

Update: if you are using the Windows service, you must also update your "wrapper.conf" file to have the line:

wrapper.java.additional.1=-XX:+UseSerialGC

(assuming you have no other uncommented "wrapper.java.additional" lines).

Restart the service after making this change.

I have set the +UseSerialGC flag and checked it was being passed through. Then stopped and restarted the scheduler.

Still getting the same problem with the scheduler taking all the memory, windows task manager shows a javaw.exe with a memory private working set of 162,418,012K and performance tab shows 99% of physical memory allocated.

The Scheduler Diagnostics does not seem to be reporting the same thing. It shows:

Running extended garbage collector...
Memory diagnostics:
Used: 20.8 MB
Allocated: 1.92 GB
Max allocation: 135.79 GB

I have just stopped the scheduler and restarted it.

The javaw.exe process is now using just 484,624K as its working set, physical memory used has dropped to 2%.

A second process running on the server has become a lot more responsive.

The scheduler diagnostics is giving almost exactly the same output, so it seems to be reporting something totally different

Running extended garbage collector...
Memory diagnostics:
Used: 20.79 MB
Allocated: 1.92 GB
Max allocation: 135.79 GB

I would expect Task Manager to something close to "Allocated", perhaps as much as 300mb larger than "Allocated". However, Task Manager does not always give an accurate picture of memory usage. It is more useful to look at overall system performance.

Can I confirm you have no more than 1 java process for all user accounts? And this is the process for the scheduler, running as a service? And your wrapper.conf file has been edited as above?

If so, do you see the memory for this process go up and then down again, or does it only go up as scheduled tasks execute?

There appears to be a separate java process for each omniscope instance, they are all running from same user. The scheduler is running under this user, not as a windows service.

How do we set it to a single java instance?

You can't; I'm merely trying to rule out confusion as to which java instance you're looking at. So, the scheduler is running as a windowed app, with a 'Stop' button? I'm a little unsure as to exactly what you're seeing with memory in Task Mgr and in Diagnostics over time as the scheduler moves between periods of running memory-intensive tasks and periods of idleness. Please can you post back here. If necessary we can pick this up with a join.me session - please use Inbox at the top to send me a message with a suitable time 9-6pm BST today.

Yes - scheduler is running as windowed app with stop button.

Java process in task manager is "using" all the RAM
When I click stop and close scheduler all the RAM gets released.

Even when I close all other omniscope processes and leave the single java process running, the diagnosics reports as before with 20MB used, 1.8GB allocated and 135GB max.

Following yesterday's conversation the situation has improved.

The forked execution option in scheduler has been unticked.

Memory is now being freed up, but there still seems to be a gradual increment of around 1GB in allocated memory each time a scheduled task is run.

This morning the reported memory usage is 10GB after all overnight tasks have run, at the same point for the past few days it was 156GB

Looking forward to your analysis of the memory dump.

Bernard, thanks for posting the summary following our call.

Thanks to your sending us the memory dump, we were able to identify and resolve several severe memory leaks introduced in 2.6.

The fixes will be available from tonight's build, 2.6 b639 or later. Please install when convenient and report back.

Steve

Are the fixes you mention apply to Omniscope as well or just the scheduler?

Some fixes apply to either.

Summary of this issue:

- By default, the 64-bit version of Omniscope (including the scheduler process) does not release memory to the operating system. For long-running services like the scheduler this can be a problem, and can be resolved by using the -XX:+UseSerialGC option in both the wrapper.conf and installconfig.properties files as documented above.

- Even with this option, the scheduler may not release memory to the operating system immediately. Using the "Diagnostics" action (available after enabling "Settings > Advanced > Show diagnosis tools") allows you to trigger a "garbage collection" which accelerates releasing of memory.

- The Diagnostics action also logs current memory usage within the scheduler process. The key items are "Used" (currently in-use) and "Allocated" (allocated to the process but free within the process - i.e. not released to the operating system - there is always a fair amount of "headroom" needed to be allocated but free).

- If you have ticked "Fork scheduled execution" in the scheduler config, this means the Diagnostics action is largely useless. Since, with this option, each task executes in a separate process, the Diagnostics action has no use, unless run within a Chain action. We recommend keeping "Fork scheduled execution" unticked, which is the default.

- As documented above, 2.6 versions before b639 have several memory leaks which result in a significant increase in baseline memory usage between tasks. If you find evidence of further leaks, please post back here and we may instruct how to send us a memory dump for analysis. A memory leak is proven by running the same task, then a Diagnostics with "garbage collection" task, and no other tasks, in succession, with "Used" memory in the Diagnostics output observed to clearly increase substantially every iteration, for several iterations.

Steve

Thanks for all the work on this. Now running Build 640 on the server and after 3 days of running all reports the memory used is 0.76GB, allocated 1.367GB and task manager shows only 2GB used by the javaw process.

Still running garbage collector as diagnostics task. Will try and upgrade server to latest version with integrated garbage collection in due course, but for now this has solved the problem

That's great. Thanks for the update.