This usually means there is a problem with the Java installation or the installconfig.properties file.

Copy your installconfig.properties file to a backup location, in case there are changes you may want to keep in future. Delete this file. Install the Full installer of the latest Omniscope version. Does this fix the problem?

I had the same problem running Omniscope under Citrix. Solution was to uncomment this line in installconfig.properties:

#MAX_MEMORY_MB=300

If this resolves it, you should go for as high as you can; 300 is too low for all but small data files. How much installed memory do you have? There are very rare cases where Omniscope automatically tries to allocate too much memory.
You may find that you have already customised this line and put a value that is TOO high, which would have the same error. Increase and reduce it until it's roughly as high as it can go.

Most of the time you can fix this problem by uninstalling the Java runtime and then reinstalling it all over again.

To do this, go to the control panel and then Programs and Features. If you’re running Windows 8.1 Update 1, just right-click on the Start button and click on Programs and Features. Locate the Java runtime, uninstall it and reboot your PC.

Next, download the latest Java Runtime from java.com, taking care to install the correct version for your PC. You will likely want the 32-bit version as the 64-bit option is specifically for 64-bit browsers. Most browsers are 32-bit on Windows, even on 64-bit editions of the OS.

Sometimes this error can be caused by not having enough system memory allocated to the Java Virtual Machine – or not enough system memory full stop.

You can also try increasing the amount of system memory allocated to the Java virtual machine by setting an environment variable.

http://net-informations.com/java/intro/jvm.htm

To do this, go back to the Control Panel and click on System, then Advanced system settings. Now click the Environment Variables button to bring up a list of all those currently set.