Would it be possible to put in an option to whether the processing can be stored to either disk (utilising solid state storage) or memory?
We have issues where a 22mb file can take up to 30 minutes loading on 64xWin7 with 16gigs of RAM. During this time, the RAM will increase to all available RAM and stay until the file is closed.
We have quite a lot of files like this, so we tend to: Use the diagnose memory use (settings-advanced or ctrl+alt+shift+M) to run garbage collection. When doing multiple field subsets for several similar formulas, create a concat field to use, so that subset5 becomes subset2 or similar. Same concat trick for aggregation
There are a couple of files where we still have issues like you describe, so those are run on a schedule on a separate server to free up desktops.
I'm working on the desktop to make changes to the design of the file, as the server is in use by another user. Even without refreshing data it takes alot of RAM and time to get to a point where I can edit the data manager, and make changes to layout in tabs.
This is also in mind for the server to leverage current resourses rather than adding more resources. There is access to solid state drives, except paging onto these is there anyway omniscope could utilise them? This would them make work on both server or desktop quicker as less time would be needed to load the file.
Martin - It may be that you are trying to do too much in just one IOK file, and you should consider using 'best practise' multi-file data flows spanning both your 24/7 server and your desktop. Please see the attached Reference Architecture documentation.
If you just want to edit the DM, apply some record filters high up, and filter to the first ten records, or a single location / product. The flow will then be much faster, and require less RAM. I know this isn't possible for all blocks, but just before large field organiser blocks it's a real time saver.