Scott - wow! 34 million cells is more of a data feed than a dashboard report...what do your clients DO with all that data if they don't have 64-bit operating systems?

The general methodology for optimising big Report IOK files up that are running up against the 32-bit Windows memory addressing limit on recipient desktops is outlined here:

http://forums.visokio.com/discussion/comment/2900/#Comment_2900

Please have a look throught this checklist and perhaps we can follow a similar approach to tuning this file.

An additional option is using automated batch reporting to 'personalise' the files to be distributed/downloaded into 2-tier distribution, i.e. Report IOK recipients having 32-bit systems and less need for re-purposing the data seeing only the first few tabs displaying fewer fields (fields are only loaded into local RAM at the point they are first displayed), and the full 'IOK Data Feed' version for 'power recipients' with 64-machines who need to see and re-purpose the entire data feed, adding in their own data etc.

Hi Thomas - thanks for your input. All users require access to all of the data, so personalising the files probably won't help. We also can't convert any of the derived fields to static values as this would restrict the ability to display filtered metrics. We've decided to significantly cut the amount of data in the iok - although this will limit the analysis available possible it will at least mean that all users are able to use the file. FYI - the data volumes in this file aren't particularly unusual - we often need to report across keyword-level data (for search campaigns) or display asset-level data (for banner campaigns), which can both generate very large volumes of data. Going forward, will memory management be improved in future releases?

Thanks for your help.

Scott - In a word, yes... there will be on-going improvement. We have held up version 2.6 at the near-unanimous request of all advertising clients (and some non-advertising clients) only long enough to put in dual-axis charts. Our first priority for version 2.7 is to implement a long list of known improvements in peak RAM footprint requirement management and formula/aggregation/subset processing to address the limitations imposed by 32-bit OS machines. We believe we can significantly increase the size of data sets that will fit in local RAM memory within the fixed limits imposed by all 32-bit operating systems; Windows, Apple, or Linux. Above and beyond that, we will make improved use of disk caching.

Although many of our clients in other verticals like Banking are already committed to all-64-bit for all staff PC deployments by the end of this year, we know that there will be a 'long tail' of 32-bit OS machines out there for a long time to come, even though only 64-bit OS options will be available in future (with solid-state hard drives that reduce the performance delays from disk caching).

Although we expect to improve significantly the data volumes that can be managed within 32-bit machines, there will always be an upper limit of addressable memory on 32-bit OS, and we think it reasonable that if someone asks you for what is essentially a processed data feed rather than a synthesized management 'dashboard' report, you will be justified in telling them that of course it is available, as long as they either have, or upgrade to, the 64-bit OS required to handle it.

Hi Thomas - great, thanks very much for the update.

Hi Thomas,

I also have an issue with Omniscope's performance.
I have a dataset with 4-5 fields and approximately 12 millions of rows. I need to aggregate for one field.
I was able to do this task until a month ago, but now Omniscope runs out of memory. I have tried also with half of the dataset and I had the same result.
I was wondering if this could be related with some change/different setting in the latest builds, as I am still using the same 64-bit with 8GB RAM.
What do you think?

Many Thanks,
Filippo

Filippo - Please make sure you are running the very latest verion (build 700+). Then, before you run the task again, go to Settings > Advanced > Diagnose object cache and click the Reset all button at the bottom of the dialogue. If Omniscope runs out of memory again, please submit the problem report, which carries no data but a lot of diagnostic information.