Matthew - There are no inherent limits in Omniscope regarding row/column counts held in memory. As long as your OS is 64 bit and the physical RAM is present on the machine, Omniscope will be able to use it. We are aware of clients running Omniscope on servers with 200 to 300 GB of RAM with no reported problems. We know of 24+ million row Source IOK files running on a mail-order Dell desktop that consolidated many smaller server-based relational databases and entirely replaced a planned (small) data warehouse.

Version 2.8 is focussing on delivering the HTML5/JavaScript deployment option for browsers (including IPads and Android devices) interacting with master Report IOK files running on a central server. This will enable you to overcome the 32-bit legacy OS problem wherein your Report IOK file recipients are still using 32-bit OS and therefore cannot open files requiring more than about 1.1 GB of addressable RAM.

Performance imporvements in Omniscope are on-going, but the most fundamental changes to scaleability and performance will now be in version 2.9, given the urgency of delivering the mobile HTML/JavaScript browser deployment option in 2.8. Many known scaling and performance improvements will be implemented in version 2.9 to both significantly increase the amount of data that fits in a given amount of RAM, and also the speed with which common server-side ETL processes in Source IOK files like multi-field aggregations, sorts, text concatenations, search/replace blocks etc. are executed.

We also anticipate much improved disk-caching of source blocks and perhaps eventually even some intelligent usage of SSD flash memory if it is present on the machine.

As data volumes continue to grow, adding inexpensive server RAM will be the best insurance since the rate of reduction in peak RAM footprint for a given data set in Omniscope, although likely to be very significant, is unlikely to match data volume growth in the long term.