The power of Omniscope on the desktop is due in part from its unique client-side in-memory architecture. All the data in the file is held in the memory of the local machine, at maximum granularity and with aggregated transforms, simultaneously. As files get larger in terms of row and column counts, the RAM required on the machine runing Omniscope increases. As computers gain ever more RAM and processing power at ever lower prices, Omniscope-based solutions become ever more compelling regardless of the size of the data sets. However, some scaling and performance considerations should be borne in mind when deploying Omniscope in situations where very large data sets or central IOK datamarts will be used.
Omniscope is a locally-installed in-memory application with no inherent data volume limits in terms of record or cell counts (cells = rows x columns). If you have sufficient RAM memory addressing capacity in your operating system (64-bit systems have much, much more than 32-bit systems), plus sufficient RAM and fast processors, the upper limits of Omniscope files extend to many multi-million record data sets on 64-bit systems running 64-bit Java with 4+ GB of RAM. We currently have clients running 34 million row data sets on mail order computers with 64-bit Windows and 16 GB of RAM. However, because reporting/publishing chains may also include less-powerful 32-bit machines on 'downstream' user desktops, reporting/publishing solutions involving very large data sets may require various optimisation techniques and tools to minimise the effects of the 32-bit memory-addressing bottleneck for files delivered to recipients' desktops.
In general, highly-scalable Omniscope-based solutions should be organised as a 'waterfall', where the most RAM-intensive tasks are done on 64-bit servers or the desktops of a few power users with abundant RAM, such that the resulting 'downstream' report files are less RAM-intensive and perform well on the typical 32-bit desktops of the downstream users. In a fully-automated Enterprise Server-based, 'waterfall' implementation, a set of Omniscope files are maintained as single table (non-normalised) 'wholesale datamarts' refreshing directly from data warehouses/cubes running on the same powerful 64-bit servers with abundant RAM. Power users at the heads of the reporting/workflow chain can usually use aggregated or time-sliced (and therefore smaller) versions of these Omniscope files which have these wholesale 'datamart' files (not the data warehouse SQL reporting views) as their linked source. The desktop report files will be smaller, and can automatically refresh from the larger 'datamart' files kept on the server. The power users can then re-configure these smaller template files to generate/refresh yet smaller, more specialised reporting files for onward distribution to typical desktops/free Viewers. Smaller reporting subsets or 'dashboards' can be configured and exported as near universally-accessible Flash SWF DataPlayers for interactive web page display and embedding in documents by anyone in the chain with an activated Omniscope. (see below for information on the expanding data capacity limits of DataPlayers).
The sections below discuss options and issues related to Omniscope scaling and performance [1] in more detail:
Stand-alone DataPlayers are a near universally-accessible file export option from any Omniscope data file. Due to inherent limitations in Flash, .SWF DataPlayers will always have much lower capacity for data in terms of record or cell counts (cells = rows x columns) than does a highly-scalable, locally-installed data analysis, management and reporting solution like Omniscope. In general, DataPlayers can contain at least 20,000 records (rows), but various factors specific to the data can affect performance and impose lower limits on record count. We are re-writing the Flash generating code in ActionScript 3, and expect both scaling and performance improvements in DataPlayers increasing the record limits above 500,000 records in a single .SWF file.
Smaller data sets, aggregations (e.g. daily data rather than hourly) or defined subsets (Named Queries) from larger Omniscope .IOK files can be automatically converted to DataPlayer 'dashboards' according to a routine schedule, or on-demand using personalised or 'permissioned' data sets delivered to the Generator from back-end repositories and/or analytical staging databases.
Managing .SWF DataPlayer scaling & performance [5]
How much data can Omniscope handle? Omniscope software has no fixed upper limit on the number of rows and columns that can be managed in a single file. The effective upper limit depends on the specification of the machine running the Omniscope file, and a complex relationship between processor speed, available Windows and Java memory addressing, data types, density of the columns and the number and type of Omniscope views employed in a given file. Effective Omniscope data volume is principally related to the number of cells (rows x columns), while the effective capacity of a computer (for this purpose) is principally related to the addressing space and amount of RAM in the machine. The best way to determine how large data sets (over 5 million cells) or very large data sets (over 15 million cells) will perform in machines with different amounts of RAM is to first try it with 32-bit Omniscope Professional. If this does not run your data sets properly, try a machine with 64-bit operating system and Java installed and about 4 GB of RAM installed (most database servers are now this minimum specification). If the test on the database server shows that 4 GB of RAM and 64-bit is sufficient, you will have to upgrade your desktop to this specification, or run Omniscope from an account on the server machine.
In general, a recent 32-bit computer with:
The less than proportional increase between 1.0 and 2.0 GB of RAM results because the 32-bit Windows/Java addressing limit is reached at about 1.2 GB...well before the 32-bit computer can utilise its full 2.0 GB. Computers running 64-bit operating systems with 2.0 GB or more of RAM will have much higher limits. We have documented files of 8 million rows and 15 columns (120,000,000 cells) running on 64-bit servers with 8 GB of RAM, and some prototyping installations are currently using up to 16 GB of RAM.
The 64-bit version of Omniscope is available on every install. Due to delays in the release of some Java libraries, there are some features not yet available on 64-bit. The largest Omniscope file we know of currently running has 34 million rows and runs on a mail-order $US 2,000 desktop machine with 64-bit Windows/Java and 16 GB of RAM. If you are running Omniscope on a 64-bit machine, there are various options you can set to manage memory availability and performance. More info [2].
If you are dealing with very large data sets, or plan to distribute Omniscope report files to 32-bit desktop machines with 256 MB or less of RAM, you can minimise peak Omniscope memory use by hiding/deleting all unused columns (use Data > Manage Fields > Hide Field). A data field (column) is not loaded into memory until it is actually displayed, so opening views that display many fields, like the Chart View and the Table View, should display only the most useful fields. Fields rarely needed for filtering should only be shown on rear tabs opened only by users who really need to view and filter by the values in that field. Users who never open these tabs will have lower peak RAM requirements and better performance.
Omniscope includes tools to help analyse memory use and optimise very large files for a range of recipient machines. These tools are documented here [3].
By default, drop-down menus in Omniscope offer all fields as options for the user to display, or to add to the filter devices on the Sidebar. Often, only a few of the fields will make sense as options in a given view, or as filters on a given tab. Hiding as many fields as possible from drop-down options/Sidebar devices will generally improve tab-to-tab performance, especially for large ( > 200 columns) column set files. More info. [6]
The use of certain views limits scalability and performance in the upper reaches of file size. Omniscope performance in some views is also sensitive to the number of unique category values being used in the view.
Pivot View: currently does not handle very large numbers of categories well, especially on slower processors. If you encounter slow performance, try reducing the number of unique categories used in the Pivot View by setting wider category limits ( 'buckets' ). Future versions will perform better with more unique categories. For more information, see Pivot View Data Truncation. [7]
If you are pushing the boundaries of Omniscope's data capacity on a given machine, fine-tuning the memory allocation [8] may help. You can override the default 75% share of memory allocated to Omniscope in favour of a higher limit you specify.
Omniscope Online [9], the zero-install Web Start version of the Omniscope free Viewer, does not currently dynamically adjust its configuration according to your PC's memory. Instead a default memory ceiling is used, which limits the data capacity of the Viewer. If you encounter out-of-memory errors using Omniscope Online, please use the customise [10] link on the download page and enter a suitable value in the "Memory limit" box. You should enter approximately 75% of the installed memory of your target audience. You can then generate a parameterised download/deploy link to expose to your audience which includes this configuration.
Operating systems and programs typically come in two versions; 32-bit (each instruction processed is 32 bits long) and 64-bit (each instruction processed is 64-bits long). Almost all microprocessors are 64-bit capable now, and most data centre servers have already been converted to 64-bit operating systems because of its superior speed and higher memory-addressing limits. The 32-bit version of Omniscope (which can be installed on both 32-bit and 64-bit Windows) is restricted by the 32-bit Windows-imposed memory limit, effectively limiting the typical Omniscope file size to about 1.2 million records because Windows/Java is unable to provide Omniscope with access to more than about 1 GB of installed RAM. Running 64-bit Omniscope on 64-bit Windows/Java platforms effectively removes this limit (just like using longer telephone numbers permits more numbers to be issued) and allows Omniscope data sets to scale up to the limits of the physical RAM installed on the machine.
Using 64-bit Omniscope on 64 bit operating systems avoids the 32-bit Windows memory addressing bottleneck, allowing far higher record/column counts in-memory. PCs running 64-bit Windows, 64-bit Java, and 64-bit Omniscope are limited only by the amount of physical RAM memory present in the machine. Running 64-bit rather than 32-bit Windows is not 'just twice as good', it is literally billions of times better, in terms of in-memory data management tools like Omniscope. It is now quite easy to purchase PCs at consumer prices with 8-12GB or more of RAM memory, although the PC must have a 64-bit Windows operating system installed for this additional RAM to be of use. Almost all new installations of Windows 7 and 8 are 64-bit, as are all new Windows 8 mobile tablets.
Both 32 and 64-bit versions of Omniscope and the bundled Java Private Virtual Machine (PVM) are available in the same downloadable and portable installer. Omniscope will detect the operating system and offer the correct installation options automatically. By default, the 64-bit version of Omniscope limits Omniscope to 75% of physical RAM memory, unless the machine has more than 8GB of RAM, in which case Omniscope will take all but 2GB, leaving that 2GB of RAM for the operating system. These default allocations can be overridden and scalability and performance further improved by customising Omniscope memory allocation [8]. If you are installing on a server, and anticipate either multiple concurrent users on the same machine, or if you are considering running both Scheduler Server and Mobile Web Server processes on the same machine, you shuold pay special attention to the options for configring Omniscope access to RAM on an account by account basis. Also, if you plan to run Omniscope Server/Publisher as a 24/7 Windows service, you must also ensure consistent settings in the RAM allocation of the wrapper, as discussed here [11].
Omniscope is an in-memory data management appliication, and should always be given access to as much (now relatively inexpensive) RAM memory as possible. By default, Omniscope limits itself to 75% of the physical intalled RAM memory it finds on the machine. This default value was chosen conservatively to ensure there is sufficient free memory in the system for the Operating System (OS) itself to perform other system tasks and to run other applications without degrading overall system performance.
The 32-bit version of Omniscope is subject to an additional OS-imposed cap of 1100MB, due to a limitation of 32-bit Windows where larger values cause Omniscope not to start due to lack of contiguous RAM memory. This default 32-bit upper limit cap has been set to cover 99% of 32-bit Windows PCs, but it could safely be set higher on many 32-bit PCs.
When you start Omniscope, the application launcher determines how much memory it should allow Omniscope to use. In very rare cases the setting may be too high relative to the situation on the specific machine, and this can cause Omniscope not to start, as discussed here [12]. More commonly, the defaults may cause Omniscope to start with less memory than would be otherwise possible, and you may wish to change change towards the upper limits if your data sets will be pushing the boundaries of accessible RAM on your machine. You can control how Omniscope manages RAM memory on both 64-bit and 32-bit installations by changing a setting found in the intallconfig.properties file as described below.
This is for advanced use only, as incorrect settings in the installation configuration files may stop your Omniscope installation from working [12].
With 32-bit Omniscope, if you specify too high a required contiguous memory setting, Omniscope will not start. If this happens, reduce the value (perhaps in 100MB steps) until Omniscope starts successfully. Depending on your machine, you may be able to go as high as 1900MB of RAM on a 32-bit machine, but it is unlikely you can go any further without changing to running 64-bit Omniscope on a 64-bit operating system.
With both 32-bit and 64-bit Omniscope, you should never specify more than the installed physical RAM memory, and you should ideally leave some 'headroom' for the operating system and other applications... otherwise machine performance will be degraded. Vista is particularly memory hungry, and if you are still running this OS, Vista should always be left at least 2 GB of memory of its own when allocating allowable memory to Omniscope.
If you will have more than one Omniscope user account operating concurrently on the same (cloud-based) machine, please see the section below on managing RAM allocations for multiple installations on the same machine.
To change default behaviors, you can edit the Omniscope installconfig.properties text file which can typically be found in one of the following locations, depending on type of PC and type of installation:
Windows: Installed for all usersOpen the installconfig.properties file in Notepad (it should look similar to the example below) and look for the line starting #MAX_MEMORY_MB= (highlighted below). By default, this line is commented out with a #. Remove the # in front (and leave no spaces at the beginning of the line or anywhere in the value), and enter the value in MB (e.g. 3500 for approximately 3.5GB) at the end of the line. Save the file with this change. You will need administrative permissions to edit this file. If using Vista or another Windows OS with user account control, you will find it easier if you copy this file onto your desktop before editing it, then copy it back after editing.
To revert back to the default 75% maximum memory allocation, replace the # sign at the start of the line to comment it out again.
On a 32-bit PC with 2GB installed RAM memory, we recommend pushing the limit up to 1750MB, providing Omniscope will still start. On some PCs this will give a significant boost to the capability of Omniscope. On a 64-bit Windows PC with a larger 12GB memory, we recommend going up from 75% (9GB) to at least 10GB(10000MB). This will give a moderate boost in file size capacity and reduce any paging to rotating disk memory which slows performance.
(make sure there are no spaces in the uncommented line before you save the change)
# This is an optional manually-specified max memory cap for the Java PVM, an integer specifying the...
Omniscope's default configuration is designed to allow ONE forefront Omniscope instance to take everything it can from the OS if it needs. In the extreme case of large data sets where Omniscope does take all memory up to the cap, all other apps, including other instances of Omniscope running concurrently on the same machine will be forced to page to disk, which can really slow down performance, unless SSD disks are on the machine.
In a multi-Omniscope user environment, unless the machine has high-speed solid state disk (SSD) memory, it is very important to make sure that Omniscope paging to disk doesn't happen in the middle of an active Omniscope user session, only when users log in or out, switch apps, or wake up an inactive app by starting to use it (the next day, for example).
To avoid this, all accounts on the same machine should have #MAX_MEMORY_MB= settings that ensure that no one session can take more RAM than it should to accomodate the data sets running concurrently on other accounts on the same machine.
Here are the calculations needed.
MAX_MEMORY_MB = Min ( RAM - Overhead , (RAM + Pagefile - Overhead) / Max_instances )
• Overhead
The amount of free memory you want to leave for other non-Omniscope processes and the OS. (e.g. on a multi-tenant server you may want more than 2GB per user) plus a minimum of 2Gb for the OS itself.
• Max_instances
The max number of simultaneous Omniscope instances
• Pagefile
The Windows page file size (if present)
• RAM
The total RAM installed on your server.
e.g. 96GB RAM, 2GB overhead, 6 Omniscope instances:
Overhead = 14 GB (2GB x 6 users + 2GB OS)
Max_instances = 6
Pagefile = 82 GB see notes below
RAM = 96
Min ( 96 - 14, (96 + 82 - 14) / 6 ) = 27.3GB
MAX_MEMORY_MB=27300
• Pagefile
Windows system page file size needed = Total of memory needed by paged applications
Those apps which are open but not being used in a given instant - to ensure there's enough page file to hold all inactive apps, without which you may find Omniscope hangs.
e.g 96GB RAM with 6 concurrent users of Omniscope.
• Default max-memory setting: 82GB ( RAM(96GB) - overhead (2GB x 6 users + 2GB OS) )
• Omniscope max-memory setting: 82GB / 6 concurrent Omniscope = 13.6 GB
• Page file size needed: 6 * Omniscope max-memory = 82GB
Shared machine performance may be further improved if you can reduce Omniscope max-memory further. So, if you know that for your data sets Omniscope will be fine within a 4GB RAM upper limit for each of the machine users, you can reduce the max limit even further since there is no chance of paging to disk, and in so doing less page file space will be needed.
wrapper.java.maxmemory=14500
The defaults will allocate the lesser of 75% of physical RAM memory on smaller machines up to 8GB RAM, or physical memory less 2GB on larger machines with over 8 GB RAM. So, on a 64-bit PC with 16GB, both the installconfig and wrapper.config maxmemory values will automatically be set to 14GB. You may choose to edit this to provide a bit more RAM if your data sets are larger, or a bit less if you wish to guarantee more memory to other applications and services running on the same machine(s) which is probably acceptable on dev/test and fail-over machines, but not ideal for production servers.
Omniscope now includes tools to help us help you to diagnose specific memory problems, whether suspected leaks or out-of-memory errors. A memory leak is when Omniscope's memory footprint increases over time during the same session for no apparent reason, resulting in eventual slow-down (typically over a significant period of time). An 'out-of-memory' error occurs when an operation is attempted for which there is not enough physical RAM memory in your PC allocated to your session of Omniscope.
To use these tools, you will need Java 1.5.0 b7+ or Java 1.6+. Check your active version of Java in Help > About. Use Java 6 if possible, as the memory tool support is much better.
If Omniscope appears to slow down over time, when the data and view complexity is relatively constant, particularly after a long period of use, this is likely to be a memory leak. Please see Memory leak analysis [4].
There are some datasets that Omniscope can't handle if there is not enough memory available. However, you may discover a fault where Omniscope attempts to request more memory than it should. If this occurs, you can configure automatic generation of memory dump files, which you can send to Visokio for analysis.
Open the Memory Diagnosis tool by pressing Ctrl+Shift+Alt+M or choosing Settings > Advanced > Diagnose memory use. You may need to enable Settings > Advanced > Show diagnosis tools and restart Omniscope first.
If you have Java 5 (1.5) or earlier you will see a Force out-of-memory button which is used to trigger a memory dump. Naturally occuring out-of-memory errors will also trigger this. You will need to have edited your installconfig.properties [13] file to enable "heap dump on out of memory" prior to starting Omniscope. When you are confident a memory leak is apparent, make sure you have saved your work, and click this button. This will cause a memory dump file (*.hprof) to be created in the program folder.
If you have Java 6 or later, you will see a tickbox Dump memory on out-of-memory. Tick this box to enable memory dumping. You will also see the button Dump memory. Use this button to create an ad-hoc memory dump file - you will be prompted to choose for a save location, then Omniscope will freeze for a few seconds as it generates the dump file. Note that memory dumps can take up a lot of space on disk, so generate memory dumps sparingly and clean up old files.
Memory dumps can be large, so please compress using ZIP or another common format. Please be sure to give us the full product version (e.g. "2.5 b231") if you send us a dump file.
A memory leak may be the cause if Omniscope's RAM memory footprint is increasing over time for no obvious reason. Memory leaks are unlikely in sable production releases of Omniscope, but more likely in daily, alpha or beta releases. If you suspect a memory leak in Omniscope, please follow these steps to verify the leak.
The Memory Diagnosis [3] tool is used to observe the patterns in the Omniscope's memory use. Press Ctrl+Shift+Alt+M in Omniscope to open this tool.
Memory management algorithms in the Java VM used by Omniscope can cause apparently random fluctuations - sudden increases and drops, or small continuous changes. We will use the Extreme GC button (which takes about 8 seconds) to ask Omniscope to free up as much non-essential memory as possible, removing this "noise" from the results. We will leave the Memory Diagnosis window open throughout this analysis.
First you must identify and narrow down the task or group of tasks causing the leak.
Then do the same exact task 3 times (or more) in a row.
After both iterations 2 and 3 (and perhaps further), click the Extreme GC button and note the memory usage. Ignore small changes (1% differences), but anything larger is a potential leak.
Example results:
Not a leak:
- exit and open report page X
- exit and open report page X -> Extreme GC -> 150mb
- exit and open report page X -> Extreme GC -> 150mb +/- approx. 1%
Probably a leak:
- exit and open report page X
- exit and open report page X -> Extreme GC -> 150mb
- exit and open report page X -> Extreme GC -> 160mb
You have to do at least 3 iterations, but may choose to do more. If, for example, you do 5 iterations, and there is a steady increase across several of the iterations, this is an even more compelling proof of a leak.
Note: it is not considered a leak if a task consumes memory which isn't released immediately, but only if the exact same task done repeatedly consumes more and more memory in a memory-confined situation (simulated by Extreme GC). This is why we discount any increase in memory used for iterations 1 to 2.
Please report from inside Omniscope using Help > Error reporting -> Report a problem.
In the report, describe the task you are repeating in precise detail. Include the results of your memory diagnosis.
You may be subsequently contacted to ask for the IOK file (in confidence), so keep a copy of the file in the correct state needed to reproduce the problem (layout, data, settings, etc.). We may also ask for a memory dump [3], although this is unlikely.
DataPlayer is no longer being developed. To use, you must enable Settings > Advanced > Show legacy features.
The maximum number of records that will perform well inside a DataPlayer is evolving along with Flash itself. The effective upper limit depends on the size and character of the data set (number of cells= rows x columns), and the configuration of the DataPlayer. In general, DataPlayers containing 5,000 records or less will perform very well on most modern machines. Depending on content and configuration, 10,000 record (and beyond) DataPlayers are possible. We are in the process of re-writing underlying code in ActionScript 3, which should dramatically increase the scaleability and performance of second-generation DataPlayers.
This section provides some links to working examples of DataPlayers containing relatively large record sets, together with some hints regarding performance optimisation in the data and settings necessary to accommodate relatively large numbers of records.
Arguably, this demo really has too many columns to be useful, since the values in most of the columns are not used for selecting, only shown in details. The record details could just as easily be shown on linked pages, rather than embedded inside the Flash file. This demo is displayed here for reference only.
This demo normally loads without warning notices and runs well in most recent browsers. However, when loading this page, some older browsers and less powerful machines may see a notice (sometimes repeatedly) from the Adobe Flash plug in:
"A script in this movie is causing Adobe Flash Player 9 to run slowly. If it continues to run your computer may become unresponsive. Do you want to abort this script?" If the user presses No, (sometimes repeatedly) the loading will continue and the DataPlayer should eventually display:
Links:
[1] http://kb.visokio.com/kb/omniscope-scaling
[2] http://kb.visokio.com/kb/64-bit-configuration
[3] http://kb.visokio.com/kb/memory-diagnosis
[4] http://kb.visokio.com/kb/memory-leak-analysis
[5] http://kb.visokio.com/node/149
[6] http://kb.visokio.com/using-drop-downs
[7] http://kb.visokio.com/kb/pivot-truncation
[8] http://kb.visokio.com/kb/omniscope-memory-allocation
[9] http://kb.visokio.com/getomniscope
[10] http://tc.visokio.com/webstart/
[11] http://kb.visokio.com/kb/windows-service
[12] http://kb.visokio.com/kb/java-memory-fragmentation
[13] http://kb.visokio.com/node/288