Managing Data Refresh Options

Repeatedly keeping central and distributed copies of data current 

Omniscope can be part of broad data integration, management, reporting, presentation and publishing solutions. Activated editions of Omniscope can connect to and refresh from data files and repositories such as transactional database tables, data warehouses, analytic data structures, etc. via standard protocols such as ODBC and JDBC. Omniscope's direct database connection can import the results of any SQL statement that returns a table. Omniscope also contains a non-SQL wizard that enables anyone to perform merges and joins to assemble and import a flat tabular structure, and these operations are remembered and re-executed on file open or refresh.

Omniscope .IOK/.IOM files can have other larger .IOK/.IOM files as their linked sources and can refresh on opening, as well as auto-refresh from these linked source files.  This means that a set of large, entity-level (e.g. people and all things joined to people, places and all things joined to places, things and all things joined to things) Omniscope files automatically refreshing from the data warehouse can act as 'datamarts' with many 'child' .IOK/.IOM files each having the central .IOK/IOM data mart file(s) as their linked source. This architecture maintains a single data source in the database, yet eliminates the 'SQL bottleneck' that arises from the need to write and maintain a proliferation of SQL queries tied to ever-changing reports. Using Omniscope, end users themselves can open/assemble the source files they need for a given report or analysis. Once their file is configured, it will auto-refresh not directly from the database but rather via the automatically-refreshed .IOK/.IOM data mart files on the server. No direct database access or 'retail' query loads, and no SQL, or other types of commands required of the user.

Below we discuss data import and refresh options including importing/refreshing data from delimited or XML-tagged data files or direct connections to relational transactional and analytical databases. Web integration options are discussed here. Omniscope also supports an outside browser option that allows menus of multiple files to be browsed and selected from within Omniscope before any .IOK file is opened.

Data Refresh Options

Omniscope files running in activated editions remember their linked sources, and refresh their data from the linked source in various configurable ways:

Note: free Viewers are not activated and will only open existing Omniscope .IOK files, and will not check linked sources for updates/refresh. Only activated Omniscopes will check linked sources for updates/refresh, either on opening, or via real-time auto-refresh.

1. Frequent data file dump and refresh - activated Omniscopes: Your proprietary application/database produces a reporting data 'dump' file is .CSV (or other supported delimited format), or in .XML, as frequently as necessary. Omniscope files having the 'dumped' data file as a linked source are configured to refresh data from this linked data source file on opening and/ via auto-refresh if necessary.  Note: only saved changes to the source data file will be detected and imported
2. Refresh on demand via the web - activated Omniscopes: You provide a full data dump on demand at a URL via HTTP (for example, by using a Java servlet in a web server dumping .CSV or .XML). Omniscope file is configured to auto-refresh data from the URL according to a frequency.
3. Automation of .IOK 'data mart' file(s) refresh using Enterprise XML Actions: You provide an XML "script" of actions, executed by Omniscope Enterprise, which update central .IOK 'data mart' files from one or more flat files or databases. You either trigger execution of this script from within your proprietary system as and when the data changes, using the Scheduler's  'watch folder' (for immediate execution) or by setting the Scheduler to refresh the 'data mart' .IOK files on the server at short intervals.
4. Client-side Automatic Refresh - Using one of the above options to populate and periodically refresh an .IOK/.IOM source 'data mart' source on a server, the client-side 'child' version of the file can be configured to auto-refresh the data from is parent over a network or the open web according to a client refresh frequency you specify. Omniscope installs with an example of a file that auto-refreshes over the open web from our server to your desktop. Try it from File > Open demo > Live data refresh demo. More detail on Using Auto-Refresh.
5. Proprietary Connector- If necessary, we can work with you to define a Connector API for retrieving just the updated data from your proprietary system at appropriate times. This will be highly specific to the nature of your own systems, the installation environment, the data structure and degree of data "liveness". Once your users' needs are defined, we will develop a Connector specifically for your API that efficiently retrieves just the updated data, only when necessary, from your system. Using custom Connector, rather than standard protocols like ODBC/JDBC, the refresh can get very very close to real time*. This option involves chargeable consultancy and development work.

A given solution architecture can employ multiple options for refreshing Omniscope files on always-on servers and from there to distibuted local or remote client installations. For example, a leading asset manager uses the Server Edition Scheduler to refresh a server-based Omniscope Source.IOK file every 30 seconds. About 15 Omniscope Desktops use the Background Auto-Refresh option to keep an open connection to the Source.IOK file, and refresh the data continuously...effectively a (near) real-time update solution.