Toby - Good idea....Omniscope is ideal for automatically archiving local copies of search-engine aggregated and structured web event 'timeslices' cheaply as compact, accessible and easily re-usable IOK files. Indeed, an expanding folder of periodic IOK 'timeslice' files is effectively a local 'datamart' that other Omniscope files can access for batch analytics as if the archive folder were single file using the Batch Append operation.

For those not familiar, Elastic Search (ES) is an open source, distributed, schema-less and RESTful search engine based on Apache Lucene. It has some very powerful features, including geo-spatial search on geo-tagged events. It runs at scale on most cloud cluster platform-as-a-service accounts, including MS Azure. Kibana is ES's visual interface for monitoring the activities and results of the search/aggregation cluster, e.g. most searched terms, sequences etc.

ES and associated tools are all open source and have Java APIs for almost everything, so close integration with Omniscope is likely very do-able...can you post a sample of what types of data sets you are currently pulling and the types of queries that you are using to pull them? How would you integrate the ES-sourced data with other data sets?

Does Kibana not give you any export options for the data it is displaying?

It's not about doing a manual Kibana export so much as being able to pull it in our visokio import process. We're not keen to invest the money for the automated version of Visiokio, but we try and make it as simple a daily manual process as possible, engaging manually with Kibana at the same time would not really be appropriate

We currently use Elastic search for a wide range of tasks, including rate search speed-ups, but we archive data for Audit trails for manual and pricing updates (we are an online hotel sales company, among other things). We also log every search performed on the site (anonymously) and report on the available v unavailable hotels. This "big data", to latch onto a phrase, is easily summarised in terms of "hotels that have low availability", "destinations with high availability" among other things, which can be reported on a daily basis based on this large data. So to start with, our first implementation would be to display a chart in Visokio of bookings made in certain destination versus the availability scores, but it would be great to be able to pull varying data ranges, and use visokio's slider interfaces to pull these across.
We could also cross reference cancellations in Visiokio, payments, deposits and costs to perform real analysis of various decisions we make.

Are you archiving the live data in Hadoop, then using ES as a Hadoop query tool, or pointing ES directly at source files?

There is a webinar on this approach/architecture:
http://info.hortonworks.com/WebinarElasticSearchandHortonwork.html