Visokio website     Downloads     Video tutorials     KnowledgeBase  
Idea: Connector - Elastic Search / Kibana? - Visokio Forums
Idea: Connector - Elastic Search / Kibana?
  • VoteVote Up3Vote Down powellblyth February 7, 2014 9:23AM
    Hi Visokio,

    We pull a lot of real-time data into Elastic Search, it would be awesome to be able to connect visokio to this to pull the later stats. We total live search reports per day etc
  •     tjbate February 8, 2014 1:46PM
    Toby - Good idea....Omniscope is ideal for automatically archiving local copies of search-engine aggregated and structured web event 'timeslices' cheaply as compact, accessible and easily re-usable IOK files. Indeed, an expanding folder of periodic IOK 'timeslice' files is effectively a local 'datamart' that other Omniscope files can access for batch analytics as if the archive folder were single file using the Batch Append operation.

    For those not familiar, Elastic Search (ES) is an open source, distributed, schema-less and RESTful search engine based on Apache Lucene. It has some very powerful features, including geo-spatial search on geo-tagged events. It runs at scale on most cloud cluster platform-as-a-service accounts, including MS Azure. Kibana is ES's visual interface for monitoring the activities and results of the search/aggregation cluster, e.g. most searched terms, sequences etc.

    ES and associated tools are all open source and have Java APIs for almost everything, so close integration with Omniscope is likely very do-able...can you post a sample of what types of data sets you are currently pulling and the types of queries that you are using to pull them? How would you integrate the ES-sourced data with other data sets?

    Does Kibana not give you any export options for the data it is displaying?
  • powellblyth February 10, 2014 5:16AM
    It's not about doing a manual Kibana export so much as being able to pull it in our visokio import process. We're not keen to invest the money for the automated version of Visiokio, but we try and make it as simple a daily manual process as possible, engaging manually with Kibana at the same time would not really be appropriate

    We currently use Elastic search for a wide range of tasks, including rate search speed-ups, but we archive data for Audit trails for manual and pricing updates (we are an online hotel sales company, among other things). We also log every search performed on the site (anonymously) and report on the available v unavailable hotels. This "big data", to latch onto a phrase, is easily summarised in terms of "hotels that have low availability", "destinations with high availability" among other things, which can be reported on a daily basis based on this large data. So to start with, our first implementation would be to display a chart in Visokio of bookings made in certain destination versus the availability scores, but it would be great to be able to pull varying data ranges, and use visokio's slider interfaces to pull these across.
    We could also cross reference cancellations in Visiokio, payments, deposits and costs to perform real analysis of various decisions we make.
  •     tjbate February 22, 2014 1:39PM
    Are you archiving the live data in Hadoop, then using ES as a Hadoop query tool, or pointing ES directly at source files?

    There is a webinar on this approach/architecture:


It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership