Idea: Connectors - creating custom connectors?
-
We are currently using a Java app to collect data from a remote database and insert into a mySQL database for reading into Omniscope. The amount of data we are aiming to collect is > 100 records per second, each record with 5-10 fields.
We are finding that this is causing significant bottlenecks on HDD or in the mySQL read/write process.
So we would like to know how we might develop our own custom Omniscope connector to collect the data from the remote database and read it directly into Omniscope memory or at least into Datamanager RAM.
Please advise whether and how we might modify our Java app to do this. -
8 Comments
-
We do not know what kind of Database is on the remote server. We are using an API to retrieve the data (apologies if this is not the correct language to describe what we are doing). We have developed a Java app that sends instructions to the API and then receives the data provided and logs it to our mySQL database. We would like somehow to put the received data directly into RAM memory accessible by Omniscope and later save to HDD/SSD during off peak traffic periods. Our aim being to reduce the latency of receiving data into Omniscope.
-
OK, so am I right in thinking that the Database is on another companies server, and they have developed an API to allow it's clients to access the data in the Database? If so are you able to let us know what company they are and what data-service your trying to access? Is the API freely available? We don't allow users to build they're own connectors at the moment, but there are a number of options available:
1. We could build the connector. Please see http://forums.visokio.com/discussion/1428. I suggest you contact us directly about this if you see this is the best solution.
2. As suggested by Steve, if the API provides data as RSS/Atom then the "Custom feed" block should allow you to import the data.
3. If you want to write your own connector, you could create a simple web service such as a Java servlet that was accessible via a URL (perhaps using different parameters to specify the data you wanted to download). You could then access the data in Omniscope using the "URL" source block.
-
Chris, yes you are correct that the Database is on another company's server, and they have developed an API to allow clients to access the data for a monthly fee and subject to having a client account.
We commissioned a programmer to write a Java program to retrieve the data and write to mySQL. This works well and we can read the data from mySQL into Omniscope. This is fine for development work.
However our objective is to produce a fast production system that will monitor the incoming data and generate output with very short delays (some milliseconds only). So we need to eliminate the time delay of reading and writing to/from mySQL.
The question then is who can/should write the appropriate code to retrieve the data and put it directly into Omniscope?
a) our current contracted programmer
b) Visokio
c) another specialist?
What are the implications of this choice?
- Ownership of the code
- Maintenance and updating costs -
Hi,
Please free free to call us now if you want to discuss this in detail. I will provide a general answer below.
You have several options here.
If you have developers who have knowledge of the API you could build a web-service (ASP/JSP/Servlet etc) that will retrieve and return the data. You could then use the "URL" or the "Custom feed" block to retrieve the data. For example your webservice could be accessed on a URL such as:
www.mysite.com/getdata?rowCount=100&format=csv&....
The disadvantage of this approach is that you would have to hard-code the parameter names and values (in this case 'rowCount' and 'format').
We are planning on creating a special "Custom source" block that allows you to configure the parameter names and default/min/max values. You would still need to set-up a web-service to retrieve the data from, but you would be able to configure the parameter values inside a form rather than hard-coding them inside the URL. We will post more details on this on the Visokio forum very soon, or we can discuss this over the phone.
For both of these options you would retain full ownership of the API code.
The final option would be to for us to create a standalone connector. There are various factors that we need to consider before we can commit to developing this, such as the usefulness of the connector to other clients and the complexity of the API. Again we can discuss this in more detail in a phone call. -
Just a note to add that database write performance has been speeded up a lot:
http://forums.visokio.com/discussion/comment/6815/#Comment_6815
Welcome!
It looks like you're new here. If you want to get involved, click one of these buttons!
Categories
- All Discussions2,595
- General680
- Blog126
- Support1,177
- Ideas527
- Demos11
- Power tips72
- 3.0 preview2
Tagged
To send files privately to Visokio email support@visokio.com the files together with a brief description of the problem.