Visokio website     Downloads     Video tutorials     KnowledgeBase  
Performance: ETL - Is Bulk Insert Possible? - Visokio Forums
Performance: ETL - Is Bulk Insert Possible?
  •     kattasureshkumar October 25, 2012 10:56AM
    We are using the Omniscope Desktop Edition to import & validate the RAW data (from flat files) and output directly to Oracle Database tables. We are not loading the data into the in-memory Omniscope block, we are only doing ETL of relational database tables.

    While importing and validating the records we had not faced any performance issues, but while inserting these records to Oracle Database tables, we are facing performance issues. It is taking around 8 Hrs to insert 2 Million records on 64 Bit, 4 GB RAM system when we use the "DATABASE OUTPUT" block.

    I suppose the tool is using manual/one by one insertion commands. Please let us know if we can do bulk inserts using this tool so that the performance when inserting large numbers of records can be improved.
  • 5 Comments
  •     chris October 25, 2012 11:04AM
    Hi - Thanks for the question. We recently improved the performance of the database insert to support bulk-inserting data. This is managed automatically and is available in the latest version of Omniscope 2.8.
  •     kattasureshkumar October 25, 2012 11:08AM
    Thank You for the quick response. When would be the release date of this version. Currently we are checking for the performance of the tool so that we can update these results to the Client. If we can have an updated version very shortly then we can test the results in the latest version.
  •     tjbate October 25, 2012 11:41AM
    Suresh - I have updated you to Alpha Partner status so that you can test/benchmark with version 2.8 alpha.
  •     kattasureshkumar October 26, 2012 7:07AM
    Thank You for giving me access to the Alpha versions. Installed the Alpha version 2.8 alpha b329 and here we are able to insert the records into the Oracle tables in a very quick time compared to the version 2.7. Here we were able to load 10 Million records into Oracle DB Tables in 20 Minutes.
    Sorry to iterate my question again, when would be the final version of 2.8 be released.
  •     tjbate October 31, 2012 2:04PM
    Suresh - Announcements about version 2.8 progress from alpha to beta to stable release will be made on this thread:

    http://forums.visokio.com/discussion/1671/omniscope-2.8-alpha-now-available-for-private-testing
Add a Comment

Welcome!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership

Categories

Tagged