Visokio website     Downloads     Video tutorials     KnowledgeBase  
Performance: DataManager De-Tokenize Block Is Slow? - Visokio Forums
Performance: DataManager De-Tokenize Block Is Slow?
  • cmollo October 16, 2013 2:46PM
    I am using a De-Tokenize block in my DataManager processing flow. About 71,000 records are fed into the De-Tokenize block and about 274,000 records flow out from the De-Tokenize block. It took 75 minutes for the De-Tokenize block to do its processing. The other blocks I'm using (Expand Value, Field Organizer) take around a minute or less to operate on the 71,000 records. I realize that the De-Tokenize block is creating a lot of new records, but 75 minutes seems really slow.

    I am using Ominscope 2.7 Build 459 x64. My computer has a quad-core i5 2.6 GHz processor with 4 GB of RAM. Nothing else of any significance was running on the computer at the time.

    Also, the progress spinner in the De-Tokenize block immediately displays 38% and stays at 38% the whole time. It would at least be nice to get an accurate indication of percent done.

    Is there any thing I can do to speed-up the de-tokenize process?

    Thanks,
    Chris
  • 2 Comments
  •     chris October 18, 2013 10:57AM
    Hi,

    Thanks for letting us know. You are right that this is very slow. I've sent you a private email asking you to send us this file for investigation.

    Thank

    Chris
  •     chris October 30, 2013 1:40PM
    Hi,

    We have now significantly improved the performance of the De-tokenize operation. In the example you describe above the De-tokenize now executes in 1 or 2 seconds. Unfortunately because of the risk of introducing bugs we have implemented this in Omniscope 2.9 (b578+).

    You need to be an Alpha partner to use Omniscope 2.9.

Welcome!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership