I am using a De-Tokenize block in my DataManager processing flow. About 71,000 records are fed into the De-Tokenize block and about 274,000 records flow out from the De-Tokenize block. It took 75 minutes for the De-Tokenize block to do its processing. The other blocks I'm using (Expand Value, Field Organizer) take around a minute or less to operate on the 71,000 records. I realize that the De-Tokenize block is creating a lot of new records, but 75 minutes seems really slow.
I am using Ominscope 2.7 Build 459 x64. My computer has a quad-core i5 2.6 GHz processor with 4 GB of RAM. Nothing else of any significance was running on the computer at the time.
Also, the progress spinner in the De-Tokenize block immediately displays 38% and stays at 38% the whole time. It would at least be nice to get an accurate indication of percent done.
Is there any thing I can do to speed-up the de-tokenize process?
We have now significantly improved the performance of the De-tokenize operation. In the example you describe above the De-tokenize now executes in 1 or 2 seconds. Unfortunately because of the risk of introducing bugs we have implemented this in Omniscope 2.9 (b578+).
You need to be an Alpha partner to use Omniscope 2.9.