Performance: DataManager - Long multi-block flows slow?

Mees · December 9, 2012 3:45AM

Hi there - We have an IOK file that is only used as DataManager file. The number of DM blocks is large (around 50) but the number of records in Omniscope (the blue block) is only 50k and 7 fields. It takes a very long time for the file to open and when it is open it reacts very slow.

It uses up to 10GB RAM to open on a 20GB RAM 64bit server?

Question: does the number of DM blocks affect the iok files performance? Even when there is hardly any data in?

Thanks, A

tjbate · December 9, 2012 10:02AM

Arjan - When you say that the file is a transformational DataManager-only file, that implies that your Output blocks are not primarily the blue, in-memory portable IOK Omniscope visual discovery file, but primarily other file types, such as relational database tables, spreadsheets etc.

If you do not plan to make any use of this transformed data set as an Omniscope DataExplorer IOK Report file, or as an input Source IOK file in other downstream file chains, then do NOT connect the blue Omniscope IOK output box into the chain, and even delete the blue box from the data flow in the workspace. This will greatly reduce the maximum RAM footprint of transformational (ETL-only) files.

Mees · December 9, 2012 4:04PM

thanks Tom, will do.

Have also split the iok files and appears to improve performance as well, just for info...

Welcome!

Categories

Ideas Parade

Tagged