Performance: DataManager Aggregation block operation (2.9-Plus)

mraburgess · October 7, 2015 4:34PM

I have a DM aggregation block running on 14 million records x 38 fields. On 1 field doing unique values (has 5 million unique values) I am running various sums, 6 first non-empty values, 1 unique value count, 1 min & 1 max

It has been running for 2 days now yet is only using 6GB of my 64GB RAM (windows 10 pro, Xeon 8 core processor, Omniscope version 2.9 build 1765 (rc) [plus] x64)

Is there anything i can do to make aggregate use the available RAM more intelligently - i have other files use far more of the RAM, but always have this issue with aggregation.

thanks

paola · October 8, 2015 5:52AM

Hi Matthew, I will send you a link for a new build to test and see if it improves the performance.

enrico68 · November 27, 2015 8:37AM

Paola,

we are also very interested in generally increasing performance both in batch and in interactive data processing.
Can I also invite you to have a look at my last post about increasing available memory?

Welcome!

Categories

Ideas Parade

Tagged