Aggregate operation - record limit?

hannah.dadds · December 3, 2010 1:14PM

Can you tell me if there is a record limit to the aggregate operation - i.e. the file input or output?

We have batch process (currently 14 files) into field organiser into aggregate with circa 1million records.

Computer is running windows 7 64 bit, running okavango v 454 64 bit version.

steve · December 3, 2010 2:15PM

How many fields, and how much memory?

hannah.dadds · December 3, 2010 3:16PM

It's a brand new quad core computer with 4GB memory. And I've got 10 fields.

steve · December 3, 2010 5:06PM

This should be OK. Please get in touch if you run into difficulties.

hannah.dadds · December 6, 2010 4:32PM

When I plug in >1 million records to the aggregate I get a list of 10 records that don't aggregate. Reducing this input to just under 1 million and the aggregate works fine.

steve · December 6, 2010 5:04PM

I'm not sure I follow. I have set up 1,200,000 records of 5 categories, randomly distributed, then am aggregating to produce 5 records, as expected.

Are there any errors or red messages shown?

hannah.dadds · December 6, 2010 5:11PM

Sorry, let me try and explain a bit better.

I have a list of over 1 million records with 10 fields, one of which is Email address. I want to aggregate by email address, taking min/max/sums of the other fields.

When I try this with over a million records it doesn't work properly and I get around 10 records where email address is duplicated, when there should be a unique entry for each email address. This doesn't happen however when I reduce the number of records to be less than 1 million.

Does that make more sense?

Welcome!

Categories

Ideas Parade